Menu
Is free
check in
the main  /  Multimedia / Tuning Yandex Mail Collection. Collect all letters from different boxes in one

Setting up Yandex Mail Collection. Collect all letters from different boxes in one

Probably only units have not come across the problem that they are electronic address Stopped them like or just not needed. But that if a large number of contacts are connected with the box, to lose which I would not want.

The first solution that can come to mind is to create a new beautiful address and at the same time to constantly check the second. It's complicated. In this case, we start creating even more boxes and get confused with a large number of addresses.

There are many inconveniences: there is a risk forget password, continues to annoy the address [Email Protected]When you are the director of a serious company, and there is a lot more!

The solution is and it is very simple. And convenient.

When you created a box with an ideal address, connect the collection of letters from the old mail. All letters from connected boxes can be found inside the main mail in the folders of the same name. Everything is neatly, understandable and ordered.

The collector saves time to switch from the box to the box and allows you to fully go to a new address without fear that someone will remain unanswered.

How to setup?

Yet again. Everything is very simple. We tell:

  1. Go to the settings of your mailbox;
  2. Select the section "Mail from other boxes";
  3. Enter the email address and password from the box from which you want to collect letters.

After connecting a new address, you will see it in total list On the same page.

In this section, you can easily manage the collector: pause work, delete or add additional boxes. The collected letters will be stored in a separate folder with internal subfolders created by you in the old box.

By the way, if you firmly decided to use a new beautiful address, but you are too lazy to notify everyone who still writes to the old mail, configure the filtering rules with an automatic response from a new address.

To do this, configure the filter and in the column "if the letters to someone" specify your old address. Select the action "Result message". If you are in the message you specify the address of the new box, then all who will write to the old mail will receive the answer and will not lose your contacts.

Switch from one tab with Gmail to another with "Yandex. Good" rather tedious. And if you have a dozen accounts In different mail providers, the morning check of the new correspondence and turns into torture at all. By combining letters in one place, you get rid of this inconvenience.

Web interface

Most email providers provide a built-in email collection feature from multiple mailboxes. For example, let's try to collect letters from all your boxes in one Gmail account.

First of all, make sure that in the postal service, from where you want to pick up letters, there are access via POP protocol. Then go to the Gmail settings and click "Settings" in the upper right corner. Click the Accounts tab and find the "Get mail from other accounts" section.

Add an account, enter the email address and click "Next", then enjoy the password. In order for the mail exchange between the providers to be safer, make sure that the option "Always use a protected connection (SSL) upon receipt of mail is enabled. Then click on "Add Account".

Now letters coming to the added address will automatically be collected in the Inbox of your Gmail. You can add so many mail addresses as you need.

Desktop clients

Email need software. Yes, you can do a lot in the mail client (sometimes much more than Google now allows you to do in Gmail). But the web interface is not compared with the native application. Even the most modern Web UI is a bottleneck in working with mail.

Rife Nidlman, CNE.com

Desktop postal clients Good in that they can aggregate letters from a variety of accounts. At the same time, you can work with dozens of addresses and not even think about what provider there is a service provider.

Most hand-made email clients have a built-in total Inbox. Even if your favorite client supports work only with separate folders for incoming letters, you can still easily assemble them along with smart filters.

Despite the fact that Outlook is designed for simultaneous management of multiple mailboxes, it still displays a separate InBox for each account. But it can be easily corrected using filters.

Outlook Outlook, go to the "Folder" tab and click "Create Search Folder" on the toolbar. Then select "Creating a custom search folder." Click "Select", but do not select the search criteria so that all new messages go to the folder. Name the folder as you like, for example "all mail".

Click on the "Overview", select all the folders and boxes from which you want to collect mail, and check the "Search in Inned Folders" option.

Now letters from all mail accounts you have added to Outlook will appear in the smart folder you created. You can enable the "Display Favorites" option so that your new mail is always in sight.

Mac email client provides a single folder for incoming letters from all connected accounts. Just add your accounts, and all new letters will be collected in one place.

In Thunderbird to collect all your letters in one folder "Incoming" is very simple. Go to the View menu (if the menu bar is not displayed, press Alt). Then select "Folders" → "Combined". Now you will have one "Inbox" folder for new letters, one "Cherniviki" folder, one folder "sent" and one "archive". You will not have to look for a long time that it lies. In this case, messages will be, as before, stored on servers of your postal providers.

Mobile clients

The combined Inbox is in many mobile mail clients, including Gmail. Gmail application collects your incoming email accounts, including Yahoo, Outlook or other services.

If you do not like the application from Google, you can try third-party mail clients such as Outlook or Mymail.

Maybe you have your ideas how to group mail in one place? Share in the comments.

This article will discuss what "Mail from other boxes" and "Collecting letters from other accounts". Views of postal services and comparisons.

As mail service functionality is updated, we will publish news at the end of this article.

Mail service post service, search engines Yandex and Google have been offering a very useful and necessary functionality for several years, which allows you to combine all other e-mail accounts in one mail. In other words, it costs to add one time and configure all your "mails", and it will be possible to manage them in one window without permanent login and password.

Mail services offer us a huge place to store all incoming letters, a convenient interface, good management functionality and much more. To test the capabilities of the listed postal services, select three parameters to evaluate their work: 1) you need to manage various e-mail accounts, including with different services and sites, through one postal client; 2) have a personal signature and a name for each mail; 3) Delete letters from servers of those mail accounts that have been added to Mail, Yandex or Google Mail.

Our task looks like this:

So, proceed. We first test the service from Mail.Ru, which tells us: "You can collect mail from all your mailboxes in one box on Mail.ru.

Set up the collection of letters from any server working on iMAP protocol or POP3. " To add an external mail other than @mail, @inbox, @list, @bk, you need to have at least one mail that is already available on the listed servers from Mail. In other words, you first register mail mail, and then add other mail accounts. The registration process on Mail is simple, it makes no sense to describe it, but adding other e-mail to this post account we describe.

Next, you will be prompted to enter a username and password from any box that you use, and within 3-5 minutes "mail collector from Mail" will create a folder with the name of the added mail next to the "Inbox" folders, "sent", "Spam" and Other.

In fact, the letter collector will earn within 10-15 minutes and in the newly created folder from the box you specified will add all letters. All incoming letters for the new e-mail will come to this folder, will also be possible to send letters from the attached box. When creating a new letter, the "from" field will appear, in which you can choose from which mailbox will be sent to the letter.

The advantages of mail service Mail.Ru can include the simplicity of adding and ease of setting up other boxes, no need to specify various protocols (POP3, SMTP). It is very important because not every user knows what it is.

The minuses of Mail.Ru are to assign a name and signature for each new mail added (any new e-mail, in addition to the main), that is, the name and signature of the main mail will be distributed on each address. Another major disadvantage is that when downloading letters in Mail, there is no possibility of automatic removal of original letters from the attached box. Why is it so important to delete letters from the added box server? This will avoid filling the attached mail, because External e-mail accounts, as a rule, have too small size to store letters. For example, when overflowing a mail space in an external account, Mail.Ru will not display any new letters until you enter into an external mail interface and directly from there you will not delete the letters to release the place. In addition, we will bring mail to Mail in the hope of getting a lot of space for the storage of our mail, which may be limited to hosting resources. Thus, despite the fact that Mail.Ru offers almost unlimited mailbox volume, when adding mail from another domain zone (for example, [Email Protected]), Your added e-mail resources will not increase in Mail.Ru, and will continue to be limited to hosting resources (in our case, resources Primer.ru).

The possibility of adding another mail to the Mail service did not please our task on two points of three, and in total scored one score.

True, Mail.Ru has another way to add an external mail, and more precisely, the connection or input to the outer mail. It looks like this:

This method is not suitable for our task, because This is not a mail collector, but a user mode that connects to the mail and each time loads all folders and letters into the Mail interface. But it is possible to add personal signature and name for each mail here, it is possible to send letters from the configured mail. Removing letters from loaded boxes does not work, because This is not a mail collector.

Important! Some users mistakenly think that if the letters appeared in Mail, the mail with which the download occurred can be cleaned. We will pay attention to once again that Mail.Ru works as a mail viewer, and if you clean the mail, it will be removed in the main box, and in Mail.

Second Test Yandex with his letter collector. Tasks are delivered similar to: Collect mail, assign your signature and name for each mail and name, as well as delete all mail from the added box.

A collector of letters in Yandex is configured by a similar scheme, but if the mail connection occurs from some site, for example, Site.ru, you will need to register additional settings, see the screenshot.

Mail service did this work for us, Yandex offers in manual mode Stop all the settings from third-party mail. There is nothing complicated here: in the Login field, we enter the full name of the mail ( [Email Protected]), In the Server field, enter only the domain (site.ru), all other settings are better to leave the default, or change if you know what you are doing. After the successful addition of mail, the inscription will appear:

Unlike Mail, the collector, Yandex proposes to choose whether to save the original letters in the added drawer or not. In the whole, the letter of letters from Yandex works on a similar scheme with Mail.Ru, it is possible to select a mailbox from which to send a letter. But with the signature of letters from Yandex, the same problem: there is no way to set an individual name for each added mailbox for each added mailbox.

Judging by the navigation, the possibility of adding an individual signature for each mailbox is, but in fact it does not work. Select mail, we prescribe the name and signature, click Save. Everything, the signature is saved and running, but it is also saved for all other mailboxes, that is, assigning the name and signature to one box, you automatically change these parameters in all united mails in the Yandex. Even in case of choosing any other box, the signature will now everywhere will be similar.

Total Yandex has successfully coped with the fee and sorting mail, knows how to remove the originals of letters from the added box, but does not know how to work with the signatures! With our task, the letter of letters from Yandex did not cope and dial two points out of three.

I would like to note that, "Running" in all three sets on the assembly of letters, it seems that Mail "slept" his functionality on the ambulance hand. Although it will satisfy most users, but, honestly, Mail made a tick mail collector to celebrate in a competitive race. It works well and without failures, but with our task the collector from the mail did not cope. In the mail interface there are several advertising blocks, but they do not interfere with working with letters.

As for Yandex, who created a letter collector with a long time ago, it seems that the support of this particular industry is not engaged, because it is not possible to correct such a simple functionality (the ability to add multiple signatures for drawers) should not be difficult for such a powerful search engine. Nevertheless, Yandex has another unique functionality, which allows you to bind domains on their DNS server, and already in the subsequent to create and configure mail. But for such operations and settings require knowledge and time. And although there is nothing complicated there, in any case, it is not for most Runet users. Great advantage - No advertising in the Yandex Mail interface! Minus - Restriction in the collector of letters on 10 mailboxes.

Finalist and winner - a collector of letters from Google, who has coped with our task, despite even restriction in 5 boxes

Probably, many of you in our practice have come across the task of collecting mail from a number of boxes. Why can it be needed? Probably because it is a universal mechanism for exchanging data between systems. Many libraries for any languages \u200b\u200bimplementing SMTP, POP3, IMAP, ready-made solutions to implement the stack of messages (as I find it difficult to call the mailbox ...), etc.

It is not surprising that many integration tasks are implemented through the mail. There is a certain service that knows how this mail is quickly pick up, categorizing and perform the necessary actions.

To whom the code below is sufficient below - may not read further:

Foreach (Var Mailbox In Mailboxes) Using (Var Client \u003d New Pop3Client ()) (Client.connect (HostName, Port, False); Client.Authenticate (User, Password); var count \u003d client.getMessageCount (); for (var i \u003d 0; I< count; i++) { Mail = client.GetMessage(i + 1); var cat = SortMail(Mail); DoSomething(Mail, cat); } }

What do we do

Immediately make a number of assumptions:
1) You need to collect mail for several systems. Maybe in the future, more for several. And yet ... In general, the solution should be universal;
2) Mail may be much - follows from clause 1 (and otherwise I would not write this post);
3) mail will have to pars
4) All boxes of service - users do not climb there.

What we will use

The system should work 24/7, so we implement it in the form of Windows Service. For these purposes, I propose to immediately use Topshelf.

Of course, everything should be parallel. Here, my favorite TPL DataFlow library comes to the scene.

We will take the mail by POP3. All "fashionable things" IMAP in this task is unnecessary - it is necessary as quickly as possible and easier to pick up the source letter and delete it to the server. Pop3 is enough for the eyes. Use OpenPop.net.

As an optional, fasten monitoring in Zabbix. (We gathered to work 24/7 and issuing a praised speed - you need to follow this).

Go

Create a regular console application. Open a nuget console and put all the necessary packages:

Install-Package NLog Install-Package OpenPop.net Install-Package Topshelf Install-Package Microsoft.tpl.Dataflow
Go to the project folder, create a app.debug.config and app.release.config. Upload a project from the studio, open its code (here and then Topcrawler.csproj). Add to section with config.

Configuration

App.config App.config


And the lower Target for MSBuild:

TRANSFORM TARGET.

$ (Targetfilename )config


Personally, I used exactly in this way - in the old manner - add transformation of configs to separate media.
For convenience, I suggest strongly-type configs. A separate class will read the configuration. (On the theoretical aspects of such a solution can be communicated in the comments). Configations, logs, monitoring - a great reason to implement the Singleton pattern.

Create a folder in the project the same name (there must be an order). Inside Create 3 Class - Config, Logger, Zabbix. OUR LOGGER:

Logger.

sTATIC Class Logger (GET; Private Set;) Public Static NLOG.logger Archive (GET; Private Set;) Static Logger () (Log \u003d LogManager.getLogger ("Global"); Archive \u003d LogManager. GetLogger ("Archivator");))


Monitoring with Zabbix deserves a separate post, so I will simply leave here a class that implements agent:

Zabbix

namespace Topcrawler.Singleton (///

/// Singleton: Zabbix Sender Class /// Static Static ZabbixSender Sender (GET; Private Set;) Static Zabbix () (Sender \u003d New ZabbixSender () (config.zabbixServer, config.zabbixport);)) Struct ZabbixItem (Public String Host; Public String Key; Public String Value ;) class ZabbixSender (internal struct SendItem (// ReSharper disable InconsistentNaming - Zabbix is \u200b\u200bcase sensitive public string host; public string key; public string value; public string clock; // ReSharper restore InconsistentNaming) #pragma warning disable 0649 internal struct ZabbixResponse ( Public String Response; Public String Info;) #pragma Warning Restore 0649 #REGION --- Constants --- Public Const String DefaultHeader \u003d "ZBXD \\ X01"; Public Const String SendRequest \u003d "Sender Data"; Public Const int defaultTimeout \u003d 10000 ; #EndRegion #REGION --- Fields --- Private Readonly DateTime _dtunixmintime \u003d datetime.specifyind (new datetime (1970, 1, 1), datetimekind.utc); Private Readonly int _timeout; Private Readonly String _zabbixserver; PRIVATE READONLY INT _zabbixport; #endregion #region --- Constructors --- public ZabbixSender (string zabbixserver, int zabbixport): this (zabbixserver, zabbixport, DefaultTimeout) () public ZabbixSender (string zabbixserver, int zabbixport, int timeout) (_zabbixserver \u003d zabbixserver; _zabbixport \u003d zabbixport; _Timeout \u003d Timeout;) #Endregion #REGION --- Methods --- Public String SendData (Return Senddata (New List (1) (ITM)); ) Public String SendData (List LSTDATA) (Try (Var Serializer \u003d New JavaScriptSerializer (); var values \u200b\u200b\u003d new list (lstdata.count); Values.Addrange (Host \u003d Itm.Host, Key \u003d Itm.Key, Value \u003d Itm.Value, Clock \u003d Math.floor ((datetime.now.touniversaltime () - _dtunixmintime). TotalSeconds) .Tostring (CultureInfo.invariantCulture)))); var json \u003d serializer.serialize (new (Request \u003d SendRequest, Data \u003d Values.toarray ())); var header \u003d encoding.ascii.getbytes (DefaultHeader); var length \u003d bitconverter.getbytes ((long) json.length); var data \u003d encoding.ascii.getbytes (json); Var Packet \u003d New Byte; Buffer.blockcopy (Header, 0, Packet, 0, Header.Length); Buffer.blockcopy (Length, 0, Packet, Header.Length, Length.Length); Buffer.blockcopy (Data, 0, Packet, Header.length + Length.Length, Data.Length); Using (Var Socket \u003d New Socket, sockettype.stream, protocoltype.tcp)) (_zabbixserver, _zabbixport); socket.send (packet); // Header Var Buffer \u003d New Byte; ReceivData (Socket , buffer, 0, buffer.length, _timeout); if (defaultheader! \u003d encoding.ascii.getstring (buffer, 0, buffer.length)) Throw New Exception ("Invalid Header"); // Message Length Buffer \u003d New Byte ; ReceivData (Socket, Buffer, 0, Buffer.Length, _Timeout); var datalength \u003d bitconver.toint32 (buffer, 0); if (datalength \u003d\u003d 0) Throw New Exception ("Invalid Data Length"); // Message Buffer \u003d New Byte; ReceivData (Socket, Buffer, 0, Buffer.Length, _Timeout); var response \u003d serializer.deserialize (Encoding.ascii.getstring (buffer, 0, buffer.length)); Return String.Format ("Response: (0), Info: (1)", response.response, response.info); )) Catch (EXCEPTION E) (Return String.format ("Exception: (0)", E);)) Private Static Void ReceivData (Socket Pobjsocket, Byte Buffer, Int Offset, int Size, int timeout) (Var StartTickCount \u003d Environment.tickcount; var received \u003d 0; do (if (environment.tickcount\u003e starttickcount + timeout) Throw New TimeoutException (); try (Received + \u003d POBJSocket.Receive (Buffer, Offset + Received, Size - Received, SocketFlags.none) ;) Catch (SocketException EX) (if (ex.socketercode \u003d\u003d socketerror.wouldblock || Ex.SocketerRorcode \u003d\u003d socketerror.iopending || Ex.SocketerRorcode \u003d\u003d socketerror.nobufferspaceAvailable) Thread.Sleep (30); Else Throw;) ) While (Received< size); } #endregion } }


Configs ... It's time to do at least something interesting. First, in the configs we will store the boxes that we interview. In the second DataFlow settings. I suggest:

Configs



So, the host and port where it is connected, the user and password - everything is clear here. Further type of box. Suppose the service is used by marketing (as well as other departments). They have drawers where autows will fall on mailing, as well as the FBL spam reports. The box itself is already categorized a letter, so for such situations immediately specify the type of box. With the DataFlow settings, it will be clear further when I start creating objects. Here we will have our own sections in the config. Manuals heap how to do it, so I just show the result:

We define types

#REGION --- Types --- Static Class MailBoxType (Public Const String BO \u003d "BO"; Public Const String CRM \u003d "CRM"; Public Const String FBL \u003d "FBL"; Public Const String Bounce \u003d "Bounce";) Class MailboxInfo (Public String Type (GET;) Public String HostName (GET; Set;) Public String User (Get; Set;) Public String Password (GET; SET;) Public Int Port (Get; Set;)) Class DatablockOptions (GET; SET;) Public int BoundDedDedcapacity (GET; SET;) Public DataBlockOptions () (maxdop \u003d 1; boundedcapacity \u003d 1;)) #Endregion


Create sections

///

/// Custom Config Section /// public class CustomSettingsConfigSection: ConfigurationSection (public CredentialsCollection CredentialItems (get (return base [ "CredentialsList"] as CredentialsCollection;)) public DataBlockOptionsCollection DataFlowOptionsItems (get (return base [ "DataFlowOptionsList"] as DataBlockOptionsCollection;)))


///

/// CUSTOM Collection - Credentials List /// Public Class CredentialScollection: ConfigurationElementCollection, Ienumerable (Protected override ConfigurationElement CreateNewElement () (return new CredentialsElement ();) protected override object GetElementKey (ConfigurationElement element) (return ((CredentialsElement) element) .Username;) public CredentialsElement this (get (return BaseGet (index) as CredentialsElement;) ) Public New Ienumerator < Count; i++) { yield return BaseGet(i) as CredentialsElement; } } } /// /// Custom Credentials Item /// Public Class CredentialSelement: ConfigurationElement (Return Base ["HostName] AS String;)) Public String Username (Return Base [" UserName "] AS String;)) Public String Password (Geturn Base ["Password"] as string;)) Public String Type (get (Return Base ["Type"] AS String;)) Public String Port (Return Base ["Port"] as string;))) /// /// Custom Collection - Datablock Options List /// Public Class DatablockOptionScollection: ConfigurationElementCollection, Ienumerable (Protected override ConfigurationElement CreateNewElement () (return new DataBlockOptionsElement ();) protected override object GetElementKey (ConfigurationElement element) (return ((DataBlockOptionsElement) element) .Name;) public CredentialsElement this (get (return BaseGet (index) as CredentialsElement;) ) Public New Ienumerator Getenumerator () (for (var i \u003d 0; i< Count; i++) { yield return BaseGet(i) as DataBlockOptionsElement; } } } /// /// Custom Datablock Options Item /// Public Class DataBlockOptionSelement: ConfigurationElement (Public String Name] AS String;)) Public String MaxDop (Return Base [MaxDop] AS String;)) Public String BoundedCapacity (Geturn Base ["BoundedCapacity"] AS String;)))


I will not write a complete implementation of the config, it is implied that in the process of developing the parameters we need to be added there.

Our custom settings will be read like this:

Ready

public List. CredentialSlist (GET; Private Set;) Public Dictionary DataFlowOptionsList (get; private set;) ... static Config () (try (var customConfig \u003d (CustomSettingsConfigSection) ConfigurationManager.GetSection ( "CustomSettings"); // Get mailboxes foreach (var item in customConfig.CredentialItems) CredentialsList.Add ( New MailboxInfo (hostname \u003d item.hostname, port \u003d convert.toint32 (item.port), user \u003d item.username, type \u003d item.Type, password \u003d item.password)); // Get DataFlow Settings Foreach (Var Item In customConfig.DataFlowOptionsItems) DataFlowOptionsList.Add (item.Name, new DataBlockOptions (Maxdop \u003d Convert.ToInt32 (item.Maxdop), BoundedCapacity \u003d Convert.ToInt32 (item.BoundedCapacity)));) catch (Exception ex) (Logger.Log. Fatal ("Error At Reading Config: (0)", EX.Message); Throw;))


Somehow it is very prolonged, and we did not even reach the most interesting.

Lower the strapping from Topshelf, performance counters, communication with the database and turn to the point! Create a CRAWLER class - core. To begin with, read mail:

Private volatile bool _stoppipeline; ... public void start () (DO (var getMailSks \u003d _config.credentialslist.select ((Credentials \u003d\u003e Task.run (() \u003d\u003e GetMails (Credentials))). Tolist (); foreach (var task in getmailstasks) Task .Wait (); Thread.Sleep (2000);) While (! _Stoppipeline); // Stop Pipeline - Wait for Completion of All EndPoints // There will be a DataFlow stop of the IF conveyor (_stoppipeline) logger.log.warn ("Pipeline HAS been stopped by user ");)
Here, Lena took his own and I decided not to bother - if the boxes of about 20-30 can be used for each TASK and not steam about the number of streams. (I allow you to throw tomatoes.)

Go to reading:

Private Void GetMails (MailboxInfo info) (TRY (USING (VAR Client \u003d New Pop3Client ()) (
Immediately, consider the timings of access to the box - it is useful for the network diagnostics and server workload.

// Get Zabbix Metrics Var StopWatch \u003d New StopWatch (); stopwatch.start (); // Get Mail Count Client.Connect (info.hostname, info.port, false); Client.Authenticate (info.user, info.password); stopwatch.stop ();
We send data to Zabbix. Everything is simple - specify the host name (as it is headed in Zabbix), the key (again strictly as in Zabbix) and a string value.

// send it to zabbix zabbix.sender.sendData (New ZabbixItem (Host \u003d config.hostkey, key \u003d info.type + config.timingkey, value \u003d stopwatch.elapsedmilliseconds.tostring ())); Logger.log.debug ("Send [(0)] Timing to Zabbix: Connected to" (1) "AS" (2) ", Timing (3) MS", info.Type, info.hostname, info.user, stopwatch.elapsedmilliseconds); var count \u003d client.getMessageCount (); if (Count \u003d\u003d 0) Return; Logger.log.debug ("We" We Got New (0) Messages In "(1)", count, info.user); // Send Messages to Sorting Block for (var i \u003d 0; i< count; i++) { try { var mailInfo = new MessageInfo { IsSpam = false, Mail = client.GetMessage(i + 1), Type = MessageType.UNKNOWN, Subtype = null, Recipient = null, Mailbox = info }; Logger.Log.Debug("Download message from "{0}". Size: {1}b", info.User, mailInfo.Mail.RawMessage.Length);
DataFlow Pipeline will be created when creating a Crawler class. We believe that our first stage is to sort the letter.

While (! _Sortmaildatablock.post (MailInfo)) thread.sleep (500);
See how simple is the conveyor alone. All others, reading mail, throw messages there one by one. If the block is busy, POST will return False and we will simply wait until it is free. The current later at that time continues to work. That I call the parallelism without worries.

The message went to the conveyor, now it is possible to save it with a calm soul in the Raw archive (yes! Everything we read - save it to the file archive. Support service will then say Thank you later).

Let us configure, for example, the rotation of the archive:

Nlog.config



Then you can raise Logstash, but this is another story ...

// Save Every Mail to Archive Logger.log.debug ("Archive Message"); Logger.Archive.info (Functions.MessageTostring (MailInfo.mail)); ) Catch (EXCEPTION EX) (Logger.log.error ("Parse Email Error: (0)", EX.Message); Functions.errorscounters.Increment (); // Archive Mail Anyway Logger.log.debug ("Archive Message "); Logger.archive.info (encoding.default.getstring (client.getMessageAsbytes (i + 1)));) if (_config.deletemail) Client.DeleteMessage (i + 1); if (_stoppipeline) break; ) Logger.log.debug ("Done with" (0) ", info.user); )) Catch (exception ex) (logger.log.error ("General Error - Type: (0), Message: (1)", EX, EX.Message); Functions.errorscounters.Increment ();))
Here we used static error meters (in the context of the types of boxes), where ErrorScounters is:

Public Static Dictionary ErrorScounters \u003d New Dictionary ();
And the meters themselves can be done like this:

Counter.cs.

class Counter (PRIVATE LONG _COUNTER; Public Counter () (_Counter \u003d 0;) Public Void Increment () (interlocked.Increment (Ref _Counter);) Public Long Read () (Return _Counter;) Public Long Refresh () (Return interlocked .Exchange (Ref _Counter, 0);) Public Void Add (Ref _Counter, Value);) Public Void Set.Exchange (Ref _Counter, Value);))


Let us turn to the creation of the conveyor. Suppose we have drawers where auto autows are rawd. Such letters must be painted (what for the auto answer, from whom, according to which mailing list, etc.) and fold the result in the repository (database). Suppose there are boxes where FBL reports fall. Such letters immediately add to the database. All other letters consider "useful" - they must be checked on spam and send to the external system, for example, CRM.

As you already understood, this example is mainly considering the application of the collector for marketing tasks - the collection of statistics on mailing delivery, spam information.

So, we decided on the workflow. We declare the necessary blocks in the CRAWLER class:

Class MessageInfo (GET; PUBLIC BOOL ISSPAM (GET; SET;) Public String Subtype (GET; SET;) Public String Recipient (GET; SET;) Public MessageType Type (GET; SET;) Public MailboxInfo Mailbox (get; set;)) class crawler (// Pipeline Private TransformBlock _SortMailDatablock; Private TRANSFORMBLOCK. _SpamfilterDatablock; Private TRANSFORMBLOCK. _CheckBouncedatablock; Private TRANSFORMBLOCK. _identifydatablock; Private ActionBlock. _Addtocrmdatablock; Private ActionBlock. _Addtofbldatablock; Private ActionBlock. _addtobouncedatablock; ...
Create an initialization method and create a conveyor blocks (to initialize blocks, we use our wonderful sections from configs):

Public void init () (// *** Create Pipeline *** // CREATE TRANSFORMBLOCK TO GET MESSAGE TYPE VAR BLOCKOPTIONS \u003d _CONFIG.GETDATABLOKOPTIONS ("_ sortmaildatablock"); _sortmaildatablock \u003d new transformblock (mail \u003d\u003e Sortmail (Mail), New ExecutionDataFlowBlockOptions (maxdegreeofparallelism \u003d blockoptions.maxdop, boundedcapacity \u003d blockoptions.boundedcapacity)); // CREATE TRANSFORMBLOCK TO FILTER SPAM BLOCKOPTIONS \u003d _CONFIG.GETDATABLOKOPTIONS ("_ SPAMFILTERDATABLOCK"); _SPamFilterDatablock \u003d new transformblock (Mail \u003d\u003e Filterspam (Mail), New ExecutionDataFlowBlockOptions (maxdegreeofparallism \u003d blockoptions.maxdop, boundedcacity \u003d blockoptions.boundedcapacity)); // CREATE TRANSFORMBLOCK TO SORT BOUNES BLOCKOPTIONS \u003d _CONFIG.GETDATABLOCOPTIONS ("_ checkbouncedatablock"); _CheckBouncedatablock \u003d new transformblock (Mail \u003d\u003e Bouncetypecheck (Mail), New ExecutionDataFlowBlockOptions (maxdegreeofparallelism \u003d blockoptions.maxdop, boundedcapacity \u003d blockoptions.boundedcapacity)); // CREATE TRANSFORMBLOCK TO Identify Bounce Owner BlockOptions \u003d _Config.getDatablockOptions ("_ IdentifyDatablock"); _identifydatablock \u003d new transformblock (Mail \u003d\u003e GetRecipient (Mail), New ExecutionDataFlowBlockOptions (maxdegreeofparallelism \u003d blockoptions.maxdop, boundedcapacity \u003d blockoptions.boundedcapacity)); // Create ActionBlock to Send Mail to CRM BLOCKOPTIONS \u003d _CONFIG.GETDATABLOCPTIONS ("_ addtocrmdatablock"); _Addtocrmdatablock \u003d new actionblock (Mail \u003d\u003e AddToCrm (Mail), New ExecutionDataFlowBlockOptions (MaxdegreeOFPARALLISM \u003d blockoptions.maxdop, boundedcapacity \u003d blockoptions.boundedcapacity)); // CREATE ACTIONBLOCK TO SEND FBL TO MAILWH BLOCKOPTIONS \u003d _CONFIG.GETDATABLOCOPTIONS ("_ addtofbldatablock"); _Addtofbldatablock \u003d new actionblock (Mail \u003d\u003e Addtofbl (Mail), New ExecutionDataFlowBlockOptions (maxdegreeofparallelism \u003d blockoptions.maxdop, boundedcapacity \u003d blockoptions.boundedcapacity)); // CREATE ACTIONBLOCK TO SEND BOUNCE TO MAILWH BLOCKOPTIONS \u003d _CONFIG.GETDATABLOCOPTIONS ("_ addtobouncedatablock"); _Addtobouncedatablock \u003d new actionblock (Mail \u003d\u003e AddTobounce (Mail), New ExecutionDataFlowBlockOptions (MaxdegreeOFPARALLISM \u003d BLOCKOPTIONS.MAXDOP, boundedcapacity \u003d blockoptions.boundedcapacity));
Collect the conveyor in accordance with our scheme:

// *** Build Pipeline *** _sortmaildatablock.linkto (_SPamFilterDatablock, info \u003d\u003e info.Type \u003d\u003d MessageType.general); _SortMailDatablock.linkto (_addtofbldatablock, info \u003d\u003e info.type \u003d\u003d messagetype.fbl); _SortMailDatablock.linkto (_CheckBouncedatablock, info \u003d\u003e info.type \u003d\u003d messagetype.bounce); _SortMailDatablock.linkto (dataflowblock.nulltarget. (), info \u003d\u003e info.Type \u003d\u003d MessageType.unknown); / * Stub * / _checkboundatablock.linkto (_identifydatablock); _identifydatablock.linkto (_addtobouncedatablock); _SpamfilterDatablock.linkto (_Addtocrmdatablock, info \u003d\u003e! info.isspam); _SpamfilterDatablock.linkto (dataflowblock.nulltarget. (), info \u003d\u003e info.isspam); / * Stub * /
As you can see, everything is extremely simple - we associate the block with the following (with the possibility of setting the connection condition). All blocks are executed in parallel. Each unit has a degree of parallelism and a container (using the tank, you can adjust the queue before block, that is, the unit message accepted, but does not proceed yet). Thus, it is possible to set a high degree of parallelism for "complex" and long operations, such as the PARSING of the contents of the letter.

I will not describe the Matchaw DataFlow, it is better to read everything in the TPL DataFlow source.

Sortmaildatablock.completion.continuewith (t \u003d\u003e (ifataflowblock) ((IDATAFLOWBLOCK) .Fault (T.Exception); else _spamfilterdatablock.complete ();)); _sortmaildatablock.completion.continuewith (t \u003d\u003e (ifataflowblock) _addtofbldatablock) .Fault (t.Exception); else _addtofbldatablock.complete ();)); _sortmaildatablock.completion.continuewith (T \u003d\u003e (ifataFlowBlock) .Fault (T.Exception); Else _CheckBouncedatablock.complete ();)); _SpamfilterDatablock.comPlettion.continuewith (T \u003d\u003e (if (t.isfaulted) ((IDATAFLOWBLOCK) .FAULT (T.Exception); Else _Addtocrmdatablock.complete ();)); _CheckBouncedatablock.comPlettion.ContinueWith (T \u003d\u003e (ifataFlowBlock) ((IDATAFLOWBLOCK) .Fault (T.Exception); ELSE _IdentifyDatablock.complete ();)); _identifydatablock.completion.continuewith (t \u003d\u003e (ifataflowblock) _addtobouncedatablock) .Fault (T.Exception); else _addtobouncedatablock.complete ();)); )
Everything, in fact, the conveyor is already working, you can post messages to it. It remains only to stop him adding our Start method:

Start.

public Void Start () (DO (var GetMailSks \u003d _config.credentialslist.select (() \u003d\u003e getmails (() \u003d\u003e getmails (())). Tolist (); Foreach (Var Task in GetMailSks) task.wait ( ); Thread.Sleep (2000);) While (! _Stoppipeline); // Stop Pipeline - Wait for Completion of all EndPoints _sortmaildatablock.complete (); _addtocrmdatablock.completion.wait (); _addtofbldatablock.comtenance.wait (); _AdDTOBOUNCEDATABLOCK .Completion.wait (); if (_stoppipeline) logger.log.warn ("Pipeline Has Been Stopped by User");)


Go to delegates.
Sorting ... Well, let's say we all just have (complicate, then we always have time to complicate):

Private MessageInfo Sortmail (MailBoxType.CRM: Mail.Type \u003d MessageType.General; Break; Case MailboxType.Bounce: Mail.Type \u003d MessageType.Bounce; Break; Case MailBoxType. FBL: Mail.Type \u003d MessageType.fbl; Break;) Return Mail;)
Spam filter. This is on homework - use Spamassassin.
Here is a delegate:

Private MessageInfo Filterspam (MessageInfo Mail) (// TODO: Add Spamassassin Logic Return Mail;)
And classes for working with API SPAMAssassin (link to the project).
And we go to the parting of letters. Parsim We are auto autows. It comes into the MEF business.
Create a project (DLL) with interfaces for our plugins (let's call Interfaces).
Add an interface:

Public Interface Icondition (String Check (Message Mimemessage);) Public Interface IconditionMetadata (Type Type (GET;))
And ... everything. Our TOPCRAWLER depends on this project and the project with plugins will also use it.
Create a new project (also DLL), let's call Conditions.
Add Types of Answan:

#region --- Types --- static class BounceType (public const string Full \u003d "BounceTypeFull"; public const string Timeout \u003d "BounceTypeTimeout"; public const string Refused \u003d "BounceTypeRefused"; public const string NotFound \u003d "BounceTypeNotFound"; public const string Inactive \u003d "BounceTypeInactive"; public const string OutOfOffice \u003d "BounceTypeOutOfOffice"; public const string HostNotFound \u003d "BounceTypeHostNotFound"; public const string NotAuthorized \u003d "BounceTypeNotAuthorized"; public const string ManyConnections \u003d "BounceTypeManyConnections";) #endregion
And classes implementing our interface:

Public Class ConditionNotFound1: Icon Mimemessage (IF (! Mimemessage.MessagePart.Ismultipart) Return Null; Const String Pattern \u003d "Diagnostic-Code:. + SMTP. + 550"; var Regexp \u003d New Regex (Pattern , REGEXOPTIONS.IGNORECASE); Return Mimemessage.MessagePart.MessageParts.Any (part \u003d\u003e part.contenttype.mediatype \u003d\u003d "Message / Delivery-Status" && regexp.ismatch (part.getbodyastext ()))? Bouncetype.notfound: null ;)) ... Public Class Conditiontimeout2: icondition (Return bouncetype.timeout;) ...
As you noticizing the entire case in attributes. With their plugins and will be loaded.
Return to our project and load plugins:

Class Crawler (... // plugins Public Ienumerable \u003e BounceTypeConditions (Get; Set;) Private void Loadplugins () (TRY (VAR Container \u003d New CompositionContainer (New DirectoryCatalog (_config.plugindirectory), true); Container.comPoseparts (this);) Catch (Exception EX) (Logger.log .Error ("Unable to Load Plugins: (0)", EX.Message);)) ...
Loadplugins pull in the designer of our class. To explain in detail about the download mechanism will not - Google will cope better.

Go to our delegate check type bounce. The conditions will be applied in turn until the first will work - the exclusive method:

Private MessageInfo BouncetyPeCheck (TRY (VAR RES \u003d CONDITION.VALUE.CHECK (MailInfo.mail); if (res \u003d\u003d null) Continue; MailInfo.subType \u003d Res; Logger.log .Debug ("Bounce Type Condition [(0)] Triggered for Message [(1)]", Condition.metadata.Type, MailInfo.mail.headers.MessageID); Break;)) Catch (Exception EX) (Logger.log .Error ("Failed to Determine Bounce Type for Message" (0) ": (1)", MailInfo.mail.headers.messageID, EX.Message); logger.errorscounters.increment ();) Return MailInfo;)
Thus, if a new logic appears just enough to add a new class to the project with plugins that implements our interface and - Voila! An example of the second plugin to define the sender of the letter will not apply - so already a long post (the auto answer generated the server itself, so the sender should also be painted from the headers of the letter).

With the recording of the results in the database, too, nothing unusual. For example, so:

Private Void AddTobounce (MailWh.Bounceadd (Mail); Functions.ProcessedCounters.Increment (); Functions.log.debug ("Send Bounce to Mailwh");) Catch (Exception EX) (Functions.log. Error ("Error Saving Bounce Message" (0) "to mailwh: (1)", mail.mail.headers.messageid, ex .Message); functions.errorscounters.increment ();))

Bounceadd.

public Static Long Bounceadd (VAR CONN \u003d New Sqlconnection (ConnectionString) Using (VAR CMD \u003d New Sqldataadapter ("Bounceadd", Conn)) (var body \u003d message.mail.findfirstPlainTextVersion () \u003d\u003d NULL? message.mail.findfirsstmlversion (). getbodyastext (): message.mail.findfirstPlainTextVersion (). getbodyastext (); var Outid \u003d New SqlParameter ("@ id", sqldbtype.bigint) (Direction \u003d ParameterDirection.Output); cmd.selectcommand .CommandType \u003d CommandType.StoredProcedure; cmd.selectcommand.parameters.add (new sqlparameter (@ Rawmessage, message.mail.rawmessage)); cmd.selectcommand.parameters.add (new sqlparameter (@ Message ", Body)) ; cmd.selectcommand.parameters.add (new sqlparameter ("@ subjecters", message.mail.headers.subject ?? "")); cmd.selectcommand.parameters.add (new sqlparameter (@ MessageID ", message.mail .Headers.MessageID ?? "")); cmd.selectcommand.parameters.add (new sqlparameter ("@ addressto", message.mail.headers.to.address ?? "")); cmd.selectcomm and.parameters.add (new sqlparameter ("@ addressfrom", message.mail.headers.from.address ?? "")); cmd.selectcommand.parameters.add (new sqlparameter ("@ daterecieved", datetime.Now)); cmd.selectcommand.parameters.add (new sqlparameter ("@ bouncetypesyname", (object) message.Subtype ?? dbnull.value)); cmd.selectcommand.parameters.add (new sourcefrom, (@ SourceFrom ", (Object) Message.Recipient ?? dbnull.Value)); // TODO: add listid support cmd.selectcommand.parameters.add (new sqlparameter ("@ listid", dbnull.value)); cmd.selectcommand.parameters.add (outid); conn.open (); cmd.selectcommand.executenonquery (); Return Outid.Value AS LONG? ?? 0; ))


Sorry I did not have time to show Topshelf - the post and so already swollen.

conclusions

In this lesson, we learned that the task of collecting mail may not be as simple. The developed kernel allows you to quickly add new process steps - DataFlow blocks without affecting the existing logic. The plug-in subsystem allows you to quickly increase the script-like logic of the parsing, and the DataFlow itself parallels all the calculations (and we have the ability to flexibly configure multithreading for a specific machine). Topshelf gives us the ability to run service both in service mode and in the console mode to facilitate debugging.

In my opinion, Gmail has one of the most convenient web interfaces to work with email. In addition to just a pleasant and convenient interface, the creators have not forgotten about the functional part. Power chains, creating various folders, filters, etc. All this made Gmail one of the most popular postal services in the world. Many do not know about such a wonderful opportunity as a mail collector. For example, you have one or more mailboxes on your domain name, but standard interface Webmail and postal clients are not satisfied with you ( Mozilla Thunderbird, Microsoft Outlook, The Bat). In this case, you can use the mail collector and process all mail through Gmail, how to do this now we will look at it.

  1. An account in Gmail is needed, the registration process is quite simple and thinking and deal with this will not be difficult.
  2. After registration, in Gmail you need to go to the settings
  3. In the setting, select the "Accounts and Import" item

  4. In this section, we are interested in two items: "Send letters as:" And "check mail from other accounts (using POP3):". They have links "Add Other Email Address" and "Add your POP3 Email Account".

  5. In order to collect mail from another mailbox, click on "Add your POP3 email account". A new window appears, in which you need to enter the address of the mailbox and click "Next". After several fields appear that you need to fill out:
    Username
    Password
    POP server and port - POP server address and port for connecting to the server from which mail will be collected. You can find out this information in supporting your postal service, it is usually published in the "Question and Answers" section.
    Keep copies of the received letters on the server - If the checkbox is not installed, after downloading letters, these letters will be removed from the mailbox. .
    Always use a secure connection (SSL) upon receipt of letters - You should install if your postal service uses a secure SSL connection.
    Assign the label of incoming messages - Label, this is a visual mark near the letter. It is convenient to use for understanding, on which mailbox a letter was sent.
    Archive incoming letters (skip the incoming) - Letters collected from this mailbox will be archived, in their incoming it will not be.

  6. Click "Add Account", after which Gmail sends a check letter to the mailbox you specified. The letter will select two options for checking, link and code. The link you need to just go, and insert the code in the confirmation field. It's easier to go through the link, the code apparently goes out just in case, suddenly the link will be displayed in the letter incorrectly.
  7. Well, in fact, all, now our account on Gmail automatically collects letters from another mailbox. If there is a need to respond from this or another address using the same Gmail, click on the "Add other Email Address" link in the settings, fill the fields:
    "Name:" - You can enter any data, such as your name.
    "E-mail address:" - Indicate the postal address, on behalf of which we will answer.
    Check mark "Use as a pseudonym. "- This address will be used as a pseudonym of your main address and when sending a letter to it, it will be displayed in your incoming, described in more detail in Google Help on the link "More ...".
    Specify another address in the "Reply" field (optional) - By default, the answer comes to the same address that was used to be sent. By clicking on the link, you can specify another mailbox to which letters sent to this address will be delivered.
    Click "Further"

  8. Next, we see a few more fields that need to be fill.
    SMTP server and port - This information should be found at the postal service.
    Username - Usually this is an email address, this is a login to access the mailbox.
    Password - Password access to the mailbox.
    Protected connection using TLS (recommended) and secure connection using SSL - The second option (SSL) can only be chosen if your postal service uses this type of secure connection. Basically you can choose the first item.

    After clicking "Add Account", a check letter will be sent, by analogy with the addition of mailbox for mailing. I confirm and everything is ready.

Well, actually, all, so you can configure the assembly of letters and the answer from other boxes via Gmail. Small nuance, mail collection schedule from other Gmail mailboxes forms independently, separately for each such mailbox. That is, a sent letter to the mailbox from which you collect mail, in the Gmail interface, it is not displayed immediately, but only after checking. The periodicity of the check depends on the number and frequency of entering letters to this mailbox, the mailbox, the more often the letters come, the more often the verification will occur. You can not change this parameter, but you can force the check in the settings.