SendGrid is our friend, but the truth is more precious

As I already wrote , not so long ago we started working with the team of a popular freemium service, which is a project management system. The service generates a fairly large number of letters, calculated in thousands per day, these are mainly various notifications to users about the changes that have occurred and daily reminders of the approach / expiration of the performance of any work. Users can also reply to incoming messages and thus update the data. As the number of users increases, we notice that our SendGrid accounts are increasing , which, being a freemium service, we would like to minimize. We also wanted to understand how efficiently we use mail, and whether we send part of mail just nowhere, paying for the delivery of SendGrid.

Increasingly, questions began to arise that someone had not received some kind of notification or his response to a letter from the service was not received and published by them. Thus, the second problem was the inability to effectively engage in supporting user problems related to mail delivery, since I constantly had to look through both my own database and the SendGrid interface. Given that it took a lot of time, it often did not give any result. After sending the letter via smtp, we didn’t have any connection between the records in our database and the result on the SendGrid side.

The situation was similar with incoming mail. For those who have not yet encountered, I will describe how it works in SendGrid: letters are received by SendGrid, they are parked, and then the received data is sent by POST to the specified address, i.e. you. So, when a situation arises that someone's letter was not published in our service, it is not clear who is to blame - SendGrid could not parse it or our script did not work. You will understand us if you find yourself in a situation where the truthful answer is “well, I don’t know, damn why the letter didn’t come” you don’t write to the user and only promise to deal with his question in a short time. So that our promises do not become a lie, I set to work.

So - we have a need for full control over the process of delivering and receiving letters. The current state of affairs was sending (via smtp) and receiving mail via SendGrid, the complete lack of communication between our and SendGrid data, thousands of emails per day and their exponential growth, resulting in ever-increasing sums and the absence of an effective mechanism for solving user problems. The task is to link our data with SendGrid data as much as possible in order to reduce costs and improve the quality and speed of customer support.
')
First of all, we turned to their documentation for answers to our questions. It turned out that SendGrid has 2 hooks that give us the necessary control over the state of the letters. The first hook is Events - SendGrid sends notifications about events generated in their system to the specified URL. To identify a specific letter, they propose using a mechanism of unique arguments and categories, which can be transferred as a special header when sending a letter via SMTP. In the future, event notifications include these arguments and categories, which allows you to determine exactly which letter this event belongs to. What does this give? First, “if the cook does not lie to us,” we always know for sure whether the letter has been delivered to the user. Secondly, you can fully or partially automate the process of re-sending letters that have received the status of "Deferred" or hit the Bounces list. In addition, now we will quickly learn about the entry of a new email into the Bounces list and can promptly take action. There are additional bonuses associated with the use of categories of letters.

As mentioned above, the sent email can be assigned to categories and according to the documentation they can be up to 10. These categories are then used when creating reports in the SendGrid interface, they can be compared with each other on pretty charts. True, unfortunately, for some reason, they only have a single-level system of categories, and it would be more convenient for us to use a two-level approach. Two-level because the first level is determined by the application that sent the letter, and the second level is the action that caused it to be sent. For example, the Todo application sends letters when creating, editing and changing the end date of a recording. Using the second category gives us a more complete picture of who generates letters and how many within one application. However, there are similar actions in other applications, so the Add category itself is not as interesting for statistics as binding to a specific application. However, it is possible that such a slice of statistics will be useful to someone.

A few words about the implementation: the project was written using CodeIgniter and the letters are sent by the standard Email library. However, it does not allow you to create your own headers when sending emails via SMTP, therefore, you need to extend it with your own class. Taking the github.com/leonbarrett/CodeIgniter-Sendgrid-API/blob/master/system/libraries/Email.php library as an example, we made our own library, discarding everything unnecessary and simplifying the rest to one method :). In fact, there are 2 methods, because the _set_header () method is declared private in the CI_Email class, which does not allow its use in successor classes.

Thus, adding unique arguments and categories to a letter is done by a single call to the addSendGridHeader () method, for example:

 $this->email->addSendGridHeader(array('unique_args'=>array('email_id'=>1), 'category'=>array('help', 'question'))); 

However, back to the hooks, the second hook, Inbound Parse , allows you to receive a response from users. After an appropriate MX setup of a domain or subdomain record, Sendgrid starts sending the processed response letter to the specified address. To our surprise, it turned out that there are no events for incoming emails, i.e. get information about what stage of processing it is now impossible. One can only hope that this process is debugged and does not fail, i.e. everything that they received is transmitted to us, it is still impossible to verify something. Proceeding from this, we can only record the stages of processing letters that occur on our side. We have the following statuses of incoming letters: 'received', 'hash_checked', 'duplicate', 'processed'. Surely you are interested in the second status of 'hash_checked', in any case, I hope that this has happened, because now I will talk about it. Besides the fact that we do not receive any events about incoming letters from SendGrid, we also receive nothing at all, which would help to identify the initial letter. No unique arguments, no categories, nothing at all. Those. to understand which letter we received the answer is simply impossible. But it would be great if SendGrid took on the task of linking sent and received letters to itself - we send unique headers with the sent letter and they are returned to us along with the answer. This is part of the same tilled field, which I wrote about in a previous post. But since this is not, you have to get out on your own.

We went by using a special hash in the address for the "Reply-To" header, it looks like:

 "db051171af45b683c50eb3d66017ecf2+@incoming.servicedomain.com" 

This hash is generated in such a way that we know exactly which record in the system it corresponds to. This could be a Todo recording, a discussion or something else. Upon receipt of the letter, we determine the record to which the letter belongs, and if it was determined, the letter is assigned the status of 'hash_checked'. Next, we check if the letter is a duplicate so as not to create several identical comments. Then a new comment is actually created and the letter is assigned the status of 'processed'.

One of the tasks that we wanted to solve was the ability to resend emails that were not delivered to users. To do this, all the necessary data is stored in the database. Since the project is written on CodeIgniter, such data is the body of the letter and the headers formed before sending, which include subject, recipients, our unique parameters and so on. The saving of letters was implemented literally in a few lines of code when overriding the send () method of the standard Email library. So, the letters are saved, but there is no point in storing the original data of the letters that were safely delivered, right? In any case, it is not for us and therefore we made small changes to the script that receives events from SendGrida - if the notification about the event is 'delivered', then we delete all records corresponding to this letter from the table. Here, the multiplying number was used not by chance, to speed up the process of saving letters to the database, we do not check whether this is the first attempt to send it or not, i.e. in the case of several attempts to send the same letter, several copies of it will be saved in the database. The logical conclusion was the addition of an action for re-sending letters from the database, this did not cause any difficulties, although it took a few changes to the above-mentioned library to work with letters. If it is interesting, write in the comments and I will tell which ones.

Here, in general terms, is all about how we control our sending and receiving mail. Having launched this new subsystem and a sub-accumulation of data, we will begin to optimize it and automate those cases where the solution is understood by us and programmed. If you have already solved a similar problem and share your experience in the comments - you will be very grateful to you.

PS With the release of the automation system described above, we are faced with an unpleasant situation. Our Production and Pre-production versions of the system use different accounts in SendGrid to ensure that test letters do not affect the statistics of the real system. On September 15, we tried to launch changes on the Production server and it turned out that for some reason we could not get data from the sent event notifications. A short investigation revealed the reason - on September 6, SendGrid launched the third version of the Event webhook, in which the format of the transmitted data was drastically changed. In versions 1 and 2 these were ordinary POST variables, now a JSON structure with an array of records is sent. It is rather strange that SendGrid did not warn its paid clients about such significant changes. According to the law of meanness, the Event Notification application in the account for Pre-production was activated until September 6th and, accordingly, we tested it on version 2 of the hook. Before the release of the 15th, we activated the notifications in the account used by the live system, and we were silently connected to version 3 of the hook. Thus, we had to postpone the release in order to change and test the notification parser.