Email - not a sparrow, fly out - you won't catch it or how we did email marketing with a machine
Over 40% of Lingualeo's revenue comes from an email channel. In 2015, we sent 300 million letters. It is extremely important for us that this sales channel work efficiently: predictable, scalable, and ROI mailing is maximized.
Today we will tell in general about our complicated email marketing system, about moving from one platform to another, about the localization system and about the technical nuances of moving. In this collectively prepared article ( Roman Domrachev , Backend Team Leader Lingualeo; Ivan Zelenin, ex-head of email marketing and CRM Lingualeo, now operational director of the cool CRM marketing agency WIM.Agency; Vladimir Vitkovsky , monetization manager of Lingualeo; Konstantin Shimanaev, a leading marketing manager for Lingualeo) there are interesting things for both marketers and techies!
')
In 2015, Lingualeo was faced with the need to move to a new platform and technology to send letters, as the current resource (the agency that worked with the ExpertSender platform) could no longer cope with our volumes and Wish lists. It was a good quality platform. But Lingualeo has grown out of it. Plus, it was important for us to control the entire e-mailing process: from design to dispatch. Therefore, we decided to transfer work to another platform and start preparing in-house mailings. We wanted to do email-marketing machine.
To select a new system, we formed the following criteria:
We opted for the Emarsys platform.
But choosing a platform based on the TK was a small fraction. The next step was integration with Lingualeo.
Any integration with the Email service provider (hereinafter - ESP) involves 3 steps:
In our case, Lingualeo also added automatic unloading of analytics from ESP, since it was important for us to understand everything that happens on the ESP side (sent letters, opening letters, clicks, non-delivery), and compare this with the actions of users on the site. Plus localization system. This is from the technical side.
With marketing it was required:
At the initial stage of integration, ExpertSender and Emarsys were connected in parallel on different mail subdomains.
Users are synchronized to both ESPs, and triggers are sent through one. To which provider the trigger belongs, configures the administrator. So we could gradually transfer the old triggers to Emarsys, and make new ones right away in it.
This scheme assumes that user synchronization should be done to all ESPs simultaneously. And also there should be an opportunity to send triggers to both systems, but send letters only from one. However, as an error protection, triggers should only be sent to one of the connected ESPs to prevent the same email from being sent again from another ESP. The administrator can manage the configuration: through what ESP, what type of letters to send.
Synchronize users and send triggers - requests to Emarsys. During synchronization, we send values ​​that are stored in the ESP database, and when sending triggers, we send the identifier and data from which the provider collects the letter.
Localization of templates - two tasks that run on the crown: one takes the templates from ESP, parses them and sends the segments for translation to WebTranslateIt (hereinafter WTI), and the second receives the translated segments from WTI and uploads them to ESP. More on localization below.
Requests for creating, updating users, and sending triggers are done via the RabbitMQ queue. The request from the site in ESP can be presented in two parts:
Sending requests from the site is done via emailing'a service. It sticks together the same type of messages in one request, so that the load on the ESP is less. Messages are sent through the queue, so we don’t have to keep the user waiting for ESP to respond.
While the application is running, we are dealing with user data, and each of the changes is repeated in ESP. The application code "subscribes" to save the user and sends the necessary data. But if you send a request with each save, we create a lot of small requests. This problem was solved in Lingualeo by redoing sending requests. Now, when changing user data, we do not send a request, but fold it into a “pocket”. When the http request is completed, we collect the requests that are in the “pocket” and process it so that as many changes of the same user turn into a single data update request in ESP.
The request is taken from the queue by the workers.
Workers are applications that get jobs from a queue and execute. The worker processes 1000 tasks, after which it dies and is restarted by the supervisor (supervisord.org).
Here the messages of the e-mailing service turn into requests that are clear to each of the providers.
In the queues you can not show the user the page “something amiss is going on in the jungle,” in case of an error, the worker must handle the situation on his own. In the queue there is no one to ask “what to do?”. It remains to fall with an error that is written to the log. The log handles Gralylog2 (www.graylog.org), which is able to send alerts to the mail and show message flows.
Logs save if the application crashes, but if it slows down, the logs do not help to understand what happened. Pinba (pinba.org) helps us detect such messages, to which we send application timers. We have a monitor in the center of the office that shows the site metrics from the data received by the pinboy.
Stages of preparation of letters:
We have a monthly mailing plan. The cycle of letter preparation takes an average week. Although periodically there are urgent letters, and we do them in 1 day. Based on the e-mail plan, the manager makes applications to jira for the design, prescribes the project, and at the same time sets the task of copywriting. As soon as copywriting is ready, we add texts to the task for designers in jira. If the designers are busy, the email manager uses a ready-made base of pictures and tries to “blind” the image with the hands based on the ready-made layout templates, so as not to draw from scratch, but to do a minimum of changes in the finished layout. In case the designers have prepared a unique template, the latter goes to the layout makers. The layout is very fast. A few hours - and we have already laid out the layout. Email-manager will check it in a specialized system (we have emailonacid) and tests how letters look in the mobile version and in different clients.
Layout has certain difficulties. Email layout is very specific. You can only use the old standards, you can not div, therefore, have to use only certain tags. Maker-ups do not really like email, because there are big restrictions and there is no possibility to use new technologies.
To get the most out of email marketing, it’s not enough to send 300 million letters a year, you need to send them as much as possible targeted (more letters to small segments) and personalized (having more specifics about the recipient in the body of the letter - name, gender, and even and recommended activities). Analytics show that this is the right step - personalized letters are converted 20% higher.
We synchronize two types of data in Emarsys - static (gender, age, etc.) and behavioral data. Static data is more suitable for filtering, and for deeper personalization we use additional fields in which the application loads data periodically. For example, a letter with user statistics on occupations is formed several hours before being sent in order to remain relevant when it reaches the user!
Triggers are an order for Emarsys to send an email immediately using the transferred data, or after the time specified in the control panel. Triggers can be used to create mailings yourself. To do this, you need to prepare a list of users and send a request to Emarsys for each of them with the necessary data. This method of mailing support more difficult than normal. But we can respond to user actions at the same moment when he performs these actions on the site.
An example of a trigger mailing is a motivational letter that a user receives after working with a dictionary. For example, a user adds words to his “Dictionary”, but does not train for the next 2 days. In this case, on the second day after the addition, he receives a motive letter with a reminder that he should learn these words. Letters of this kind have one of the best indicators of the involvement of users of the service.
One of the features of the work of Lingualeo is a wide geography. Our students live not only in Russia and the near abroad, but also in Brazil and Turkey. Therefore, our email communications should be localized.
It is very important that everyone leaves their letter. The localization system works as follows: we have a template, it is divided into certain lines that we want to translate. This text is transmitted to the system for translation - this is WTI. (More information about the localization system of the product Lingualeo in the post of Igor Lyubimov , Head of Localization Lingualeo.) In this system, translators get the lines, then "magic" happens. According to the results of “witchcraft,” the translation comes to Emarsys as templates. Next, we check by sending to localization managers: how truthful the letter is, and whether it is worth changing something visually.
The technical side of the localization “magic” is as follows: marketers create a master-template in Russian. Then this template is picked up by the localization service, and on its basis localized slave templates are created. The user's locale is stored in his card in Emarsys.
The task of localization is divided into two stages:
Both tasks are solved by the localization service: EmarsysLocalizationService. The launch of the localization system occurs every hour.
Algorithm for exporting segments from Emarsys to WTI:
Algorithm for importing segments from WTI to Emarsys:
- The content is saved in the slave template (the template is created if it was not or is updated if it is already
existed).
WTI-Emarsys interaction diagram:
To assess the effectiveness of mailings, in addition to the basic metrics (open rate, click rate, unsubsribe rate, etc.) available in the Emarsys office, we use a number of additional ones that are calculated based on the user's activity on the service after switching from the letter.
This is possible due to the fact that information about all shipments, discoveries of letters and following links from them is loaded into our analytical repository, where you can calculate any metrics and funnels in arbitrary sections. For each mailing you can build, for example, such a funnel:
Step 1. Number of shipments.
Step 2. The number of discoveries.
Step 3. The number of transitions from the letters.
Step 4. The number of users who received XP during the day after the transition from the letter.
Step 5. Number of purchases during the day + revenue from each shipment during the day after mailing.
All listed metrics can be arbitrarily segmented, for example, by gender, age, number of days from the moment of registration (newcomers / oldies), registration platform, etc. Based on this information, we optimize the list.
On the technical side:
On the business marketing side:
Today we will tell in general about our complicated email marketing system, about moving from one platform to another, about the localization system and about the technical nuances of moving. In this collectively prepared article ( Roman Domrachev , Backend Team Leader Lingualeo; Ivan Zelenin, ex-head of email marketing and CRM Lingualeo, now operational director of the cool CRM marketing agency WIM.Agency; Vladimir Vitkovsky , monetization manager of Lingualeo; Konstantin Shimanaev, a leading marketing manager for Lingualeo) there are interesting things for both marketers and techies!
')
In 2015, Lingualeo was faced with the need to move to a new platform and technology to send letters, as the current resource (the agency that worked with the ExpertSender platform) could no longer cope with our volumes and Wish lists. It was a good quality platform. But Lingualeo has grown out of it. Plus, it was important for us to control the entire e-mailing process: from design to dispatch. Therefore, we decided to transfer work to another platform and start preparing in-house mailings. We wanted to do email-marketing machine.
To select a new system, we formed the following criteria:
- cost per 1000 letters;
- the ability to work with different language locales;
- maximum deliverability to email-domains worldwide;
- convenience of the interface for administering a large number of email chains;
- the delivery time of large volumes of letters within a few hours (up to this we could send as much as 1-2 million per day, while it was necessary to ask the agency to do manual “twists” in the system);
- detailed analytics (we wanted to know what is happening with our letters, what actions customers take in relation to them, but were not ready to build an analytical system on their side);
- uptime> 99.5%;
- easy integration with the Lingualeo backend;
- Russian-language technical support and its physical presence in Russia, including technical personnel.
We opted for the Emarsys platform.
But choosing a platform based on the TK was a small fraction. The next step was integration with Lingualeo.
Integration scheme
Any integration with the Email service provider (hereinafter - ESP) involves 3 steps:
- regular synchronization of the contact database using CSV uploads;
- instant synchronization using the API .;
- setting up API triggers to send events.
In our case, Lingualeo also added automatic unloading of analytics from ESP, since it was important for us to understand everything that happens on the ESP side (sent letters, opening letters, clicks, non-delivery), and compare this with the actions of users on the site. Plus localization system. This is from the technical side.
With marketing it was required:
- Customize the logic of the triggers and chains of letters;
- Update, test and roll out the design and layout of letters;
- Track deliverability, adjust postmasters and ensure smooth warm-up;
- Update analytics format and increase its semantic significance.
At the initial stage of integration, ExpertSender and Emarsys were connected in parallel on different mail subdomains.
Users are synchronized to both ESPs, and triggers are sent through one. To which provider the trigger belongs, configures the administrator. So we could gradually transfer the old triggers to Emarsys, and make new ones right away in it.
This scheme assumes that user synchronization should be done to all ESPs simultaneously. And also there should be an opportunity to send triggers to both systems, but send letters only from one. However, as an error protection, triggers should only be sent to one of the connected ESPs to prevent the same email from being sent again from another ESP. The administrator can manage the configuration: through what ESP, what type of letters to send.
Synchronize users and send triggers - requests to Emarsys. During synchronization, we send values ​​that are stored in the ESP database, and when sending triggers, we send the identifier and data from which the provider collects the letter.
Localization of templates - two tasks that run on the crown: one takes the templates from ESP, parses them and sends the segments for translation to WebTranslateIt (hereinafter WTI), and the second receives the translated segments from WTI and uploads them to ESP. More on localization below.
Requests for creating, updating users, and sending triggers are done via the RabbitMQ queue. The request from the site in ESP can be presented in two parts:
- request hit the queue;
- processing and execution of the request.
Send request to queue
Sending requests from the site is done via emailing'a service. It sticks together the same type of messages in one request, so that the load on the ESP is less. Messages are sent through the queue, so we don’t have to keep the user waiting for ESP to respond.
While the application is running, we are dealing with user data, and each of the changes is repeated in ESP. The application code "subscribes" to save the user and sends the necessary data. But if you send a request with each save, we create a lot of small requests. This problem was solved in Lingualeo by redoing sending requests. Now, when changing user data, we do not send a request, but fold it into a “pocket”. When the http request is completed, we collect the requests that are in the “pocket” and process it so that as many changes of the same user turn into a single data update request in ESP.
Processing and execution of the request
The request is taken from the queue by the workers.
Workers are applications that get jobs from a queue and execute. The worker processes 1000 tasks, after which it dies and is restarted by the supervisor (supervisord.org).
Here the messages of the e-mailing service turn into requests that are clear to each of the providers.
In the queues you can not show the user the page “something amiss is going on in the jungle,” in case of an error, the worker must handle the situation on his own. In the queue there is no one to ask “what to do?”. It remains to fall with an error that is written to the log. The log handles Gralylog2 (www.graylog.org), which is able to send alerts to the mail and show message flows.
Logs save if the application crashes, but if it slows down, the logs do not help to understand what happened. Pinba (pinba.org) helps us detect such messages, to which we send application timers. We have a monitor in the center of the office that shows the site metrics from the data received by the pinboy.
Letter preparation
Stages of preparation of letters:
- forming an idea
- lock-up letters,
- copywriting
- design,
- layout,
- typesetting testing,
- personalization and customization,
- testing,
- localization.
Idea, mock-up, copywriting, design and layout
We have a monthly mailing plan. The cycle of letter preparation takes an average week. Although periodically there are urgent letters, and we do them in 1 day. Based on the e-mail plan, the manager makes applications to jira for the design, prescribes the project, and at the same time sets the task of copywriting. As soon as copywriting is ready, we add texts to the task for designers in jira. If the designers are busy, the email manager uses a ready-made base of pictures and tries to “blind” the image with the hands based on the ready-made layout templates, so as not to draw from scratch, but to do a minimum of changes in the finished layout. In case the designers have prepared a unique template, the latter goes to the layout makers. The layout is very fast. A few hours - and we have already laid out the layout. Email-manager will check it in a specialized system (we have emailonacid) and tests how letters look in the mobile version and in different clients.
Layout has certain difficulties. Email layout is very specific. You can only use the old standards, you can not div, therefore, have to use only certain tags. Maker-ups do not really like email, because there are big restrictions and there is no possibility to use new technologies.
Personalization
To get the most out of email marketing, it’s not enough to send 300 million letters a year, you need to send them as much as possible targeted (more letters to small segments) and personalized (having more specifics about the recipient in the body of the letter - name, gender, and even and recommended activities). Analytics show that this is the right step - personalized letters are converted 20% higher.
Mailings to user groups
We synchronize two types of data in Emarsys - static (gender, age, etc.) and behavioral data. Static data is more suitable for filtering, and for deeper personalization we use additional fields in which the application loads data periodically. For example, a letter with user statistics on occupations is formed several hours before being sent in order to remain relevant when it reaches the user!
Trigger mailings
Triggers are an order for Emarsys to send an email immediately using the transferred data, or after the time specified in the control panel. Triggers can be used to create mailings yourself. To do this, you need to prepare a list of users and send a request to Emarsys for each of them with the necessary data. This method of mailing support more difficult than normal. But we can respond to user actions at the same moment when he performs these actions on the site.
An example of a trigger mailing is a motivational letter that a user receives after working with a dictionary. For example, a user adds words to his “Dictionary”, but does not train for the next 2 days. In this case, on the second day after the addition, he receives a motive letter with a reminder that he should learn these words. Letters of this kind have one of the best indicators of the involvement of users of the service.
Localization
One of the features of the work of Lingualeo is a wide geography. Our students live not only in Russia and the near abroad, but also in Brazil and Turkey. Therefore, our email communications should be localized.
It is very important that everyone leaves their letter. The localization system works as follows: we have a template, it is divided into certain lines that we want to translate. This text is transmitted to the system for translation - this is WTI. (More information about the localization system of the product Lingualeo in the post of Igor Lyubimov , Head of Localization Lingualeo.) In this system, translators get the lines, then "magic" happens. According to the results of “witchcraft,” the translation comes to Emarsys as templates. Next, we check by sending to localization managers: how truthful the letter is, and whether it is worth changing something visually.
The technical side of the localization “magic” is as follows: marketers create a master-template in Russian. Then this template is picked up by the localization service, and on its basis localized slave templates are created. The user's locale is stored in his card in Emarsys.
The task of localization is divided into two stages:
- Parsing master templates and exporting segments to WTI.
- Retrieving translations from WTI and creating / updating localized slave templates.
Both tasks are solved by the localization service: EmarsysLocalizationService. The launch of the localization system occurs every hour.
Algorithm for exporting segments from Emarsys to WTI:
- All templates come from Emarsys.
- The system filters out only master templates.
- For each master template its content is requested from Emarsys.
- Segments are parsed from the template content (key = value of the data-translate-segment attribute, value = content inside the tag).
- A set of segments from a template is sent to WTI (each template has its own file in WTI).
Algorithm for importing segments from WTI to Emarsys:
- All data for the Lingualeo-Email-Templates project is requested from WTI.
- All templates come from Emarsys.
- The system filters out only master templates.
- For each master template its content is requested from Emarsys:
- From the WTI project, information about the WTI master file is extracted that matches the current template.
- For each enabled locale, segments are requested from WTI (by the ID of the wti-master file).
- The content is saved in the slave template (the template is created if it was not or is updated if it is already
existed).
WTI-Emarsys interaction diagram:
ANALYTICS
To assess the effectiveness of mailings, in addition to the basic metrics (open rate, click rate, unsubsribe rate, etc.) available in the Emarsys office, we use a number of additional ones that are calculated based on the user's activity on the service after switching from the letter.
This is possible due to the fact that information about all shipments, discoveries of letters and following links from them is loaded into our analytical repository, where you can calculate any metrics and funnels in arbitrary sections. For each mailing you can build, for example, such a funnel:
Step 1. Number of shipments.
Step 2. The number of discoveries.
Step 3. The number of transitions from the letters.
Step 4. The number of users who received XP during the day after the transition from the letter.
Step 5. Number of purchases during the day + revenue from each shipment during the day after mailing.
All listed metrics can be arbitrarily segmented, for example, by gender, age, number of days from the moment of registration (newcomers / oldies), registration platform, etc. Based on this information, we optimize the list.
TOTAL
On the technical side:
- Increased the possible volume of sent messages to 45 million monthly and more;
- Received the ability to send up to 20 million letters per day;
- The delivery time of 2 million messages has been reduced from days to 1-2 hours and is limited only by the ability to receive a certain number of students on the site;
- The process of preparing letters for different locales simplified and began to occupy at least half the time and resources;
- To introduce new triggers, intervention from the backend development team is less necessary.
On the business marketing side:
- Access to advanced analytics, including a list of specific actions for each email campaign;
- Accelerated the process of creating and sending an urgent letter from a week to one day if necessary;
- Accelerated the process of implementing a regular letter from two weeks to a week;
- Improved the quality of letters by involving internal designers in the process of creating letters and control on their part;
- Significantly reduced the cost of sending a letter.
- Tens of percent improved product and marketing metrics related to emailing (and we still have work to do)
Source: https://habr.com/ru/post/302268/
All Articles