What should we build an email marketing service? An inside look, part two
How difficult is it to build a full-fledged email marketing service? What do you need to foresee? What pitfalls can meet on the way of inquiring minds of developers?
Let's try to figure it out together. Within the framework of several articles, I will talk about how I have been doing my own email-mailing service for more than a year, what lessons I have learned for myself and what I plan to do with all this further.
')
At once I will make a reservation that the article deals only with the technical side of the issue.
The first part can be read here .
I have been writing in Python for 5 years, I mainly use Django, PostgreSQL, I can write JavaScript at the jQuery + KnockoutJS level, sometimes I write on React. In my free time, I do freelance on UpWork and my own Internet projects, one of which I now plan to tell about. I have been involved in this project for about a year and a half.
In the first part, I briefly talked about what technologies I use as core in my product (Python, Django, PostgreSQL), how the tracking of letters works, and also brought some statistics about the service.
1. How did I manage to relieve users of pain using asynchronous tasks. Consider performing multiple tasks simultaneously using Celery .
2. How I organized the work with tables containing several million records in a very resource-constrained server. I'll tell you why sometimes denormalized tables are good.
3. How I managed to quickly and with minimal time costs migrate some of the functionality of the monolithic Django project into separate microservices and what technologies I used for this.
So let's get started.
Sooner or later, all maturing projects are faced with performance problems, especially if we are talking about web applications that process requests from users' browsers in synchronous mode.
Suppose you are using a technology stack typical for a python project: a fresh build of uwsgi, nginx as a web server + your python application. Nginx gives static files and sends the root to uwsgi. In the uwsgi configuration, you have N workers.
All this will work well smoothly until you encounter a large number of "heavy" requests (loading large files, for example), which will require considerable time, close or more than harakiri uwsgi, for processing. After that, the uwsgi workers will not have time to process requests from users, and users will begin to receive errors from nginx and 502 and swear in the reviews about your "service".
Of course, you can resort to solutions like tornado, asyncio + aiohttp, but this is a little about something else, because we already have a synchronous Django application, which is a very bad thing tothrow out a rewrite.
An inquisitive reader will say that you can combine Django with aiohttp using multiple upstream for nginx, but now I want to consider a more standard approach.
Today, the most standard approach in synchronous python projects is to use a library to handle asynchronous Celery tasks.
All information below is valid for version 3.x, because I use it exactly. In version 4 of Celery, a lot of things have changed.
For those readers who are still unfamiliar with this library, a brief remark that it can:
1. You can create tasks (which are normal functions) that Celery will perform asynchronously (or run at the right time with beat).
An example of the simplest task that sends a certain system email:
This task is invoked as follows:
2. If you use Django, you will get a fairly convenient interface for managing these tasks, which will allow you to determine, for example, the launch parameters for each task at runtime.
3. This entire system most often works on the basis of RabbitMQ (used as a message broker) and, most often, is controlled by the supervisord. Also, a database can be used as a broker (the slowest option, but quite suitable for development), Redis (Celery authors do not advise using it, since the protocol is very difficult to maintain), Amazon SQS.
My project provides users with email and transactional mail services.
Accordingly, potentially we have 2 types of asynchronous tasks:
1. Long tasks. Mailings can be up to 20-25 thousand letters, carried out within 6-7 hours because of the necessary timeouts when sending letters.
2. Short tasks. The user has registered on the site, whose backend sent a request to send an email with registration confirmation to the API of my service and within a few seconds (or better - faster), the user should see this email in his inbox.
If you plan on scheduling and run all tasks within one queue (that is, by default), all short tasks, if there is already a long task in progress, will wait for it to complete.
It is clear that receiving transactional emails after 6-7 hours is not an option, so we come to using several queues for parallel processing of tasks.
To do this, configure the queues themselves in our project:
In my project I use three queues:
1. Transactional - the queue for sending transactional letters
2. Bulk - queue to send mailings
3. Celery - the queue for the remaining tasks.
Accordingly, in supervisor.d, the configuration takes the following form:
It is clear that many configuration options are omitted here. The bottom line is that every queue needs to start its process at the supervisor.
Task setting to a specific queue occurs as follows:
As a result, we get simultaneous processing of several tasks, users seem to be happy, the service works fairly quickly, and the uwsgi workers are free.
View the status of tasks is quite simple:
You can also use the Flower solution for more convenient monitoring.
I want to note that I do not consider myself to be a super expert at Celery and will be happy if readers share their experiences using it.
Sooner or later you will accumulate data. And it will be necessary to work with them somehow.
My service processes about 400,000 emails per month (this is very small), while I clean the data about emails older than 90 days, and still have about 2 million table entries that contain tracking data.
These data must be shown in statistics, you need to work with them, you need to make analytics on them in order to identify spammers, etc.
In the beginning, my tables were super-normalized, built according to all the canons of SQL. However, the reality is that ORM Django often makes life very difficult for you and your project due to the rather complex JOINs between your tables.
Plus, it is not always obvious how many queries occur when iterating a queryset, especially for novice developers.
As a result, to find a fairly simple query:
It took me about 100 seconds. Awful time, isn't it? This happened for several reasons:
1. First, never do this: AND id IN (...) , if you can have about 100,000 elements in an array :)
2. Secondly, it is much better to write d.date = me.date_created :: date than d.date = to_char (date_trunc ('day', me.date_created), 'YYYY-MM-DD') .
3. Thirdly, in my table there were foreign keys for several other keys. Because of what JOINs worked very slowly.
1. I added fields for explicit values of related records in addition to foreign keys (partially denormalize tables).
2. Rewrote the query:
As a result, the processing time of this request was reduced to 3 seconds, which is already quite tolerable. Explain:
First, the table takes up more disk space. Secondly, the string values in the denormalized table should be relevant. Those. if you change the string identifier in the parent table, you must change it in the child denormalized table. Fortunately, Django migrations make it easy to implement.
One of the problems of growing up of any more or less large project is that it is becoming harder and harder to maintain. The code base is growing, the number of tests is growing. Plus, if you make a mistake in one part of the project, others may suffer because of it. For example, a random SyntaxError will fill up the entire project.
Also, performance problems associated with the heavy queries I have already described are becoming increasingly relevant.
The solution is to separate part of the functionality into a separate microservice, i.e .:
1. To carry out the code base and tests related to this functionality in a separate project.
2. Create a separate uwsgi instance for it and a separate upstream or server in nginx.
3. Link it with a broker like RabbitMQ or HTTP API with the main project.
...
4. PROFIT!
1. There is no more danger to overwhelm a project due to an error in one of its functional parts.
2. You can reuse it in other projects. Especially important for all kinds of links shorteners, image compressors, and similar application microservices.
3. You can update service dependencies independently. For example, if you are using the legacy version of Django in the main project, you can use newer versions in microservices without having to adapt a significant code base.
In my case, I needed to implement a simple admin panel to manage my project. It had to work separately, due to the fact that it has quite heavy reports on the use of the service by my users, and I did not want to hammer in the uwsgi workers of the main project.
1. Django Rest Framework (hereinafter referred to as DRF) for building the HTTP API within the framework of the main project.
2. For writing the client administration system - ReactJS, NodeJS, Babel, ES6 and other incomprehensible words from the world of the frontend.
DRF allows you to very quickly build convenient REST API interfaces within existing projects. In my case, I needed to build an interface that would satisfy the following requirements:
1. Access to API handles must be secure. Only certain users with administrative rights must have access to the administration system.
2. The interface should provide a convenient way of interaction for the client SPA application.
3. Pens should work out of the box to get lists of objects, information on a specific object, delete objects, etc. In general, the API should be sufficient for building a normal CRUD interface on the client side.
To authorize requests from the client, I used a ready-made solution from DRF - Token authentication.
All that needed to be done to enable it:
1. Add 'rest_framework.authtoken' to INSTALLED_APPS.
2. Start the migration.
3. Create tokens for users who should have access to the administration system.
Next, we create a general mixin, which we will mix into the viewsets we need:
Then we can simply use this mixin to connect authorization to our ViewSet:
To interact with the API from the client side, the axios library was perfect .
An example of a request to the API:
I’ll not give the code for the components of the React application, since it's pretty standard.
In the following articles I will talk about how to painlessly organize logging in a large project based on logstash, kibana and elasticsearch, as well as touch on client documentation and support based on HelpScout and GitBook solutions.
Thanks for attention!
Let's try to figure it out together. Within the framework of several articles, I will talk about how I have been doing my own email-mailing service for more than a year, what lessons I have learned for myself and what I plan to do with all this further.
')
At once I will make a reservation that the article deals only with the technical side of the issue.
The first part can be read here .
Briefly about yourself
I have been writing in Python for 5 years, I mainly use Django, PostgreSQL, I can write JavaScript at the jQuery + KnockoutJS level, sometimes I write on React. In my free time, I do freelance on UpWork and my own Internet projects, one of which I now plan to tell about. I have been involved in this project for about a year and a half.
What is this article about?
In the first part, I briefly talked about what technologies I use as core in my product (Python, Django, PostgreSQL), how the tracking of letters works, and also brought some statistics about the service.
In this part I will tell:
1. How did I manage to relieve users of pain using asynchronous tasks. Consider performing multiple tasks simultaneously using Celery .
2. How I organized the work with tables containing several million records in a very resource-constrained server. I'll tell you why sometimes denormalized tables are good.
3. How I managed to quickly and with minimal time costs migrate some of the functionality of the monolithic Django project into separate microservices and what technologies I used for this.
So let's get started.
Asynchronous tasks in python projects, or why your user should not wait
Sooner or later, all maturing projects are faced with performance problems, especially if we are talking about web applications that process requests from users' browsers in synchronous mode.
Simplest example
Suppose you are using a technology stack typical for a python project: a fresh build of uwsgi, nginx as a web server + your python application. Nginx gives static files and sends the root to uwsgi. In the uwsgi configuration, you have N workers.
All this will work well smoothly until you encounter a large number of "heavy" requests (loading large files, for example), which will require considerable time, close or more than harakiri uwsgi, for processing. After that, the uwsgi workers will not have time to process requests from users, and users will begin to receive errors from nginx and 502 and swear in the reviews about your "service".
Of course, you can resort to solutions like tornado, asyncio + aiohttp, but this is a little about something else, because we already have a synchronous Django application, which is a very bad thing to
An inquisitive reader will say that you can combine Django with aiohttp using multiple upstream for nginx, but now I want to consider a more standard approach.
Today, the most standard approach in synchronous python projects is to use a library to handle asynchronous Celery tasks.
All information below is valid for version 3.x, because I use it exactly. In version 4 of Celery, a lot of things have changed.
For those readers who are still unfamiliar with this library, a brief remark that it can:
1. You can create tasks (which are normal functions) that Celery will perform asynchronously (or run at the right time with beat).
An example of the simplest task that sends a certain system email:
from celery import current_app ... @current_app.task() def send_service_message(subject, recipients, template, html_template, context): html_mail( subject=subject, template=template, html_template=html_template, recipients=recipients, context=context ) This task is invoked as follows:
from app.message.tasks import send_service_message ... send_service_message.delay( _(u'%s: ' % subscriber.list.project.domain), [user.email], '', 'site/subscriber/email/subscription_notification.html', context ) 2. If you use Django, you will get a fairly convenient interface for managing these tasks, which will allow you to determine, for example, the launch parameters for each task at runtime.
3. This entire system most often works on the basis of RabbitMQ (used as a message broker) and, most often, is controlled by the supervisord. Also, a database can be used as a broker (the slowest option, but quite suitable for development), Redis (Celery authors do not advise using it, since the protocol is very difficult to maintain), Amazon SQS.
An example of use in my project
My project provides users with email and transactional mail services.
Accordingly, potentially we have 2 types of asynchronous tasks:
1. Long tasks. Mailings can be up to 20-25 thousand letters, carried out within 6-7 hours because of the necessary timeouts when sending letters.
2. Short tasks. The user has registered on the site, whose backend sent a request to send an email with registration confirmation to the API of my service and within a few seconds (or better - faster), the user should see this email in his inbox.
If you plan on scheduling and run all tasks within one queue (that is, by default), all short tasks, if there is already a long task in progress, will wait for it to complete.
It is clear that receiving transactional emails after 6-7 hours is not an option, so we come to using several queues for parallel processing of tasks.
To do this, configure the queues themselves in our project:
# settings.py ... CELERY_QUEUES = ( Queue('celery', Exchange('celery'), routing_key='celery'), Queue('bulk', Exchange('bulk'), routing_key='bulk'), Queue('transactional', Exchange('transactional'), routing_key='transactional'), ) CELERY_ROUTES = { 'app.campaign.tasks.start_campaign': { 'queue': 'bulk' }, 'app.message.tasks.send_service_message': { 'queue': 'transactional' }, 'app.message.api.tasks.send_message': { 'queue': 'transactional' } } In my project I use three queues:
1. Transactional - the queue for sending transactional letters
2. Bulk - queue to send mailings
3. Celery - the queue for the remaining tasks.
Accordingly, in supervisor.d, the configuration takes the following form:
[program:celery] command=<PROJECT_ROOT>/venv/bin/celery worker -A conf.celery -l INFO --concurrency=4 --pidfile=/var/run/celery/celery.pid -Q celery ... [program:bulk] command=<PROJECT_ROOT>/venv/bin/celery worker -A conf.celery -l INFO --concurrency=4 --pidfile=/var/run/celery/bulk.pid -Q bulk ... [program:transactional] command=<PROJECT_ROOT>/venv/bin/celery worker -A conf.celery -l INFO --concurrency=4 --pidfile=/var/run/celery/transactional.pid -Q transactional ... It is clear that many configuration options are omitted here. The bottom line is that every queue needs to start its process at the supervisor.
Task setting to a specific queue occurs as follows:
send_message.apply_async(kwargs={'mailer': self, 'message': message}, queue='transactional') As a result, we get simultaneous processing of several tasks, users seem to be happy, the service works fairly quickly, and the uwsgi workers are free.
Task status tracking
View the status of tasks is quite simple:
celery -A conf.celery inspect scheduled celery -A conf.celery inspect active celery -A conf.celery inspect reserved You can also use the Flower solution for more convenient monitoring.
I want to note that I do not consider myself to be a super expert at Celery and will be happy if readers share their experiences using it.
Denormalization of large tables - for and against
Sooner or later you will accumulate data. And it will be necessary to work with them somehow.
My service processes about 400,000 emails per month (this is very small), while I clean the data about emails older than 90 days, and still have about 2 million table entries that contain tracking data.
These data must be shown in statistics, you need to work with them, you need to make analytics on them in order to identify spammers, etc.
In the beginning, my tables were super-normalized, built according to all the canons of SQL. However, the reality is that ORM Django often makes life very difficult for you and your project due to the rather complex JOINs between your tables.
Plus, it is not always obvious how many queries occur when iterating a queryset, especially for novice developers.
As a result, to find a fairly simple query:
SELECT d.date, SUM(CASE WHEN me.status = 'SENT' THEN 1 ELSE 0 END) AS sent_count, SUM(CASE WHEN me.status = 'OPENED' THEN 1 ELSE 0 END) AS opened_count, SUM(CASE WHEN me.status = 'REDIRECTED' THEN 1 ELSE 0 END) AS redirected_count, SUM(CASE WHEN me.status = 'HARDBOUNCED' THEN 1 ELSE 0 END) AS hardbounced_count, SUM(CASE WHEN me.status = 'SOFTBOUNCED' THEN 1 ELSE 0 END) AS softbounced_count, SUM(CASE WHEN me.status = 'UNSUBSCRIBED' THEN 1 ELSE 0 END) AS unsubscribed_count FROM ( SELECT TO_CHAR(date_trunc('day', ('2017-03-01'::DATE - offs)), 'YYYY-MM-DD') AS date FROM generate_series(0, 30, 1) AS offs ) d LEFT OUTER JOIN MESSAGE_MESSAGEEVENT me ON (d.date=to_char(date_trunc('day', me.date_created), 'YYYY-MM-DD') AND status IN ('SENT', 'OPENED', 'REDIRECTED', 'HARDBOUNCED', 'SOFTBOUNCED', 'UNSUBSCRIBED') AND id IN (...)) GROUP BY date ORDER BY date It took me about 100 seconds. Awful time, isn't it? This happened for several reasons:
1. First, never do this: AND id IN (...) , if you can have about 100,000 elements in an array :)
2. Secondly, it is much better to write d.date = me.date_created :: date than d.date = to_char (date_trunc ('day', me.date_created), 'YYYY-MM-DD') .
3. Thirdly, in my table there were foreign keys for several other keys. Because of what JOINs worked very slowly.
What I came to
1. I added fields for explicit values of related records in addition to foreign keys (partially denormalize tables).
2. Rewrote the query:
SELECT d.date, SUM(CASE WHEN me.status = 'SENT' THEN 1 ELSE 0 END) AS sent, SUM(CASE WHEN me.status = 'OPENED' THEN 1 ELSE 0 END) AS opened, SUM(CASE WHEN me.status = 'HARDBOUNCED' THEN 1 ELSE 0 END) AS hardbounced, SUM(CASE WHEN me.status = 'SOFTBOUNCED' THEN 1 ELSE 0 END) AS softbounced, SUM(CASE WHEN me.status = 'UNSUBSCRIBED' THEN 1 ELSE 0 END) AS unsubscribed, SUM(CASE WHEN me.status = 'REDIRECTED' THEN 1 ELSE 0 END) AS redirected FROM ( SELECT ('2017-03-01'::DATE - offs)::date AS date FROM generate_series(0, 30, 1) AS offs ) d LEFT OUTER JOIN MESSAGE_MESSAGEEVENT me ON ( d.date=me.date_created::date AND me.date_created >= '2017-02-01 00:00:00' AND me.date_created <= '2017-03-01 23:59:59' AND true = true AND me.project_id = ... ) GROUP BY date ORDER BY date; As a result, the processing time of this request was reduced to 3 seconds, which is already quite tolerable. Explain:
GroupAggregate (cost=61552.47..82232.11 rows=200 width=11) (actual time=2445.092..2986.524 rows=31 loops=1) -> Merge Left Join (cost=61552.47..73614.51 rows=491920 width=11) (actual time=2445.048..2857.069 rows=69314 loops=1) Merge Cond: ((('2017-03-01'::date - offs.offs)) = ((me.date_created)::date)) -> Sort (cost=62.33..64.83 rows=1000 width=4) (actual time=0.127..0.174 rows=31 loops=1) Sort Key: (('2017-03-01'::date - offs.offs)) Sort Method: quicksort Memory: 26kB -> Function Scan on generate_series offs (cost=0.00..12.50 rows=1000 width=4) (actual time=0.034..0.069 rows=31 loops=1) -> Materialize (cost=61490.14..62719.94 rows=98384 width=15) (actual time=2444.913..2695.888 rows=69312 loops=1) -> Sort (cost=61490.14..61736.10 rows=98384 width=15) (actual time=2444.908..2539.290 rows=69312 loops=1) Sort Key: ((me.date_created)::date) Sort Method: external merge Disk: 2152kB -> Bitmap Heap Scan on message_messageevent me (cost=11442.78..51647.59 rows=98384 width=15) (actual time=1922.446..2253.982 rows=69312 loops=1) Recheck Cond: (project_id = ...) Filter: ((date_created >= '2017-02-01 00:00:00+01'::timestamp with time zone) AND (date_created <= '2017-03-01 23:59:59+01'::timestamp with time zone)) -> Bitmap Index Scan on message_messageevent_b098ad43 (cost=0.00..11418.19 rows=236446 width=0) (actual time=1870.742..1870.742 rows=284445 loops=1) Index Cond: (project_id = ...) Total runtime: 2988.107 ms What are the disadvantages of this solution?
First, the table takes up more disk space. Secondly, the string values in the denormalized table should be relevant. Those. if you change the string identifier in the parent table, you must change it in the child denormalized table. Fortunately, Django migrations make it easy to implement.
Migration of the functionality of a monolithic project to microservices
One of the problems of growing up of any more or less large project is that it is becoming harder and harder to maintain. The code base is growing, the number of tests is growing. Plus, if you make a mistake in one part of the project, others may suffer because of it. For example, a random SyntaxError will fill up the entire project.
Also, performance problems associated with the heavy queries I have already described are becoming increasingly relevant.
The solution is to separate part of the functionality into a separate microservice, i.e .:
1. To carry out the code base and tests related to this functionality in a separate project.
2. Create a separate uwsgi instance for it and a separate upstream or server in nginx.
3. Link it with a broker like RabbitMQ or HTTP API with the main project.
...
4. PROFIT!
Advantages of microservice architecture:
1. There is no more danger to overwhelm a project due to an error in one of its functional parts.
2. You can reuse it in other projects. Especially important for all kinds of links shorteners, image compressors, and similar application microservices.
3. You can update service dependencies independently. For example, if you are using the legacy version of Django in the main project, you can use newer versions in microservices without having to adapt a significant code base.
In my case, I needed to implement a simple admin panel to manage my project. It had to work separately, due to the fact that it has quite heavy reports on the use of the service by my users, and I did not want to hammer in the uwsgi workers of the main project.
Technologies that I used:
1. Django Rest Framework (hereinafter referred to as DRF) for building the HTTP API within the framework of the main project.
2. For writing the client administration system - ReactJS, NodeJS, Babel, ES6 and other incomprehensible words from the world of the frontend.
Build API
DRF allows you to very quickly build convenient REST API interfaces within existing projects. In my case, I needed to build an interface that would satisfy the following requirements:
1. Access to API handles must be secure. Only certain users with administrative rights must have access to the administration system.
2. The interface should provide a convenient way of interaction for the client SPA application.
3. Pens should work out of the box to get lists of objects, information on a specific object, delete objects, etc. In general, the API should be sufficient for building a normal CRUD interface on the client side.
Authorization
To authorize requests from the client, I used a ready-made solution from DRF - Token authentication.
All that needed to be done to enable it:
1. Add 'rest_framework.authtoken' to INSTALLED_APPS.
2. Start the migration.
3. Create tokens for users who should have access to the administration system.
Next, we create a general mixin, which we will mix into the viewsets we need:
# permissions.py from rest_framework.permissions import IsAdminUser class IsProjectAdminUser(IsAdminUser): def has_permission(self, request, view): return request.user.is_authenticated and request.user.is_admin # views.py from rest_framework.authentication import TokenAuthentication from app.contrib.api.permissions import IsProjectAdminUser class AdminAuthMixin(object): authentication_classes = [TokenAuthentication] permission_classes = [IsProjectAdminUser] Then we can simply use this mixin to connect authorization to our ViewSet:
... from app.contrib.api.views import AdminAuthMixin, MultiSerializerViewSet ... class CampaignsViewSet(AdminAuthMixin, MultiSerializerViewSet): queryset = Campaign.objects.filter(date_archived__isnull=True) filter_class = CampaignsFilter serializers = { 'list': CampaignsListSerializer, 'retrieve': CampaignDetailSerializer, } Customer interaction
To interact with the API from the client side, the axios library was perfect .
An example of a request to the API:
axios.get('http://example.com/api/entrypoint/', {params: this.state.query_params}) .then(response => { const items = response.data.results.map(obj => obj); this.setState({ is_loading: false, items: items, count: response.data.count }); }); I’ll not give the code for the components of the React application, since it's pretty standard.
PS
In the following articles I will talk about how to painlessly organize logging in a large project based on logstash, kibana and elasticsearch, as well as touch on client documentation and support based on HelpScout and GitBook solutions.
Thanks for attention!
Source: https://habr.com/ru/post/324174/
All Articles