Trigger mailings
Lately, email marketing is increasingly using automatic mailings to specific groups of consumers. Typical tasks:
In this article, we will describe how we solved this task - from writing each individual mailing by the developer from scratch 3 years ago, to the institution being sent by the manager via the web interface at the present time. The story may be interesting not only to those involved in email marketing, but also to everyone who has to implement the periodic execution of complex operations on certain consumer samples (I know that sounds very abstract, but in the end we had to solve just this abstract problem).
The answer to the first question was obvious for us: information is stored in our system about all significant actions performed by the consumer (logging in to the site, changing personal data) or above it (drawing a prize, sending a notification). In addition, we use actions for a variety of technical notes of consumers. So when sending an automatic mailing, we also decided to give the consumer a special action-marker , as a mark, that this automatic mailing had already been sent to him. In order not to re-send the newsletter, the condition “the consumer has no token action” is always added to the distribution condition.
On the second question, we stuffed a lot of cones associated with locks in the database, and eventually came up with the following pattern:
Of course, after the implementation of several mailings on this template, we decided to render the template code. For this, the BatchMailing class was created, and for each new mailing we created and registered it in a special registry of its successor. In the successor, it was necessary to overload the following properties and methods:
The property and the first two methods never caused any problems, but compiling the Expression was not quite easy. This Expression was used twice - first in the Read Uncommitted request to pull the consumer Id, and then in the Serializable transaction, to re-check whether the consumer fits the condition. You had to write it in such a way that Linq to SQL could translate it into T-SQL. Conditions could be quite complicated and they always had problems. Not one newsletter could not start without writing a handful of tests on it. In addition, for sending SMS and email, we have got different intermediate heirs from BatchMailing. When we had to send both email and SMS, we had to copy-paste. I had ideas on how to fix this, but since automatic mailings were not so often requested by customers, this was a low-priority task.
')
In our situation:
Since, apart from sending mailings, this entity can now perform any arbitrary operations on the consumer, it is incorrect to call it a mailing list. In fact, this is a class that does some abstract work on a given sample of consumers. Not inventing anything better, we began to call it triggers (in marketing they are called something like this, so the name is not bad). Frankly speaking, I was a little bit scared by the fact that I introduced a very abstract entity into the system, which can be called DoSomeWorkOnSomeCustomers . But there was no point in specializing triggers, so I decided not to bother about it, and in principle there are no big problems for clients with understanding what a trigger is.
The registration of the trigger looked something like this:
TriggerAction's interface is extremely simple:
The base class for trigger conditions is as follows:
But, for example, in the step that gives out the prize, it looks like this:
Also, I again applied my favorite replacement of inheritance to composition and instead of individual heirs for periodic and one-time triggers I made a strategy that checks whether the execution of the trigger should be repeated on the current consumer. This strategy takes Expression <Func <CustomerAction, bool >> from the marker step of the trigger and forms Expression <Func <Customer, bool >> using it, for additional checking whether it is necessary to execute the trigger on the consumer. Here is the implementation for a one-time trigger:
But for periodic:
Storage scheme of all this stuff in the database looks like this:
The essence of OperationStepGroup with one field looks rather strange, but it allows different entities (triggers, operations on the site, etc.) to refer to a group of records in a relational database. Moreover, later in this entity additional fields appeared, so that everything is not so scary.
In addition to getting rid of unnecessary marker action patterns, we can use the IsMarkerExpression obtained from the marker step of the trigger in order to display statistics on the number of trigger events. We can also add chains of triggers and operations (in operations, steps are also used, one of which is marked as marker).
As a result, the manager can start a trigger directly in the admin panel without the developer’s participation, although they often have to prompt: the establishment of a new trigger is not an easy task, but such is the price for the flexibility of this solution. A simpler solution would be less flexible, although we, of course, will have to work a lot more to simplify the UI without losing the current flexibility of our architecture (for example, you can make Wizards for simple triggers).
How it all looks in the UI, you can see here .
- happy birthday
- call the site if the consumer has not visited him for a long time
- make a personalized offer (divide customers into segments and send each letter its own segment)
In this article, we will describe how we solved this task - from writing each individual mailing by the developer from scratch 3 years ago, to the institution being sent by the manager via the web interface at the present time. The story may be interesting not only to those involved in email marketing, but also to everyone who has to implement the periodic execution of complex operations on certain consumer samples (I know that sounds very abstract, but in the end we had to solve just this abstract problem).
First implementations
3 years ago, similar tasks arose extremely rarely and every time we implemented them from scratch. In this case, the same questions arose:- How to tag consumers to whom we have already sent this letter?
- How to quickly process all consumers and not slow down the work of sites (which access the same records in the database)?
The answer to the first question was obvious for us: information is stored in our system about all significant actions performed by the consumer (logging in to the site, changing personal data) or above it (drawing a prize, sending a notification). In addition, we use actions for a variety of technical notes of consumers. So when sending an automatic mailing, we also decided to give the consumer a special action-marker , as a mark, that this automatic mailing had already been sent to him. In order not to re-send the newsletter, the condition “the consumer has no token action” is always added to the distribution condition.
On the second question, we stuffed a lot of cones associated with locks in the database, and eventually came up with the following pattern:
- Sending mailings comes from the windows-service, which periodically checks to see if there are new consumers that fit the conditions.
- In the service, the first step is a single query to the database with the Read Uncommitted isolation level . This request pulls out the Id of all consumers who need to send a letter. Due to the low isolation level, such a request does not impose locks on the records in the database and, as a result, has a very weak effect on the operation of the site. However, it does not guarantee the purity of the data and must be re-checked with a higher level of isolation.
- After we pulled out Id consumers, for each consumer we perform a separate transaction with a Serializable isolation level. In this transaction, we re-check whether the consumer is suitable for the conditions and, if so, send him a letter and issue an action marker. Since we process each consumer in a separate transaction, locks are imposed only on the data of one consumer and do not affect the work of other consumers. Since such a transaction is very short, the consumer, to whom the letter is sent, will also have no special problems if he walks the site at this time. The isolation level of a transaction must be Serializable, in order not to inadvertently send one letter twice, or not send a letter to the consumer who suddenly ceased to meet the conditions. Although, if we guarantee that sending the same mailing can only come from one stream and from one server, and also forget about the small probability that two mailings can be sent to one consumer with mutually exclusive conditions, you can use the Read Committed transaction .
Of course, after the implementation of several mailings on this template, we decided to render the template code. For this, the BatchMailing class was created, and for each new mailing we created and registered it in a special registry of its successor. In the successor, it was necessary to overload the following properties and methods:
- template of action-marker (earlier we called the template the type of action: I think this is a more understandable term for developers), which is issued when sending a letter
- sending method
- a method that performs additional actions (for example, together with sending birthday wishes, we can issue points to the consumer)
- the method that forms the Expression <Func <Customer, bool >> , which checks that the consumer fits the condition
The property and the first two methods never caused any problems, but compiling the Expression was not quite easy. This Expression was used twice - first in the Read Uncommitted request to pull the consumer Id, and then in the Serializable transaction, to re-check whether the consumer fits the condition. You had to write it in such a way that Linq to SQL could translate it into T-SQL. Conditions could be quite complicated and they always had problems. Not one newsletter could not start without writing a handful of tests on it. In addition, for sending SMS and email, we have got different intermediate heirs from BatchMailing. When we had to send both email and SMS, we had to copy-paste. I had ideas on how to fix this, but since automatic mailings were not so often requested by customers, this was a low-priority task.
')
Replace inheritance with composition
2 years ago, when developing a new advertising campaign, a client asked him to make 8 different automatic mailings at once. In this part of the conditions in the mailings were repeated. There was no doubt that it was impossible to live like this anymore, and I took up the rewriting of our architecture. In order to cope with all the problems described above, it was enough to apply our favorite technique: replacing inheritance with composition. This technique helped us so many times that I advise you to use composition instead of inheritance wherever possible (or at least consider this option). If you create a base abstract class with the idea “for each specific task I will have a successor overloading methods and properties,” immediately ask yourself “why would I not instead register an instance of a class for each task, passing it different settings”. And only if you are sure that the composition is not suitable here, use inheritance. If this and that is suitable, always lean towards the composition - this is how a much more flexible and understandable architecture turns out.In our situation:
- instead of overloading the property that returns the marker action template, this property is set to the class instance
- instead of overloading the sending / sms methods and performing additional logic, an arbitrary operation is placed on the class instance that needs to be performed on the consumer. The operation can be a combination of other operations.
- instead of overloading the method of forming Expression, a condition is put to the class instance. In this case, conditions can be combined through AND / OR
Since, apart from sending mailings, this entity can now perform any arbitrary operations on the consumer, it is incorrect to call it a mailing list. In fact, this is a class that does some abstract work on a given sample of consumers. Not inventing anything better, we began to call it triggers (in marketing they are called something like this, so the name is not bad). Frankly speaking, I was a little bit scared by the fact that I introduced a very abstract entity into the system, which can be called DoSomeWorkOnSomeCustomers . But there was no point in specializing triggers, so I decided not to bother about it, and in principle there are no big problems for clients with understanding what a trigger is.
The registration of the trigger looked something like this:
Add(new Trigger(“ one-to-one”) { MarkerActionTemplateSystemName = “InvitationMarker”, TriggerAction = new TriggerActionCombination( new GeneratePasswordForCustomerTriggerAction(), new SendEmailTriggerAction(“InvitationMailing”)), TriggerCondition = new AndTriggerConditionSet( new CustomerHasSubscripionCondition(), new CustomerHasEmailTriggerCondition(), new CustomerHadFirstActionOverChannelCondition(“OneToOne”)), }); TriggerAction's interface is extremely simple:
public interface ITriggerAction { void Execute( ModelContext modelContext, // Customer customer); } The base class for trigger conditions is as follows:
public class TriggerCondition { private readonly Func<ModelContext, Expression<Func<Customer, bool>>> triggerExpressionBuilder; public TriggerCondition(Func<ModelContext, Expression<Func<Customer, bool>>> triggerExpressionBuilder) { if (triggerExpressionBuilder == null) throw new ArgumentNullException("triggerExpressionBuilder"); this.triggerExpressionBuilder = triggerExpressionBuilder; } public Expression<Func<Customer, bool>> GetExpression(ModelContext modelContext) { return triggerExpressionBuilder(modelContext, brand); } // Read Uncommitted c Id , public IQueryable<Customer> ChooseCustomers(ModelContext modelContext, IQueryable<Customer> customers) { if (modelContext == null) throw new ArgumentNullException("modelContext"); if (customers == null) throw new ArgumentNullException("customers"); var expression = GetExpression(modelContext); return customers.Where(expression).ExpandExpressions(); } // Serializable , , public bool ShouldTrigger(ModelContext modelContext, Customer customer) { if (modelContext == null) throw new ArgumentNullException("modelContext"); if (customer == null) throw new ArgumentNullException("customer"); var expression = GetExpression(modelContext); // expression.Evaluate(customer), // return modelContext.Repositories.Get<CustomerRepository>().Items .Where(aCustomer => aCustomer == customer) .Where(aCustomer => expression.Evaluate(aCustomer)) .ExpandExpressions() .Any(); } } For frequently used conditions, we created successors from TriggerCondition , in which a specific Expression was built depending on the parameters passed to the constructor.All tired, start your own triggers
Using the architecture described above, we wound up the trigger in less than half an hour, by combining the conditions already written and the TriggerActions. However, this was not enough for us. The next step we wanted to completely eliminate the developers from the trigger establishment process. And in general I understood how to do it in a couple of months after the implementation of the previous version of the architecture. The conditions of the triggers were one to one similar to the filters that we use in the admin panel. Our filter system allows you to describe complex conditions, including requests for related entities, and also allows you to combine them through AND / OR. The filter forms an Expression, with the help of which you can already filter out entities in the database. And for all this, UI and serialization have already been written. It only remained to add a couple of filters that are often needed for triggers, but did not make sense during normal work with the list of consumers (for example: “N days have passed since the action”). For TriggerActions, you had to write a UI and a structure for storing them in the database, but here, too, in general, everything was clear. However, there were still some minor issues that had to be broken over:- By this time we began to register sending any letter as an action, and the marker action became superfluous - we could already determine to whom we sent the letter, and generally we would like to get rid of issuing unnecessary actions wherever possible.
- In addition to simple triggers that performed a specific set of operations once on each consumer, we had periodic triggers . It was necessary to think up how to transfer all this to the database and at the same time allow the use of arbitrary markers
- marketers invent triggers not separately from each other, but as chains in which there are both triggers and operations performed by the consumer on the site (a letter inviting you to visit the site and do something → the consumer performs several operations on the site → bonus points are added and sent a letter about it). I would like to, if not implement it immediately, then leave a reserve for the future, so that it is not difficult to describe the dependencies between triggers and operations
Need more Expression
Since the trigger performs an abstract operation step (formerly TriggerAction) on the consumer, and almost always this operation step is unique (for example, a certain letter is sent or a certain prize is issued only from this trigger), then this step can be used to check whether the checking algorithm has been executed. Since the trigger can have several steps of the operation, the manager will have to choose which of them is a marker (it makes no sense to check the performance of each step). However, it is easy to implement a method in the operation step that returns Expression <Func <Customer, bool >> it is impossible, since in each operation step one Expression for one-time triggers and the other for periodic ones would have to be formed. Here we are saved by the fact that almost any operation on a user in our system gives him an action. Accordingly, the operation step can filter out the actions that were issued to them. Most steps of an operation produce a specific action and for them the method that forms the Expression for filtering actions looks like this: public sealed override Expression<Func<CustomerAction, bool>> GetIsMarkerExpression(ModelContext modelContext) { return action => action.ActionTemplateId == ActionTemplateId; } But, for example, in the step that gives out the prize, it looks like this:
public override Expression<Func<CustomerAction, bool>> GetIsMarkerExpression(ModelContext modelContext) { IQueryable<anchor>habracut</anchor> customerPrizes = modelContext.Repositories.Get<CustomerPrizeRepository>().GetByPrizes(Prize); // , return action => customerPrizes.Any(prize => prize.CustomerActionId == action.Id); } Also, I again applied my favorite replacement of inheritance to composition and instead of individual heirs for periodic and one-time triggers I made a strategy that checks whether the execution of the trigger should be repeated on the current consumer. This strategy takes Expression <Func <CustomerAction, bool >> from the marker step of the trigger and forms Expression <Func <Customer, bool >> using it, for additional checking whether it is necessary to execute the trigger on the consumer. Here is the implementation for a one-time trigger:
public override Expression<Func<Customer, bool>> BuildShouldRepeatExpression(ModelContext modelContext, Expression<Func<CustomerAction, bool>> isMarkerExpression) { var markerActions = modelContext.Repositories.Get<CustomerActionRepository>().Items .Where(isMarkerExpression.ExpandExpressions()); return customer => !markerActions.Any(action => action.Customer == customer); } But for periodic:
public override Expression<Func<Customer, bool>> BuildShouldRepeatExpression( ModelContext modelContext, Expression<Func<CustomerAction, bool>> isMarkerExpression) { var isInPeriodExpression = PeriodType.BuildIsInPeriodExpression(modelContext, PeriodValue); var markerActions = modelContext.Repositories.Get<CustomerActionRepository>().Items .Where(isMarkerExpression.ExpandExpressions()); var markerActionsInPeriod = markerActions.Where(isInPeriodExpression.ExpandExpressions()); if (MaxRepeatCount == null) { return customer => !markerActionsInPeriod.Any(action => action.Customer == customer); } else { return customer => !markerActionsInPeriod.Any(action => action.Customer == customer) && markerActions.Count() < MaxRepeatCount.Value; } } It supports not only once every N days, but also once a calendar month / year, so Expression, which checks whether an action is in a given period, has been moved to a special class, PeriodType. Also supported by limiting the number of repetitions.Storage scheme of all this stuff in the database looks like this:
The essence of OperationStepGroup with one field looks rather strange, but it allows different entities (triggers, operations on the site, etc.) to refer to a group of records in a relational database. Moreover, later in this entity additional fields appeared, so that everything is not so scary.
In addition to getting rid of unnecessary marker action patterns, we can use the IsMarkerExpression obtained from the marker step of the trigger in order to display statistics on the number of trigger events. We can also add chains of triggers and operations (in operations, steps are also used, one of which is marked as marker).
As a result, the manager can start a trigger directly in the admin panel without the developer’s participation, although they often have to prompt: the establishment of a new trigger is not an easy task, but such is the price for the flexibility of this solution. A simpler solution would be less flexible, although we, of course, will have to work a lot more to simplify the UI without losing the current flexibility of our architecture (for example, you can make Wizards for simple triggers).
How it all looks in the UI, you can see here .
Source: https://habr.com/ru/post/251915/
All Articles