How did we do the service of advertising campaigns that comply with the provisions of GDPR
The GDPR, which took effect in May of this year, has seriously affected the Internet marketing market. Its participants want to form the most accurate audience for displaying ads, but now you need to get the explicit consent of the user, otherwise even a small niche resource can run into multi-million dollar fines. Some resources have closed, but many are being converted to meet new requirements. And our campaign management campaign project for a US customer is a great example.
Our client, an American company from Los Angeles, focuses on marketing in the music industry. His service provides support for complex advertising campaigns for groups of various sizes - from gangs around the corner to the stars of the world scene.
In essence, our client’s service is a startup that develops a specialized marketing platform that managed to collect investments from record companies, since it takes into account the peculiarities of the audience’s behavior in a single industry. The platform allows you to target advertising by analyzing the behavior of users who once showed interest in a particular team. The main users of the resource are musicians and industry workers (managers, record label employees, concert organizers, and many others).
')
By the time our project began, the service was already functioning and had managed to gain a certain position in the market. However, it was rather a prototype testing the main business hypothesis, especially since part of the functionality was implemented through partner services. We joined the development to create a full-fledged architecture, and the GDPR was the very impetus that launched all the transformations. Our task was not just to update the existing service, but to rebuild it in the light of new legislation, to refresh and lead to greater uniformity of the user interface.
The whole project was divided into several stages.
At the first stage, we rewrote from scratch the part that previously worked on PHP - the shortener of links and the authorization service.
The link shortener is a part of the functionality that is available for free to all registered users and is used to attract customers to the company's paid services. The service allows you to bring a long link to a short form in order to publish it in social networks. In this case, you can customize the link by setting up geo-targeting, for example: depending on the country from which the user came, you can show him different landing pages.
Updating the backend, we used OpenJDK 1.8, Kotlin and Spring boot / data / web - a standard framework for a project that does not expect high loads. By the way, it was in this project that we tried Kotlin “in battle” for the first time, and due to its syntax, it allowed us to significantly speed up development. Definitely, we will use it in other projects.
The database of the authorization service and the shortener implemented in the first stage is built on the basis of PostgreSQL - the relational storage model is well suited for solving these problems.
At the second stage, we took up the management of advertising campaigns. This functionality at the client platform also existed earlier, but through the service of the partner, for which you had to pay. Now the customer’s platform has grown to the scale when it was necessary to take care of its infrastructure. Own service, unlike third-party, is much easier to develop in the right direction, quickly making the required changes.
In this part of the project, we have implemented only the external API for the campaign manager. But at the third stage this module was developed - the API was finalized and a full-fledged UI was made for the campaign manager.
In parallel, we developed a small, proprietary DMP (data management platform) that manages visitor data collection. DMP data is stored in MongoDB, since we decided to leave the future horizontal scaling of this base. Now DMP serves up to 2 thousand requests per second (in peaks), while we focused on storing approximately a terabyte of data. Such a volume could be preserved in PostgreSQL, but in the long run, great efforts would have to be made to scale.
The same MongoDB is used in the campaign manager database (Campaign DB) - here the model of the document-oriented database is well suited to us. Campaigns go as single entities and can expand.
All caches work on Redis.
Front end: Angular 5, TypeScript, HTML5, Sass, Node.js, npm, Angular CLI.
During the last stage of the project, we have integrated the client systems and the key partner service, which opens the company access to a huge musician base, which will ensure an increase in the number of users.
There were some difficulties in the project. The customer wanted to save the user data and registrations accumulated during the service, while slightly changing the model. In parallel with the alteration of the architecture, we transferred registrations and changed its principles - from username to email as a login, which brought us many sleepless nights due to restrictions on the number of users with the same email address. The rights model in the system has changed. Previously, the service worked on a two-tier model, but we implemented the rights model without restrictions (including the ability to issue limited rights).
In parallel, expanded functionality. For example, at one of the stages a multistore appeared, with the help of which for end users who follow short links to a certain composition, you can customize pages with a list of services where these compositions are available for legal purchase or listening.
The main market of our client is the USA, but from Europe came about 10% of the traffic that the company did not want to lose. Since the entry into force of the new regulation (May 25, 2018), its targeting has simply been disabled. After consulting with lawyers, we built the service in such a way that it does not contradict the GDPR, and since August 16 we have launched the targeting again.
Honestly, we all two weeks studied the subtleties of the rules. The difficulty at this stage was that, given the general vagueness of the wording, there is still no law enforcement practice - real cases that could show how to do it right and how wrong. However, now (after our own research and consultations with lawyers) we are confident in the implemented solution architecture.
The logic of the service operation implied that a user, who follows a short link, should be assigned to a specific audience, so that later he could show ads. From the point of view of the GDPR, this cannot be done without the explicit consent of the user himself. And we implemented a request for consent - when clicking on the link at the bottom of the page, a dash appears with a question and two buttons. The request is opened for users whose IP belongs to Europe, and their response is stored as a cookie, so visitors do not have to press buttons every time.
If there is no user consent (i.e. cookies), we simply do not show him the pixel, i.e. in the general statistics of the service will be counted the fact of a visit, but user data will not be collected and taken into account.
GDPR imposes architectural restrictions - personal data must not only be correctly collected, but also safely stored. In our case, these restrictions relate to the implemented DMP (despite the fact that there are stored anonymous data with anonymous identifiers, which only hypothetically allow them to deanonymize) and the base of the authorization service. In our architecture, these databases are clearly allocated in separate blocks. In accordance with the regulations, access by third parties to these “sensitive” modules is naturally limited.
Only the corresponding service has access to the authorization base. All other services do not know anything about the personal data of the user of the platform - they are sufficiently validated by the authorization service.
Only the service of the same name enters the DMP database. In this case, only aggregates of data about the audience are given from the database, but not the data itself. These units form the basis of analytics, which users of the music industry receive in their personal account.
We also have a mandatory procedure for uploading and deleting user data. Here was the question of how to verify that the user is indeed legitimately requesting their data. And we use the “cookie” saved by the user as a factor for verification.
The project was completed quite recently, so it’s too early to talk about any numerical results.
Article author: Nikolay Eremin
PS We publish our articles on several sites Runet. Subscribe to our pages on the VK , FB or Telegram channel to learn about all of our publications and other news from Maxilect.
About service
Our client, an American company from Los Angeles, focuses on marketing in the music industry. His service provides support for complex advertising campaigns for groups of various sizes - from gangs around the corner to the stars of the world scene.
In essence, our client’s service is a startup that develops a specialized marketing platform that managed to collect investments from record companies, since it takes into account the peculiarities of the audience’s behavior in a single industry. The platform allows you to target advertising by analyzing the behavior of users who once showed interest in a particular team. The main users of the resource are musicians and industry workers (managers, record label employees, concert organizers, and many others).
')
By the time our project began, the service was already functioning and had managed to gain a certain position in the market. However, it was rather a prototype testing the main business hypothesis, especially since part of the functionality was implemented through partner services. We joined the development to create a full-fledged architecture, and the GDPR was the very impetus that launched all the transformations. Our task was not just to update the existing service, but to rebuild it in the light of new legislation, to refresh and lead to greater uniformity of the user interface.
Implementation
The whole project was divided into several stages.
At the first stage, we rewrote from scratch the part that previously worked on PHP - the shortener of links and the authorization service.
The link shortener is a part of the functionality that is available for free to all registered users and is used to attract customers to the company's paid services. The service allows you to bring a long link to a short form in order to publish it in social networks. In this case, you can customize the link by setting up geo-targeting, for example: depending on the country from which the user came, you can show him different landing pages.
Updating the backend, we used OpenJDK 1.8, Kotlin and Spring boot / data / web - a standard framework for a project that does not expect high loads. By the way, it was in this project that we tried Kotlin “in battle” for the first time, and due to its syntax, it allowed us to significantly speed up development. Definitely, we will use it in other projects.
The database of the authorization service and the shortener implemented in the first stage is built on the basis of PostgreSQL - the relational storage model is well suited for solving these problems.
At the second stage, we took up the management of advertising campaigns. This functionality at the client platform also existed earlier, but through the service of the partner, for which you had to pay. Now the customer’s platform has grown to the scale when it was necessary to take care of its infrastructure. Own service, unlike third-party, is much easier to develop in the right direction, quickly making the required changes.
In this part of the project, we have implemented only the external API for the campaign manager. But at the third stage this module was developed - the API was finalized and a full-fledged UI was made for the campaign manager.
In parallel, we developed a small, proprietary DMP (data management platform) that manages visitor data collection. DMP data is stored in MongoDB, since we decided to leave the future horizontal scaling of this base. Now DMP serves up to 2 thousand requests per second (in peaks), while we focused on storing approximately a terabyte of data. Such a volume could be preserved in PostgreSQL, but in the long run, great efforts would have to be made to scale.
The same MongoDB is used in the campaign manager database (Campaign DB) - here the model of the document-oriented database is well suited to us. Campaigns go as single entities and can expand.
All caches work on Redis.
Front end: Angular 5, TypeScript, HTML5, Sass, Node.js, npm, Angular CLI.
During the last stage of the project, we have integrated the client systems and the key partner service, which opens the company access to a huge musician base, which will ensure an increase in the number of users.
There were some difficulties in the project. The customer wanted to save the user data and registrations accumulated during the service, while slightly changing the model. In parallel with the alteration of the architecture, we transferred registrations and changed its principles - from username to email as a login, which brought us many sleepless nights due to restrictions on the number of users with the same email address. The rights model in the system has changed. Previously, the service worked on a two-tier model, but we implemented the rights model without restrictions (including the ability to issue limited rights).
In parallel, expanded functionality. For example, at one of the stages a multistore appeared, with the help of which for end users who follow short links to a certain composition, you can customize pages with a list of services where these compositions are available for legal purchase or listening.
About GDPR should be said separately
The main market of our client is the USA, but from Europe came about 10% of the traffic that the company did not want to lose. Since the entry into force of the new regulation (May 25, 2018), its targeting has simply been disabled. After consulting with lawyers, we built the service in such a way that it does not contradict the GDPR, and since August 16 we have launched the targeting again.
Honestly, we all two weeks studied the subtleties of the rules. The difficulty at this stage was that, given the general vagueness of the wording, there is still no law enforcement practice - real cases that could show how to do it right and how wrong. However, now (after our own research and consultations with lawyers) we are confident in the implemented solution architecture.
The logic of the service operation implied that a user, who follows a short link, should be assigned to a specific audience, so that later he could show ads. From the point of view of the GDPR, this cannot be done without the explicit consent of the user himself. And we implemented a request for consent - when clicking on the link at the bottom of the page, a dash appears with a question and two buttons. The request is opened for users whose IP belongs to Europe, and their response is stored as a cookie, so visitors do not have to press buttons every time.
If there is no user consent (i.e. cookies), we simply do not show him the pixel, i.e. in the general statistics of the service will be counted the fact of a visit, but user data will not be collected and taken into account.
GDPR imposes architectural restrictions - personal data must not only be correctly collected, but also safely stored. In our case, these restrictions relate to the implemented DMP (despite the fact that there are stored anonymous data with anonymous identifiers, which only hypothetically allow them to deanonymize) and the base of the authorization service. In our architecture, these databases are clearly allocated in separate blocks. In accordance with the regulations, access by third parties to these “sensitive” modules is naturally limited.
Only the corresponding service has access to the authorization base. All other services do not know anything about the personal data of the user of the platform - they are sufficiently validated by the authorization service.
Only the service of the same name enters the DMP database. In this case, only aggregates of data about the audience are given from the database, but not the data itself. These units form the basis of analytics, which users of the music industry receive in their personal account.
We also have a mandatory procedure for uploading and deleting user data. Here was the question of how to verify that the user is indeed legitimately requesting their data. And we use the “cookie” saved by the user as a factor for verification.
The project was completed quite recently, so it’s too early to talk about any numerical results.
Article author: Nikolay Eremin
PS We publish our articles on several sites Runet. Subscribe to our pages on the VK , FB or Telegram channel to learn about all of our publications and other news from Maxilect.
Source: https://habr.com/ru/post/432772/
All Articles