How we build a UI for advertising systems

Instead of intro

Earlier in our blog, we wrote what the company IPONWEB does - we automate the display of advertising on the Internet. Our systems make decisions not only on the basis of historical data, but also actively use information obtained in real time. In the case of DSP (Demand Side Platform - an advertising platform for advertisers), the advertiser (or his representative) must create and upload an advertising banner (creative) in one of the formats (image, video, interactive banner, image + text, etc.) , select the audience of users for whom this banner will be shown, determine how many times you can show ads to a single user, in which countries, on which sites, on which devices, and reflect this (and much more) in the advertising campaign targeting settings, as well as distribute advertising budget s. For SSP (Supply Side Platform - an advertising platform for owners of advertising sites), the owner of the site (mobile application, billboard, television channel) must determine the advertising spaces on his resource and indicate, for example, which categories of advertising he is ready to show on them. All these settings are made manually in advance (not at the time of advertising) using the user interface. In this article I will talk about our approach to building such interfaces, provided that there are many of them, they are similar to each other and at the same time have individual features.

How it all began

We began to engage in advertising business back in 2007, but we didn’t start to make interfaces right away, but only since 2014. We traditionally deal with the development of custom platforms, which are completely designed in accordance with the specifics of the business of each individual client - among the dozens of platforms that we have built, no two are alike. And since our advertising platforms were designed without restrictions on customization possibilities, the user interface had to meet the same requirements.

When five years ago we received the first request for an ad interface for DSP, our choice fell on a popular and convenient technology stack: JavaScript and AngularJS at the front end, and at the back end - Python, Django and Django Rest Framework (DRF). From this was made the most common project, whose main task was to provide CRUD functionality. The result of his work was the file with the settings for the advertising system in XML format. Now this interaction protocol may seem strange, but, as we have already discussed, we began to build the first advertising systems (still without UI) in the “zero”, and this format has been preserved until now.
')
After the successful launch of the first project, the following did not take long to wait. These were also UIs for DSP and the requirements for them were the same as for the first project. Nearly. Despite the fact that everything was very similar, the devil was in the details - there is a slightly different hierarchy of objects, a couple of fields were added there ... The most obvious way to get a second project very similar to the first one, but with modifications, was the replication method, which we used . And it entailed problems familiar to many - along with the “good” code they copied and bugs, patches for which needed to be spread by hand. The same thing happened with all the new features that were rolled out on all active projects.

In this mode, it was possible to work while there were few projects, but when their number exceeded 20, the usual approach ceased to scale. Therefore, we decided to bring the common parts of the projects to the library, from which the project will connect the components it needs. If a bug is detected, it is repaired once in the library and distributed to projects automatically when the library version is updated, and the same with the reuse of new features.

Configuration and Terminology

We had several iterations in the implementation of this approach, and they all flowed into each other evolutionarily, starting with our usual project on pure DRF. In the latest implementation, our project is described using DSL based on JSON (see picture). This JSON describes both the structure of the project components and their interconnections, and it can read both the frontend and the backend.

After initializing the Angular application, the frontend requests a JSON config from the backend. The backend does not just give a static configuration file, but additionally processes it, complementing with various metadata or deleting parts of the config that are responsible for parts of the system that are not accessible to the user. This allows you to show the interface to different users in different ways, including interactive forms, CSS styles of the entire application and specific design elements. The latter is especially important for user interfaces of platforms used by different types of clients with different roles and access levels.



The backend, in contrast to the frontend, reads the configuration once during the initialization stage of a Django application. Thus, the full amount of functionality is registered on the backend, and access to various parts of the system is checked on the fly.

Before turning to the most interesting - the database structure - I want to introduce several concepts that we use when talking about the structure of our projects in order to be on the same wavelength with the reader.

These concepts - Entity and Feature - are well illustrated on the data entry form (see picture). The entire form is an Entity, and the individual fields on it are the Feature. The picture also shows Endpoint (just in case). So, Entity is an independent object in the system, on which CRUD operations can be performed, while Feature is only part of “something bigger”, part of Entity. With Feature, CRUD operations cannot be performed without being tied to any Entity. For example: an advertising campaign budget without reference to the campaign itself is simply a number that cannot be used without having information about the parent campaign.



The same concepts can be found in the JSON configuration of the project (see picture).

Database structure

The most interesting part of our projects is the database structure and its supporting mechanics. Having started using PostgreSQL for the very first versions of our projects, we remain on this technology today. At the same time, we actively use Django ORM. In earlier implementations, we used the standard model of relations between objects (entities) on Foreign Key, however, this approach caused difficulties when it was necessary to change the hierarchy of relations. For example, some clients needed to enter the Agency level (Business Unit -> Agency -> Advertiser -> ...) into the standard DSP Business Unit -> Advertiser -> Campaign hierarchy. Therefore, we gradually abandoned the use of Foreign Key and organized connections between objects using Many To Many connections through a separate table, we have it called `LinkRegistry`.

In addition, we gradually abandoned the hardcode for filling the entities and began to store most of the fields in separate tables, linking them also via `LinkRegistry` (see picture). Why was this necessary? For each client, the content of the entity may differ - some fields will be added or deleted. So, we have to store a superset of fields in each entity across all of our clients. At the same time, they will all have to be made optional, so that “alien” mandatory fields do not interfere with the work.



Consider the example in the picture: here is described the database structure for a creative with one additional field - `image_url`. The creative table only stores its `id`, and` image_url` is stored in a separate table, their relationship is described by another entry in the `LinkRegistry` table. Thus, this creative will be described by three entries, one in each of the tables. Accordingly, in order to preserve such a creative, you need to make an entry in each of them, and in order to read - in the same way, visit 3 tables. It would be very inconvenient to write such processing every time from scratch, so our library abstracts all these details from the programmer. To work with the data in Django and DRF models and serializers are used, described by the code. In our projects, the set of fields in models and serializers is determined in runtime by JSON configuration, model classes are created dynamically (using the type function) and stored in a special register, from where they are available during application operation. We also use special base classes for these models and serializers, which help in working with non-standard base structure.

When you save a new object (or update an existing one), the data received from the frontend enters the serializer, where they are validated - there is nothing unusual here, the standard DRF mechanisms work. But saving and updating are redefined. The serializer always knows which model it works with, and from the internal representation of our dynamic model, he can understand which table to put the data of the next field in. We encode this information in the custom fields of the models (remember how `ForeignKey` is described in Django - the related model is passed inside the field, we do the same). In these special fields, we also abstract the need to add a third binding entry to LinkRegistry using the descriptor mechanism — in the code you write `creative.image_url = 'http: // foo.bar'`, and in the overridden` __set__` method we write to `LinkRegistry`.
This is with regard to writing to the database. And now let's deal with reading. How is a tuple taken out of a database converted to a Django instance instance? In the basic Django model, there is a `from_db` method that is called for each resulting tuple when executing a query in` queryset`. At the entrance, it receives a tuple and returns the instance of the Django model. We have redefined this method in our base model, where, based on the resulting tuple of the main model (where only ʻid` comes in), we obtain data from other related tables and, having this complete set, we will instantiate the model. Of course, we also worked on optimizing the Django prefetch mechanism for our non-standard use case.

Testing

Our framework is quite complex, so we write a lot of tests. We have tests for both frontend and backend. I will focus on the backend tests.

To run the tests we use pytest. On the backend, we have two large classes of tests: tests of our framework (we also call it “core”) and design tests.

In the kernel, we write both isolated unit tests and functional ones for endpoint testing using the pytest-django plugin. In general, all work with the database is mainly tested through requests to the API, as it happens in production.

Functional tests can specify the JSON configuration. In order not to become attached to project terminology, we use “dummy” names for entities with which we test our Features in the core (“Emma”, “Alla”, “Karl”, “Maria” and so on). Because, having written the image_url feature, we don’t want to limit the developer’s consciousness to the fact that it can be used only with the Creative entity - features and entities are universal, and they can be combined with each other in any combinations that are relevant for a particular client.

As for testing projects, in them all test cases are run with production configuration, no dummy entities, since it is important for us to check exactly what the client will work with. In the project, you can write any tests that will cover the features of the business logic of the project. At the same time, basic CRUD tests can be connected to the project from the kernel. They are written in a general form, and they can be connected to any project: the feature test can read the JSON configuration of the project, determine to which entities this feature is connected, and run checks only for the required entities. For the convenience of preparing test data, we have developed a system of helpers who, also based on the JSON configuration, are able to prepare test data sets. A special place in testing projects is taken by E2E tests on Protractor, which check all the basic functions of the project. These tests are also described using JSON, they are written and supported by front-end developers.

Afterword

In this article, we reviewed the modular project approach developed by the UI department of IPONWEB. This solution has been successfully operating in production for three years. However, this solution still has a number of limitations that do not allow us to stop there. First, our code base is still quite complex. Secondly, the basic code that supports dynamic models is associated with such critical components as search, mass loading of objects, differentiation of access rights and others. Because of this, changes in one of the components can significantly affect others. In an effort to get rid of these limitations, we continue to actively rework our library, breaking it into many independent parts and reducing the complexity of the code. We will surely tell you about the results in the following articles.

The article is an extended transcript of my speech at MoscowPythonConf ++ 2019, so I also share links to videos and slides .