Feed Manager development for automated traffic purchase
Mobile marketing involves the interaction of two or three key figures.
The client is an advertiser or mobile application developer who wants to buy good quality traffic.
Publisher - arbitrator, group administrator in the social. network, application developer with in-app advertising.
')
And also often the intermediary is a mobile marketing agency, which deals with the selection of offers from customers and the optimization of partner traffic.
Since Mobio is an agency, in the article we will look at exactly how, from our point of view, work with traffic occurs.
Mainly used 2 types of work with traffic:
Manual (manual) - each offer is manually created in the tracker, after which traffic sources are also manually connected to it, payments are set up. Changes are also controlled by the manager independently.
API-feed (automatic) - automated collection of a large number (several thousand) of offers, their processing, uploading to the system and connecting sources. Including, automatic introduction of all changes received from the advertiser in real time.
When we first started working with API-feed at Mobio , we decided to explore solutions that already exist on the market. The task was to find something that can be adapted to the needs of the company. But we faced a number of difficulties that hampered the effectiveness of the team. Therefore, it was decided to go its own way, creating its own system of working with API-feed (feed-management).
Axonite is one of the largest players on the API feed market. It was originally sharpened under the HasOffers tracker. Mobio has successfully configured Affise integration. There is a ready-made integration with the majority of large advertisers in the system, it’s enough to insert the key and additional data for the necessary client, after which the offers are automatically loaded into the system.
Select the connection in Axonite. The list is large, in the screenshot only a small part of it.
Then they are filtered by adjustable parameters. We choose which partners need to be connected to each of the streams (the stream is a sample of offers from one / several advertisers according to the parameters specified in the filter). The system exports data to the tracker and maintains its relevance.
But after the launch, we encountered many difficulties:
1. Inconvenient flow formation
Each stream must be separate for each advertiser. To connect a partner, you also need to create a separate “Source-Source” relationship. In practice, this means that in order to connect 20 sources to 2 streams, you must create (manually!) 40 connections. It takes a lot of time. In addition, the cost of the tariff plan limits the number of such connections. It also forces you to waste time on removing connections when there are too many of them. Or pay more money.
Creating a stream-source connection. Enter all data, save, open a new form - repeat N times
2. Limit of requests from the IP address
Under load, the server cannot cope with the prompt sending of information. The Axonite staff explained this with the tracker limits, but later, when developing our system, we were convinced that this was not entirely true.
Of course, we were expanding the instance, but this was done far from as quickly as was required for comfortable business expansion.
3. Outdated Connections
Many connections are outdated. That large list of clients to which the connection was configured was not as relevant as we expected. Often you had to wait for the connection to be reconfigured, since Axonite themselves do not update the information until requested by the user.
4. The inability to make a copy of the connection as a separate category
For example, an Advertiser advertiser provides 2 lists of offers: one without complicated KPIs, the second - highly sensitive offers with difficult-to-do KPIs. The first can be connected to all sources, the second - only the best. However, Axonite will mix all offers in one thread, which spoils the whole picture.
As a result, after working with Axonite for several months, we realized that it would be easier and more useful to make our own system. For all the time with the work of Axonite, we have accumulated enough cases with nuances to which we could be oriented during the development.
CPAPI is an API integration solution implemented by Affise. We tested them before Axonite, but we didn’t take a permanent job. At that time, when we were just starting our activity on setting up API integration, the system was rather raw. There were few ready-made connections, we could not influence the queue of their creation. CPAPI had its own list of priorities. Now the list has expanded, but still there are some flaws:
1. The list is small, many large customers do not, especially Chinese.
List of available connections
2. Connecting partners is not automated.
After sending the offers to the tracker, you must manually connect the partners to the offers. Considering the volume of sources appearing with time, this is extremely suboptimal. Especially when it comes to a system designed to get rid of such problems.
But you can choose which fields to synchronize constantly, and which - no. For example, if you want to change something, and not leave it "as it is."
In addition to the specific disadvantages, there are also common "Wishlist", not implemented by default in systems on the market. This is not a drawback of the system, but at the same time, the introduction of something like this will require considerable time, cost and coordination. For example, you cannot block individual sub-id traffic sources, and this is an integral part of the business. Or analytics of the level of conversion of offers with disconnection of non-converting - for such an analysis requires access to the tracker statistics, which the API systems do not possess.
As a result, we realized that outsourcing can be managed only at the initial stage of work, and since our growth rates required expanding technical limitations, we sat down to think through and develop our own system of API offers.
The system had the following requirements, which were included in the TK for developers:
In our field, many advertisers use their own trackers. Everyone has different APIs and data storage formats. Offer categories (payment for the installation or for the event) are transferred differently. Ultimately, all this information should not only be collected, but also brought to a single form. Moreover, this view must adequately pass into our own tracker, given its capabilities.
It was decided to write a separate collector for GoLang for each client tracker. This made it possible to quickly collect and update raw data.
2. Match fields
Next, the developer created a pull request for the system, helping her to understand which field in the reservoir corresponds to which. If necessary, simple calculations were added, for example, if the advertiser sends the monthly cap (maximum number of installations per month), it had to be divided by 30, since Affise tracker has only daily cap (per day) and total cap (all time).
3. Check the activity of the offer
The check was carried out in the following way: if at the next request for data the offer that was already loaded earlier was not found, it was transferred to the inactivity status.
When first loading data into the system, only active records from the collector were affected. Further, the system also compared the statuses of already loaded offers.
4. Filtering Offers
After the first data load, filtering was performed according to the specified rules. Offers were removed with uninteresting GEO business, low payouts, inappropriate currency, OS, device and offer type, motive or not motive (we work exclusively with unmotivated offers).
In the basic version, filtering was the only level of verification before the offer got into our tracker. However, it was suboptimal - many offers came to us "broken", i.e. the link did not go where you want. Some offers had so many redirects on the way to the final link that it was completely uninteresting to work with them.
The decision came from our business colleagues. Mobrand gave us the opportunity to use our Offertest - this is a third-party system that checks that the click from the desired GEO and OS leads exactly where it is needed, and also reports through how many redirects you had to go. The working conditions with this system are much more favorable than with the popular analogue Affilitest, while the functionality is no less. Therefore, after filtering the offers, those that have passed the filter have been tested on the Offertest. The main filtering rule is compliance of the final page stated in the offer. In addition, the maximum allowed number of redirects was individually set.
We are satisfied with the system and can recommend this product for use. It allows you to significantly improve the quality of the received sets of offers and is several times cheaper than the analogues. The stability of the work was checked personally by us at the maximum possible capacities - no falls were noticed. With all this, colleagues regularly improve their system, observing versioning, i.e. a fresh update will not break the old settings.
But that's not all! Even after all the checks, there were overlapping offers. For example, there were 5 offers of conditional Yandex. Browser for Android and RU-geo, but with different payouts. Our system analyzed them, creating a top-group - a group of offers with a unique application identifier and GEO, and then choosing 2 with the highest payout. If one or both offers stopped, the next ones came in their place.
All such checks are carried out in real time for all active or new offers in order to maximize the existing stream of offers.
Finally, all the remaining offers are loaded by API into our Affise tracker. The work of traffic sources with offers is happening exclusively in it, leaving the feed system closed from third-party users.
5. Work with traffic sources
In addition to processing and downloading offers, it was also necessary to work comfortably with traffic sources, quickly connecting and disconnecting them from different advertisers. This was also done in the feed system interface. It was not necessary to form any flows, as it was in Axonite, the connection was made intuitive. In addition, it was possible to configure connections separately for each offer, in order to enable the most detailed setting of the traffic flow.
Additionally, a CR rating system was implemented (the ratio of conversions to clicks). In case it is too low (the offer does not convert on traffic from a certain source), the source is disconnected from the specific offer in order to switch to those that will have good CR and give a positive profit.
Plans include adding integration with the FraudScore system. In order to hit the source of a random fraud - to block this source. At the moment, this is done manually, but otherwise the level of manual work is as low as possible, especially in comparison with the solutions that are on the market.
Difficulties
1. Unification
As mentioned earlier - all the data received in the collectors was necessary to bring to a general form. Sometimes the format of the response from the advertiser was difficult to understand and unify. For example, the FuseClick tracker considers CPI-offers as a subtype of CPA - which is a gross mistake from the point of view of our system. Or sometimes there is no preview-link in the answer - you had to take package_name (application identifier) ​​and create a link to the application store yourself.
Such aspects often arise when writing a collector, so it was extremely important both to understand what is meant in a particular field and to maintain constant contact with the advertiser when developing - to clarify unclear points.
2. Limit the number of requests
Like any system, Affise tracker has a limit of requests that can be sent from IP. Therefore, the speed of the feed system is limited by this limit - we had to take this factor into account. With current capacities, it would be possible to achieve greater work speed, however, it is worth saying thanks to colleagues who raised the limit for the system by 2 times. Currently, the average update rate is 10 minutes for the entire stream, which is a very good market indicator.
In the process of constant improvements to different systems, it is often necessary to think through and rewrite the modules that were responsible for the logic of work - since it can change several times. It all depends on the developer - our employee thought out the modularity and extensibility of components. It is important not to sacrifice this for the sake of low development time - because if you make a monolithic system, then the improvements will take a lot of time (I dare to assume that solutions presented on the market are wrong with this - because our thoughts to add the really necessary module to customers often ran into failure due to “technical difficulties”)
4. Tracker limitations
Another problem is the limited capabilities of the tracker to which offers are exported. Despite the fact that Affise doesn’t have any particular problems with this, some of the things familiar to the feeds market were initially absent. For example, the tracker does not have a separate field for package_name, and its transfer is crucial for sources. Of course, you can create a “crutch” using a different text field for this parameter, which is present in the tracker, but not really needed for feeds - but it would be great to make everything beautiful. Well, we are sure that our colleagues will not let us down and will remove the restrictions that are currently present.
Conclusion
In the process of constant improvements to different systems, it is often necessary to think through and rewrite those modules that were responsible for the logic of work, since it can change several times.
It all depends on the developer. Our employee thought out the modularity and extensibility of components. It is important not to sacrifice this for the sake of low development time, because if you make a monolithic system, the improvements will take a lot of time.
Despite all the difficulties encountered in the development of their own solutions, the team never regretted that they decided to spend their own resources and funds on this project. The time that was ultimately saved, and the unique advantages that we received, repeatedly paid off all the development costs. Perhaps we will think about preparing a public version of the product so that anyone can appreciate our work.
For now, if you want to get excellent traffic from trusted sources, or you yourself have a good inventory of traffic and are looking for a suitable set of offers for it, we will be happy to talk about potential cooperation.
Subscribe and read interesting news, analytics and other insights about mobile marketing in the Mobio blog .
The client is an advertiser or mobile application developer who wants to buy good quality traffic.
Publisher - arbitrator, group administrator in the social. network, application developer with in-app advertising.
')
And also often the intermediary is a mobile marketing agency, which deals with the selection of offers from customers and the optimization of partner traffic.
Since Mobio is an agency, in the article we will look at exactly how, from our point of view, work with traffic occurs.
Mainly used 2 types of work with traffic:
Manual (manual) - each offer is manually created in the tracker, after which traffic sources are also manually connected to it, payments are set up. Changes are also controlled by the manager independently.
API-feed (automatic) - automated collection of a large number (several thousand) of offers, their processing, uploading to the system and connecting sources. Including, automatic introduction of all changes received from the advertiser in real time.
When we first started working with API-feed at Mobio , we decided to explore solutions that already exist on the market. The task was to find something that can be adapted to the needs of the company. But we faced a number of difficulties that hampered the effectiveness of the team. Therefore, it was decided to go its own way, creating its own system of working with API-feed (feed-management).
Third-party solutions on the market
Axonite
Axonite is one of the largest players on the API feed market. It was originally sharpened under the HasOffers tracker. Mobio has successfully configured Affise integration. There is a ready-made integration with the majority of large advertisers in the system, it’s enough to insert the key and additional data for the necessary client, after which the offers are automatically loaded into the system.
Select the connection in Axonite. The list is large, in the screenshot only a small part of it.
Then they are filtered by adjustable parameters. We choose which partners need to be connected to each of the streams (the stream is a sample of offers from one / several advertisers according to the parameters specified in the filter). The system exports data to the tracker and maintains its relevance.
But after the launch, we encountered many difficulties:
1. Inconvenient flow formation
Each stream must be separate for each advertiser. To connect a partner, you also need to create a separate “Source-Source” relationship. In practice, this means that in order to connect 20 sources to 2 streams, you must create (manually!) 40 connections. It takes a lot of time. In addition, the cost of the tariff plan limits the number of such connections. It also forces you to waste time on removing connections when there are too many of them. Or pay more money.
Creating a stream-source connection. Enter all data, save, open a new form - repeat N times
2. Limit of requests from the IP address
Under load, the server cannot cope with the prompt sending of information. The Axonite staff explained this with the tracker limits, but later, when developing our system, we were convinced that this was not entirely true.
Of course, we were expanding the instance, but this was done far from as quickly as was required for comfortable business expansion.
3. Outdated Connections
Many connections are outdated. That large list of clients to which the connection was configured was not as relevant as we expected. Often you had to wait for the connection to be reconfigured, since Axonite themselves do not update the information until requested by the user.
4. The inability to make a copy of the connection as a separate category
For example, an Advertiser advertiser provides 2 lists of offers: one without complicated KPIs, the second - highly sensitive offers with difficult-to-do KPIs. The first can be connected to all sources, the second - only the best. However, Axonite will mix all offers in one thread, which spoils the whole picture.
As a result, after working with Axonite for several months, we realized that it would be easier and more useful to make our own system. For all the time with the work of Axonite, we have accumulated enough cases with nuances to which we could be oriented during the development.
CPAPI
CPAPI is an API integration solution implemented by Affise. We tested them before Axonite, but we didn’t take a permanent job. At that time, when we were just starting our activity on setting up API integration, the system was rather raw. There were few ready-made connections, we could not influence the queue of their creation. CPAPI had its own list of priorities. Now the list has expanded, but still there are some flaws:
1. The list is small, many large customers do not, especially Chinese.
List of available connections
2. Connecting partners is not automated.
After sending the offers to the tracker, you must manually connect the partners to the offers. Considering the volume of sources appearing with time, this is extremely suboptimal. Especially when it comes to a system designed to get rid of such problems.
But you can choose which fields to synchronize constantly, and which - no. For example, if you want to change something, and not leave it "as it is."
In addition to the specific disadvantages, there are also common "Wishlist", not implemented by default in systems on the market. This is not a drawback of the system, but at the same time, the introduction of something like this will require considerable time, cost and coordination. For example, you cannot block individual sub-id traffic sources, and this is an integral part of the business. Or analytics of the level of conversion of offers with disconnection of non-converting - for such an analysis requires access to the tracker statistics, which the API systems do not possess.
As a result, we realized that outsourcing can be managed only at the initial stage of work, and since our growth rates required expanding technical limitations, we sat down to think through and develop our own system of API offers.
Building your own solution: a general scheme of work and additional chips
The system had the following requirements, which were included in the TK for developers:
1. Uniformity of information
In our field, many advertisers use their own trackers. Everyone has different APIs and data storage formats. Offer categories (payment for the installation or for the event) are transferred differently. Ultimately, all this information should not only be collected, but also brought to a single form. Moreover, this view must adequately pass into our own tracker, given its capabilities.
It was decided to write a separate collector for GoLang for each client tracker. This made it possible to quickly collect and update raw data.
2. Match fields
Next, the developer created a pull request for the system, helping her to understand which field in the reservoir corresponds to which. If necessary, simple calculations were added, for example, if the advertiser sends the monthly cap (maximum number of installations per month), it had to be divided by 30, since Affise tracker has only daily cap (per day) and total cap (all time).
3. Check the activity of the offer
The check was carried out in the following way: if at the next request for data the offer that was already loaded earlier was not found, it was transferred to the inactivity status.
When first loading data into the system, only active records from the collector were affected. Further, the system also compared the statuses of already loaded offers.
4. Filtering Offers
After the first data load, filtering was performed according to the specified rules. Offers were removed with uninteresting GEO business, low payouts, inappropriate currency, OS, device and offer type, motive or not motive (we work exclusively with unmotivated offers).
In the basic version, filtering was the only level of verification before the offer got into our tracker. However, it was suboptimal - many offers came to us "broken", i.e. the link did not go where you want. Some offers had so many redirects on the way to the final link that it was completely uninteresting to work with them.
The decision came from our business colleagues. Mobrand gave us the opportunity to use our Offertest - this is a third-party system that checks that the click from the desired GEO and OS leads exactly where it is needed, and also reports through how many redirects you had to go. The working conditions with this system are much more favorable than with the popular analogue Affilitest, while the functionality is no less. Therefore, after filtering the offers, those that have passed the filter have been tested on the Offertest. The main filtering rule is compliance of the final page stated in the offer. In addition, the maximum allowed number of redirects was individually set.
We are satisfied with the system and can recommend this product for use. It allows you to significantly improve the quality of the received sets of offers and is several times cheaper than the analogues. The stability of the work was checked personally by us at the maximum possible capacities - no falls were noticed. With all this, colleagues regularly improve their system, observing versioning, i.e. a fresh update will not break the old settings.
But that's not all! Even after all the checks, there were overlapping offers. For example, there were 5 offers of conditional Yandex. Browser for Android and RU-geo, but with different payouts. Our system analyzed them, creating a top-group - a group of offers with a unique application identifier and GEO, and then choosing 2 with the highest payout. If one or both offers stopped, the next ones came in their place.
All such checks are carried out in real time for all active or new offers in order to maximize the existing stream of offers.
Finally, all the remaining offers are loaded by API into our Affise tracker. The work of traffic sources with offers is happening exclusively in it, leaving the feed system closed from third-party users.
5. Work with traffic sources
In addition to processing and downloading offers, it was also necessary to work comfortably with traffic sources, quickly connecting and disconnecting them from different advertisers. This was also done in the feed system interface. It was not necessary to form any flows, as it was in Axonite, the connection was made intuitive. In addition, it was possible to configure connections separately for each offer, in order to enable the most detailed setting of the traffic flow.
Additionally, a CR rating system was implemented (the ratio of conversions to clicks). In case it is too low (the offer does not convert on traffic from a certain source), the source is disconnected from the specific offer in order to switch to those that will have good CR and give a positive profit.
Plans include adding integration with the FraudScore system. In order to hit the source of a random fraud - to block this source. At the moment, this is done manually, but otherwise the level of manual work is as low as possible, especially in comparison with the solutions that are on the market.
Difficulties
1. Unification
As mentioned earlier - all the data received in the collectors was necessary to bring to a general form. Sometimes the format of the response from the advertiser was difficult to understand and unify. For example, the FuseClick tracker considers CPI-offers as a subtype of CPA - which is a gross mistake from the point of view of our system. Or sometimes there is no preview-link in the answer - you had to take package_name (application identifier) ​​and create a link to the application store yourself.
Such aspects often arise when writing a collector, so it was extremely important both to understand what is meant in a particular field and to maintain constant contact with the advertiser when developing - to clarify unclear points.
2. Limit the number of requests
Like any system, Affise tracker has a limit of requests that can be sent from IP. Therefore, the speed of the feed system is limited by this limit - we had to take this factor into account. With current capacities, it would be possible to achieve greater work speed, however, it is worth saying thanks to colleagues who raised the limit for the system by 2 times. Currently, the average update rate is 10 minutes for the entire stream, which is a very good market indicator.
3. Change of logic
In the process of constant improvements to different systems, it is often necessary to think through and rewrite the modules that were responsible for the logic of work - since it can change several times. It all depends on the developer - our employee thought out the modularity and extensibility of components. It is important not to sacrifice this for the sake of low development time - because if you make a monolithic system, then the improvements will take a lot of time (I dare to assume that solutions presented on the market are wrong with this - because our thoughts to add the really necessary module to customers often ran into failure due to “technical difficulties”)
4. Tracker limitations
Another problem is the limited capabilities of the tracker to which offers are exported. Despite the fact that Affise doesn’t have any particular problems with this, some of the things familiar to the feeds market were initially absent. For example, the tracker does not have a separate field for package_name, and its transfer is crucial for sources. Of course, you can create a “crutch” using a different text field for this parameter, which is present in the tracker, but not really needed for feeds - but it would be great to make everything beautiful. Well, we are sure that our colleagues will not let us down and will remove the restrictions that are currently present.
Conclusion
In the process of constant improvements to different systems, it is often necessary to think through and rewrite those modules that were responsible for the logic of work, since it can change several times.
It all depends on the developer. Our employee thought out the modularity and extensibility of components. It is important not to sacrifice this for the sake of low development time, because if you make a monolithic system, the improvements will take a lot of time.
Despite all the difficulties encountered in the development of their own solutions, the team never regretted that they decided to spend their own resources and funds on this project. The time that was ultimately saved, and the unique advantages that we received, repeatedly paid off all the development costs. Perhaps we will think about preparing a public version of the product so that anyone can appreciate our work.
For now, if you want to get excellent traffic from trusted sources, or you yourself have a good inventory of traffic and are looking for a suitable set of offers for it, we will be happy to talk about potential cooperation.
Subscribe and read interesting news, analytics and other insights about mobile marketing in the Mobio blog .
Source: https://habr.com/ru/post/339298/
All Articles