Dreams of an ideal API or how difficulties were overcome in the AdHands project
Users are accustomed to the principle of one device and one interface. This trend is also reflected in business: multifunctional corporate systems and portals, single trading and procurement platforms, etc. Online advertising has not stood aside - services are being developed aimed at significantly optimizing the work of advertisers and advertising agencies. And one of the serious technical issues in creating dashboards and statistics aggregators is working with the API, a software interface through which data is exchanged between systems. Is there a dream API, how to create it and why is it needed?
Advertising on the Internet passes through several channels at once and is placed on a variety of sites to reach the widest possible audience. Campaign management is carried out through the interfaces of various advertising systems (Yandex.Direct, Google AdWords, Vkontakte placement system, etc.). Each time to view statistics, you need to log in, switch, and in case you need to compare statistics from different sites, you also have to aggregate data in Excel tables or manually import data into Google Analytics.
Realweb, like no one, knows about the need for convenient and efficient work with online advertising data: we run hundreds of clients with tens of thousands of campaigns, so a visual representation of statistics is a critical factor for us. For this we use the system of collecting advertising statistics AdHands . This is a kind of dashboard: information on all advertising campaigns is aggregated in a single interface. This approach provides convenience, broad analytical capabilities and an accurate understanding of which advertising is effective and which is not.
')
In the case of advertising on the Internet to assess the "exhaust" from it is extremely difficult. In the courtyard of the XXI century, gentlemen! And for measuring the performance of advertising on the Internet there are interesting functional tools that are ready to work through the API with all services.
AdHands are one such solution. This is a system that connects to online advertising sites (Yandex.Direct, Yandex.Market, Google Adwords, Vkontakte, Target@Mail.ru, DoubleClick Bid Manager, Yandex.Auction, etc.) via an API.
We have been thinking for a long time what kind of unified advertising campaign management system on the Internet should be. As a result, it was decided to develop a system that can overcome several problems at once. The first and main feature of AdHands was a single interface for accessing all collected statisticians - this solution made it possible to eliminate the need to log in and work in different GUIs each time. It turned out a single convenient panel, dashboard, on which all the necessary advertising indicators are presented.
Of course, no statistics have practical value if they are not aggregated and not processed properly. The system has been added the ability to automatically calculate the cost of a goal, order, call, ROI calculation. Media planning was introduced - a valuable feature for those who manage other people's budgets or have a rigid business planning system in a company, within which it is necessary to calculate the marketing and advertising budget in advance.
AdHands works for the end user, making his life a lot easier. However, when creating the system itself, a number of difficulties arise due to the fact that it is necessary to interact with various external systems.
Everyone knows how complicated the world of programming and development is - despite the creation of permanent descriptions of standards and attempts to agree, each developer introduces something in his code. In the case of the API, this sometimes puzzles the developers of external systems, for example, AdHands. We will tell you more about what we have encountered, using other people's APIs when creating our online advertising management system.
Working with third-party systems, we have developed a number of criteria by which we determine how convenient the system that we connect to AdHands. Broadly speaking, such a system should first of all be convenient from the point of view of the developer. And if we consider in particular, it is important that the connected system meets several parameters.
Of course, ideally, it would be desirable for all the connected systems to have a uniform universal API and one code for all systems could be used. Certain steps have already been taken in this direction, but until complete unification is so far that for the time being it seems impossible.
Discrepancy in the API terminology of sites - different systems may not have an entity or be named differently. For example, Yandex.Direct recently introduced the ad group entity, and for a long time before that, only AdWords had such an entity. A similar situation arises, for example, with the term “creative”, which occurs in BaYaN and DBM, but in fact for the developer, the creative in these two systems are different entities. Overcoming such discrepancies takes time and therefore has to find a solution using the construction of a special AdHands architecture. However, this problem is not as significant as the others.
Differences in the data structure are a problem arising from differences in the structure of the sites themselves (again, for example, there may be no ad groups). This is an important question for AdHands, because we do not copy the interfaces of the systems in one application, but collect data for presentation to the user. I do not want to change the structure of the pages each time, adjusting to the systems. We circumvent the problem by bringing data from various sources to a single structure. In fact, we create hierarchical and hybrid solutions that are similar in structure, when the entity remains available when choosing one system and closed to others. AdHands achieve uniformity and convenience of the interface by abstracting entities.
All developers refer to documentation in different ways: someone thoroughly documents each variable, someone describes in general, and some refuse any documentation. When working with the external system API, the availability of detailed documentation is the key to success. However, even with detailed manuals and a community of developers, there are problems, among which the main one is the discrepancy between the documentation and the real situation : sometimes it happens that documentation, software interface and graphical user interface disagree. As a rule, this is due to the fact that the API is changing and supplemented quickly enough, and the documentation is updated more slowly, so you can find outdated functions in it or not find something from the new features. Ideally, the release of a new version of the API occurs simultaneously with the release of the documentation.
From the experience of Realweb, we can say that there is practically no documentation that does not correspond to reality. However, the human factor comes into play here, and sometimes due to the lack of notifications and newsletters about changes, it is not immediately possible to find out.
Each API system is periodically modified and refined. Constant change of the API itself is a necessary phenomenon - the software interface develops with the system. It is bad if changes come out, but, let's say, the signature of methods has not changed. It would be nice if during the update of the API all libraries were updated according to the schedule.
The discrepancy between the API and the interface is not a very common problem. Of course, the API provider understands that the entities must be the same for both the software interface and the user interface. It happens, the API does not provide methods for some elements of the graphical interface. If there are no entities in the software interface, you have to create pseudo-entities, a bundle in order to fit into the universal structure of AdHands.
It is extremely rare, but it still happens that there is no API at all . For example, we encountered such a situation while working with the CMS banner placement system of Yandex. In such cases, you have to parse the interface using a crawler and log in as a user through the login / password combination. Here the main stumbling block occurs if the graphical user interface changes - the layout changes, the parser returns errors. But, basically, all modern systems provide access through the API. But those who are not ready to provide adequate libraries and documentation, risk remaining outside the competition.
The lack of convenient and universal OAuth 2.0 authorization is a problem that continues to occur. As you know, the OAuth 2.0 authorization protocol is in the finalization of the specification , but it is already quite ready to replace OAuth. This is a convenient protocol that is distinguished by its ease of use, the absence of a long chain of requests and complex signature schemes. Some platforms and advertising aggregators do not use it yet, apparently, in anticipation of a stable specification. This greatly complicates the process of authorization and authentication when requesting data from external systems and transferring data. For example, BaYan from Yandex works on tokenization by login / password. From the point of view of the developer, this is not completely safe, since it can cause appreciable damage if the base in which the usernames and passwords of access to BAYAN are compromised is compromised.
Now most of the major systems have already switched to OAuth 2.0. Besides the fact that it is safe, it is still a universal standard and a guarantee of quick connection of the system to AdHands.
The lack of a unified universal data exchange protocol is another problem. In case it existed, AdHands would simply forward these systems as a single control panel. However, if such a protocol is created on the market, there will be a lot of companies willing to build a business for innovation, as it usually happens. And this may lead to ill-conceived solutions that will compromise serious systems in the eyes of the end user. In addition, the creation of a single protocol is not very profitable for large sites that are not ready and are not going to give up their graphical interfaces.
Another problem that we can have little influence on is the API limits — the limited number of requests per day that ad systems set. Each API method has its own limit of calls, which is determined by the resource intensity and the “weight” of the method. In principle, Google AdWords and Yandex.Direct have so many requests that it’s difficult to feel the restriction, but there are also problem services, for example, Yandex.Market, which sometimes recalls an attempt to get a ticket to the right doctor. Sites often meet and extend the limits, however, attention is also paid to the structure of the aggregator systems themselves, which may not optimally use queries.
Of course, this is not a complete list of problems and minor obstacles that AdHands developers face every day. We are always looking for compromises and are not afraid to use solutions in the code that may not seem very elegant to someone. This is due to the large number of joining systems, the diversity of their GUI and API. We write parsers for systems without an API, create unnecessary entities, introduce pieces of code with checks, implement a new functionality that is not embedded in AdHands. All these decisions can have an impact on the manageability and ease of administration of the system, on its speed. But at the same time, we continuously optimize the code, adapt to the system of limits and do everything for the convenience of our users.
Recently, there has been a tendency to unify the API of large sites. For example, the prototype Yandex.Metrica API in the Core reporting part copies the Google Analytics API. We hope that other API providers will not live their lives, but will look at the best practices and follow not a fashionable, but a very useful trend. And by joint efforts we will do better for those for whom everything was started, for end users, advertising agencies and advertisers. Most of them do not know about the API and its problems, they need to quickly, conveniently and clearly. Our business is to provide it.
Advertising on the Internet passes through several channels at once and is placed on a variety of sites to reach the widest possible audience. Campaign management is carried out through the interfaces of various advertising systems (Yandex.Direct, Google AdWords, Vkontakte placement system, etc.). Each time to view statistics, you need to log in, switch, and in case you need to compare statistics from different sites, you also have to aggregate data in Excel tables or manually import data into Google Analytics.
Realweb, like no one, knows about the need for convenient and efficient work with online advertising data: we run hundreds of clients with tens of thousands of campaigns, so a visual representation of statistics is a critical factor for us. For this we use the system of collecting advertising statistics AdHands . This is a kind of dashboard: information on all advertising campaigns is aggregated in a single interface. This approach provides convenience, broad analytical capabilities and an accurate understanding of which advertising is effective and which is not.
')
In the case of advertising on the Internet to assess the "exhaust" from it is extremely difficult. In the courtyard of the XXI century, gentlemen! And for measuring the performance of advertising on the Internet there are interesting functional tools that are ready to work through the API with all services.
AdHands - long arm online advertising
AdHands are one such solution. This is a system that connects to online advertising sites (Yandex.Direct, Yandex.Market, Google Adwords, Vkontakte, Target@Mail.ru, DoubleClick Bid Manager, Yandex.Auction, etc.) via an API.
We have been thinking for a long time what kind of unified advertising campaign management system on the Internet should be. As a result, it was decided to develop a system that can overcome several problems at once. The first and main feature of AdHands was a single interface for accessing all collected statisticians - this solution made it possible to eliminate the need to log in and work in different GUIs each time. It turned out a single convenient panel, dashboard, on which all the necessary advertising indicators are presented.
Of course, no statistics have practical value if they are not aggregated and not processed properly. The system has been added the ability to automatically calculate the cost of a goal, order, call, ROI calculation. Media planning was introduced - a valuable feature for those who manage other people's budgets or have a rigid business planning system in a company, within which it is necessary to calculate the marketing and advertising budget in advance.
AdHands works for the end user, making his life a lot easier. However, when creating the system itself, a number of difficulties arise due to the fact that it is necessary to interact with various external systems.
Working with API - building bridges
Everyone knows how complicated the world of programming and development is - despite the creation of permanent descriptions of standards and attempts to agree, each developer introduces something in his code. In the case of the API, this sometimes puzzles the developers of external systems, for example, AdHands. We will tell you more about what we have encountered, using other people's APIs when creating our online advertising management system.
Working with third-party systems, we have developed a number of criteria by which we determine how convenient the system that we connect to AdHands. Broadly speaking, such a system should first of all be convenient from the point of view of the developer. And if we consider in particular, it is important that the connected system meets several parameters.
- The stability of the system is its availability 99.9% of the time in a year. It is probably unnecessary to explain how important this is for the system that collects and transmits the data of online advertising campaigns that are broadcast every second across the Internet.
- Availability of documentation is an absolutely critical requirement. Understanding the API without documentation is extremely difficult, especially if you have to handle errors and abnormal situations. You need to understand that the API is actually someone else's code, and despite the desire of the largest online platforms for universality, the happy future is still far away and the API must be accompanied by detailed developer documentation, for example, as implemented by Google AdWords and Yandex.Direct.
- OAuth 2.0 authorization support - a convenient and secure open authorization protocol.
- Responsive technical support of the system, of course, can not be a requirement, but this wish - unfortunately, it is not always possible to quickly get a qualified answer. And solving problems in such systems as AdHands requires maximum efficiency in order to avoid losing customer loyalty.
- Understandable - API methods and functionality must match the semantics of what they do. For example, the “add user” method should add a user without solving a number of other tasks.
Of course, ideally, it would be desirable for all the connected systems to have a uniform universal API and one code for all systems could be used. Certain steps have already been taken in this direction, but until complete unification is so far that for the time being it seems impossible.
Working with API - let's talk about problems
Discrepancy in the API terminology of sites - different systems may not have an entity or be named differently. For example, Yandex.Direct recently introduced the ad group entity, and for a long time before that, only AdWords had such an entity. A similar situation arises, for example, with the term “creative”, which occurs in BaYaN and DBM, but in fact for the developer, the creative in these two systems are different entities. Overcoming such discrepancies takes time and therefore has to find a solution using the construction of a special AdHands architecture. However, this problem is not as significant as the others.
Differences in the data structure are a problem arising from differences in the structure of the sites themselves (again, for example, there may be no ad groups). This is an important question for AdHands, because we do not copy the interfaces of the systems in one application, but collect data for presentation to the user. I do not want to change the structure of the pages each time, adjusting to the systems. We circumvent the problem by bringing data from various sources to a single structure. In fact, we create hierarchical and hybrid solutions that are similar in structure, when the entity remains available when choosing one system and closed to others. AdHands achieve uniformity and convenience of the interface by abstracting entities.
All developers refer to documentation in different ways: someone thoroughly documents each variable, someone describes in general, and some refuse any documentation. When working with the external system API, the availability of detailed documentation is the key to success. However, even with detailed manuals and a community of developers, there are problems, among which the main one is the discrepancy between the documentation and the real situation : sometimes it happens that documentation, software interface and graphical user interface disagree. As a rule, this is due to the fact that the API is changing and supplemented quickly enough, and the documentation is updated more slowly, so you can find outdated functions in it or not find something from the new features. Ideally, the release of a new version of the API occurs simultaneously with the release of the documentation.
From the experience of Realweb, we can say that there is practically no documentation that does not correspond to reality. However, the human factor comes into play here, and sometimes due to the lack of notifications and newsletters about changes, it is not immediately possible to find out.
Each API system is periodically modified and refined. Constant change of the API itself is a necessary phenomenon - the software interface develops with the system. It is bad if changes come out, but, let's say, the signature of methods has not changed. It would be nice if during the update of the API all libraries were updated according to the schedule.
The discrepancy between the API and the interface is not a very common problem. Of course, the API provider understands that the entities must be the same for both the software interface and the user interface. It happens, the API does not provide methods for some elements of the graphical interface. If there are no entities in the software interface, you have to create pseudo-entities, a bundle in order to fit into the universal structure of AdHands.
It is extremely rare, but it still happens that there is no API at all . For example, we encountered such a situation while working with the CMS banner placement system of Yandex. In such cases, you have to parse the interface using a crawler and log in as a user through the login / password combination. Here the main stumbling block occurs if the graphical user interface changes - the layout changes, the parser returns errors. But, basically, all modern systems provide access through the API. But those who are not ready to provide adequate libraries and documentation, risk remaining outside the competition.
The lack of convenient and universal OAuth 2.0 authorization is a problem that continues to occur. As you know, the OAuth 2.0 authorization protocol is in the finalization of the specification , but it is already quite ready to replace OAuth. This is a convenient protocol that is distinguished by its ease of use, the absence of a long chain of requests and complex signature schemes. Some platforms and advertising aggregators do not use it yet, apparently, in anticipation of a stable specification. This greatly complicates the process of authorization and authentication when requesting data from external systems and transferring data. For example, BaYan from Yandex works on tokenization by login / password. From the point of view of the developer, this is not completely safe, since it can cause appreciable damage if the base in which the usernames and passwords of access to BAYAN are compromised is compromised.
Now most of the major systems have already switched to OAuth 2.0. Besides the fact that it is safe, it is still a universal standard and a guarantee of quick connection of the system to AdHands.
The lack of a unified universal data exchange protocol is another problem. In case it existed, AdHands would simply forward these systems as a single control panel. However, if such a protocol is created on the market, there will be a lot of companies willing to build a business for innovation, as it usually happens. And this may lead to ill-conceived solutions that will compromise serious systems in the eyes of the end user. In addition, the creation of a single protocol is not very profitable for large sites that are not ready and are not going to give up their graphical interfaces.
Another problem that we can have little influence on is the API limits — the limited number of requests per day that ad systems set. Each API method has its own limit of calls, which is determined by the resource intensity and the “weight” of the method. In principle, Google AdWords and Yandex.Direct have so many requests that it’s difficult to feel the restriction, but there are also problem services, for example, Yandex.Market, which sometimes recalls an attempt to get a ticket to the right doctor. Sites often meet and extend the limits, however, attention is also paid to the structure of the aggregator systems themselves, which may not optimally use queries.
Of course, this is not a complete list of problems and minor obstacles that AdHands developers face every day. We are always looking for compromises and are not afraid to use solutions in the code that may not seem very elegant to someone. This is due to the large number of joining systems, the diversity of their GUI and API. We write parsers for systems without an API, create unnecessary entities, introduce pieces of code with checks, implement a new functionality that is not embedded in AdHands. All these decisions can have an impact on the manageability and ease of administration of the system, on its speed. But at the same time, we continuously optimize the code, adapt to the system of limits and do everything for the convenience of our users.
Recently, there has been a tendency to unify the API of large sites. For example, the prototype Yandex.Metrica API in the Core reporting part copies the Google Analytics API. We hope that other API providers will not live their lives, but will look at the best practices and follow not a fashionable, but a very useful trend. And by joint efforts we will do better for those for whom everything was started, for end users, advertising agencies and advertisers. Most of them do not know about the API and its problems, they need to quickly, conveniently and clearly. Our business is to provide it.
Source: https://habr.com/ru/post/262573/
All Articles