5 stages of the API: what we understood by writing two versions
Today we want to talk about the secret - we have an API.
We wrote and then rewrote it again for four years. And during this time, almost all the classical stages of “accepting the inevitable” have passed. Except for one - the fourth. And we want to share theoverwhelmingly hard-earned conclusions about what to do and not to do if you decide to make your “powerful epiai”.
')
The process of creating API uCoz sometimes resembled the plot of the series The Knick (Nikerkoker Hospital) - with unsuccessful operations, guts and experiments on living people.
In general, API for site designers is a rarity even now. And in 2010, no one else had such a tool on the market.
On the one hand, there was a definite need for an API: we saw that people actively googled “scripts for uCoz”, received requests directly - some users wanted to create their own additions and modifications. On the other hand, and this was a problem , in those days all resources were thrown onto other projects.
The question "to be or not to be" was solved simply. The API was needed by us — to run the PHP support function in the constructor. We singled out one developer, and in six months he made our “initial API” - it was a get-only interface, to which it was possible to get page data from 11 modules. Data was given only in XML. By the time of the planned PHP announcement, we didn’t have time to establish at least the addition of content, but the API had some advantages: with it, you could run php scripts in a free version of Yukoza.
In general, we went to the users. It turned out straight on the classics:
They didn’t accept us very much ... Or rather, they praised me mostly. But for PHP. And in the “epiay” people did not see what they imagined with this magic word. The tech support started asking questions: “How to add materials? How to edit? How to get in json? ”And all this was not.
To the question that the API needs to be developed, they returned to the company somewhere in a year. The priorities were mobile clients - and we decided to write everything anew taking into account the requirements of the guys who made clients for iOS and Android. The initial version remained to live on its own (and even still alive, because some people still use it), and we began to pick up the performer for a new project.
“Myself a manager.” An intelligent guy Ilya came to the Rostov office: he knew Perl, on which part of the old uCoz was written, and when he was traditionally offered several tasks, he chose to work on the API. The problem was that at the time the guy had to become his own manager.
Then anger came: “As it turned out in the process, I understood the Perl syntax, but the spirit did not.
Therefore, for a long time I tried not to work, but to make the world better: I thought to rewrite everything, to introduce objects-classes-patterns. It took a lot of time to get comfortable with the ideology of the language. And then, what at first seemed scary and I wanted to rewrite, it seemed to be the norm. ”
By the time of enlightenment, a manager appeared and began to demand a report. And only content delivery by several modules was ready ... In general, once again anger, now on the other hand, - and Ilya was transferred to a new project. Its API version was never destined to see the light.
“The ideal learning challenge.” By the time the company tried to open an R & D office in Kazan. On-site, we needed a carrier of knowledge about the entire system - and the idea was to “grow” it through work on an API that would affect the main modules of the system.
So in this story Rinat appeared:
On the one hand, he was quite passionate to take on the project (he is a bit extreme at all). On the other hand, the work is calm and reasonable: not only 700+ parachute jumps, but also 20 years of experience in IT.
He was also familiar with Perl and had a fresh experience with other people's APIs — he integrated Metric and GA into the webmaster panel.
The first thing that came in handy to us was his judgment. Rinat asked for a pause in order to examine in more detail the intricate “guts” of the system, and only then announce the deadlines and the implementation plan. A month later, he came out with the following proposal:
* We rewrite and duplicate part of the functions purely under the API - during the creation of the system a lot of old code has accumulated in it. And some functions, if you try to shove something else in there, you would have turned out to be kilometer-long. So cleaned twins are needed purely under the API.
* We use REST - to simplify the query architecture, which will help increase performance.
* Use Oauth 1.0a - authentication, which seemed the most secure at the time.
* We give in different formats - JSON, XML, Text Plain.
* Well, get, post, put, delete, peace, labor, may ...
To a common understanding of how to make it convenient, we sort of came. But the devil was in the details and arguments.
The principal issue was the scope of tokens. Our server system, as Rinat says, “is a complete encapsulated figovina”: N users spin on each server, and when the space on it ends, a new hardware is taken.
Rinat wanted to use one token to access all sites. “Ok, let's talk about this,” and we called our colleagues to the Skype chat. As it happens in collective chats, they argued, argued, but did not come to a collective decision. And in the heat of development, the topic was forgotten and re-emerged when integration was ready for half of the modules.
We didn’t have another option on how to replicate the token: this is how an architecture with one main authorization server appeared - having received one token there, it can be used everywhere.
Designed as an array or object? For us, it was not a problem of discharging what is right and what is wrong. It was a problem of details.
For example, we give data in JSON - and there is a problem with typed languages. Not all structures received from the API are conveniently parsed - because the amount of code on the client side increases.
But the API is focused on web pribludy, so we listened to the opinion of Java and C ++ developers and came to the standard: we give the fields in any case, we supplement the named parameters with code.
Fields and parameters. Here the trades and discussions went all the time. For example, is it worth issuing an empty parameter? Since we were also oriented towards mobile development using the API, Rinat proved in the process that it was not necessary: ​​after all, traffic is important in mobile applications.
And what fields are needed? Here I, the project manager, had to look for and give examples from real practice. Often I took the help of the hall - I interviewed future API users. If a third-party case sent seemed valuable, we implemented the necessary fields in the API itself so that people would not do it through crutches. And later they came up with a more elegant solution - we will tell about it later.
The work scheduled for the year took almost two. In the process there were completely unforeseen difficulties. We quickly realized that:
Problems with the second developer. According to the plan, after Rinat made authorization, a second developer had to connect to the API integration with the modules.
We painted a google doc, putting the modules in alphabetical order. Work on them was divided between the performers in half. Defined schedule - one module per month from each. When it was time to start, the second man left. And we have already made a new project - the designer of uKit, to which the main resources were thrown. With the loss of a second programmer, almost 7 months were added to the implementation period.
Problems with testing. In theory, the process was conceived as follows: after the developer’s server, everything goes to the alpha server, and if the tester says “come on,” then it is being beta tested and the update is spread over a hundred servers.
But it turned out that our usual testers were not very suitable for working with the API - because they are “sharpened” for testing web pages. The API is a subtle thing. For example, when there was a change inside the module with which we integrated, something could fall off - but the testers did not notice this at first (then we taught them).
Then we opened 4 test servers and invited advanced users at the earliest stages. Such a crowd, and therefore not very controlled (you are not sure when a person will do something and will not quit), testing also affected the increase in the time frame.
In the end, we have come to terms with the inevitable - the API, like the repair, can be started, but it is not so easy to finish. Here are some recipes for how to organize the process to make it easier for you.
1. Get feedback. More communication.
In addition to the forum thread and blog posts, i.e. public means of exchanging comments, I started a section “Laboratory” on the site with new documentation. In fact, it was a feedback form where you could discuss your case in personal correspondence.
We informed the early users about the new API in February 2015, and completed the process of rolling out all the modules to the servers only this year. All this time we received reports, suggestions and interesting cases from users (which I also used in the “auction” with Rinat) through the “Laboratory”.
The flow of applications has decreased only in the last two months.
Each time, having received an appeal “how to do this or what to do here,” we took the time and sat down to watch, but what better? Sometimes interesting stories came from this.
2. Personally help those who did not understand or [Script as a gift].
Sometimes, the message from the “Lab” was so interesting that we started writing the script ourselves. And then gave the ready solution to the user,
What is the profit? As they say, “if you want to find a bug, be a bug.” And if you want to immerse yourself in a project, do as much as possible on it to check the performance and convenience. At the same time, the first solutions appeared on the uScript exchange with the use of uAPI - authorization through social networks and non-buggy output of pictures in the block of recommended materials.
3. Run an internal hackathon.
As you remember, we lacked testers. This is how the idea of ​​an internal hackathon for product verification appeared. For some of our developers, working with the API was a completely new and unusual task, so we had the opportunity to closely observe how people act with different skills.
Rinat did not usually communicate directly with users, but came to Kazan developers from the Kazan hackathon. He says, “with the exception of trolling, I was satisfied with the data collected in the process and criticism.”
4. Try to automate something.
My favorite feature for working with API was the semi-automatic creation of an application and a token.
Initially, this system appeared from my personal needs - at one moment I was tired of wasting time on signing requests while creating an application. Then I decided to write a semi-automatic mode - so that I do not have to think about anything, but I copied the data - and the application is ready. As it turned out, now people mostly use a semi-automatic solution.
5. Consult and change documentation.
Initially, we painted an example query in PHP and CURL.
It was. As it turned out in the process, no one used CURL.
But people asked, for example, in what format and which way to send the request. It became clear that the first implementation was not obvious to everyone.
We decided that there should be an additional module (written in a modern, OOP), where the signature for oauth will be automatically generated. Once I called this module - and then you just write the path and method of the request.
In parallel, I walked through our programmers and asked - the documentation for which APIs do they consider worthy role models? I looked at the recommended examples and compiled a new version of the documentation with their consideration:
And as a test before publication, I asked a colleague from our second office to check the compliance of the API work with the descriptions in the documentation for all methods.
6. Implement mobility.
Firstly, it will help you get good reviews and broaden your audience:
Secondly, it can be a valuable and new experience for the internal team. For example, on the hackathon, several of our guys wrote down this application:
Here you can see the application code.
7. Do not limit the desires of users.
Let's return to the question of fields. Since you can’t foresee all cases, in the end we made a query builder with which you can realize any bold fantasies on this topic:
Finally, we came to the conclusion that the API is a two-way traffic. Today, using uAPI, you can make authorization for any site, transfer materials from uCoz and vice versa, use only our database, but output data on a third-party site. The more user cases you take into the tool, the longer the implementation. But at the same time the area of ​​its application is growing.
PS
The whole process of rolling out and combat testing of the new version took us 14 months and 20 updates. Here is a visualization .
It happens, after the next update, they write to us: “When will you finish it already?” But the process is really very difficult to stop (we were not joking about the repair). Sometimes - for technical reasons: for example, when the system update requires a change in the API. And sometimes - for creative. For example, now we are thinking: when all the integrations with the modules are completed, why not work through the themes of changing the design and settings of the site via API?
We wrote and then rewrote it again for four years. And during this time, almost all the classical stages of “accepting the inevitable” have passed. Except for one - the fourth. And we want to share the
')
The process of creating API uCoz sometimes resembled the plot of the series The Knick (Nikerkoker Hospital) - with unsuccessful operations, guts and experiments on living people.
Stage One - Denial
According to the concept of the five stages:
At the first stage, a person refuses to accept what happened to him.
In general, API for site designers is a rarity even now. And in 2010, no one else had such a tool on the market.
On the one hand, there was a definite need for an API: we saw that people actively googled “scripts for uCoz”, received requests directly - some users wanted to create their own additions and modifications. On the other hand, and this was a problem , in those days all resources were thrown onto other projects.
The question "to be or not to be" was solved simply. The API was needed by us — to run the PHP support function in the constructor. We singled out one developer, and in six months he made our “initial API” - it was a get-only interface, to which it was possible to get page data from 11 modules. Data was given only in XML. By the time of the planned PHP announcement, we didn’t have time to establish at least the addition of content, but the API had some advantages: with it, you could run php scripts in a free version of Yukoza.
In general, we went to the users. It turned out straight on the classics:
They didn’t accept us very much ... Or rather, they praised me mostly. But for PHP. And in the “epiay” people did not see what they imagined with this magic word. The tech support started asking questions: “How to add materials? How to edit? How to get in json? ”And all this was not.
Stage Two - Anger
According to the concept of the five stages:
At this stage, there is aggression towards the whole world.
To the question that the API needs to be developed, they returned to the company somewhere in a year. The priorities were mobile clients - and we decided to write everything anew taking into account the requirements of the guys who made clients for iOS and Android. The initial version remained to live on its own (and even still alive, because some people still use it), and we began to pick up the performer for a new project.
“Myself a manager.” An intelligent guy Ilya came to the Rostov office: he knew Perl, on which part of the old uCoz was written, and when he was traditionally offered several tasks, he chose to work on the API. The problem was that at the time the guy had to become his own manager.
Then anger came: “As it turned out in the process, I understood the Perl syntax, but the spirit did not.
Therefore, for a long time I tried not to work, but to make the world better: I thought to rewrite everything, to introduce objects-classes-patterns. It took a lot of time to get comfortable with the ideology of the language. And then, what at first seemed scary and I wanted to rewrite, it seemed to be the norm. ”
By the time of enlightenment, a manager appeared and began to demand a report. And only content delivery by several modules was ready ... In general, once again anger, now on the other hand, - and Ilya was transferred to a new project. Its API version was never destined to see the light.
... Between the stages ...
“The ideal learning challenge.” By the time the company tried to open an R & D office in Kazan. On-site, we needed a carrier of knowledge about the entire system - and the idea was to “grow” it through work on an API that would affect the main modules of the system.
So in this story Rinat appeared:
On the one hand, he was quite passionate to take on the project (he is a bit extreme at all). On the other hand, the work is calm and reasonable: not only 700+ parachute jumps, but also 20 years of experience in IT.
He was also familiar with Perl and had a fresh experience with other people's APIs — he integrated Metric and GA into the webmaster panel.
The first thing that came in handy to us was his judgment. Rinat asked for a pause in order to examine in more detail the intricate “guts” of the system, and only then announce the deadlines and the implementation plan. A month later, he came out with the following proposal:
* We rewrite and duplicate part of the functions purely under the API - during the creation of the system a lot of old code has accumulated in it. And some functions, if you try to shove something else in there, you would have turned out to be kilometer-long. So cleaned twins are needed purely under the API.
* We use REST - to simplify the query architecture, which will help increase performance.
* Use Oauth 1.0a - authentication, which seemed the most secure at the time.
* We give in different formats - JSON, XML, Text Plain.
* Well, get, post, put, delete, peace, labor, may ...
Third Stage - Bargaining
According to the concept of the five stages:
Here you have thoughts about how to agree with someone about a better fate.
To a common understanding of how to make it convenient, we sort of came. But the devil was in the details and arguments.
The principal issue was the scope of tokens. Our server system, as Rinat says, “is a complete encapsulated figovina”: N users spin on each server, and when the space on it ends, a new hardware is taken.
Rinat wanted to use one token to access all sites. “Ok, let's talk about this,” and we called our colleagues to the Skype chat. As it happens in collective chats, they argued, argued, but did not come to a collective decision. And in the heat of development, the topic was forgotten and re-emerged when integration was ready for half of the modules.
We didn’t have another option on how to replicate the token: this is how an architecture with one main authorization server appeared - having received one token there, it can be used everywhere.
Designed as an array or object? For us, it was not a problem of discharging what is right and what is wrong. It was a problem of details.
For example, we give data in JSON - and there is a problem with typed languages. Not all structures received from the API are conveniently parsed - because the amount of code on the client side increases.
But the API is focused on web pribludy, so we listened to the opinion of Java and C ++ developers and came to the standard: we give the fields in any case, we supplement the named parameters with code.
Fields and parameters. Here the trades and discussions went all the time. For example, is it worth issuing an empty parameter? Since we were also oriented towards mobile development using the API, Rinat proved in the process that it was not necessary: ​​after all, traffic is important in mobile applications.
And what fields are needed? Here I, the project manager, had to look for and give examples from real practice. Often I took the help of the hall - I interviewed future API users. If a third-party case sent seemed valuable, we implemented the necessary fields in the API itself so that people would not do it through crutches. And later they came up with a more elegant solution - we will tell about it later.
The fourth stage - ... (and about terms)
According to the concept of the five stages:
In the fourth stage, you have depression. [We somehow passed this stage]
The work scheduled for the year took almost two. In the process there were completely unforeseen difficulties. We quickly realized that:
Problems with the second developer. According to the plan, after Rinat made authorization, a second developer had to connect to the API integration with the modules.
We painted a google doc, putting the modules in alphabetical order. Work on them was divided between the performers in half. Defined schedule - one module per month from each. When it was time to start, the second man left. And we have already made a new project - the designer of uKit, to which the main resources were thrown. With the loss of a second programmer, almost 7 months were added to the implementation period.
Problems with testing. In theory, the process was conceived as follows: after the developer’s server, everything goes to the alpha server, and if the tester says “come on,” then it is being beta tested and the update is spread over a hundred servers.
But it turned out that our usual testers were not very suitable for working with the API - because they are “sharpened” for testing web pages. The API is a subtle thing. For example, when there was a change inside the module with which we integrated, something could fall off - but the testers did not notice this at first (then we taught them).
Then we opened 4 test servers and invited advanced users at the earliest stages. Such a crowd, and therefore not very controlled (you are not sure when a person will do something and will not quit), testing also affected the increase in the time frame.
Fifth Stage - Humility (and Conclusions)
According to the concept of the five stages:
According to the canons, there is agreement with the inevitable
In the end, we have come to terms with the inevitable - the API, like the repair, can be started, but it is not so easy to finish. Here are some recipes for how to organize the process to make it easier for you.
1. Get feedback. More communication.
In addition to the forum thread and blog posts, i.e. public means of exchanging comments, I started a section “Laboratory” on the site with new documentation. In fact, it was a feedback form where you could discuss your case in personal correspondence.
We informed the early users about the new API in February 2015, and completed the process of rolling out all the modules to the servers only this year. All this time we received reports, suggestions and interesting cases from users (which I also used in the “auction” with Rinat) through the “Laboratory”.
The flow of applications has decreased only in the last two months.
Each time, having received an appeal “how to do this or what to do here,” we took the time and sat down to watch, but what better? Sometimes interesting stories came from this.
2. Personally help those who did not understand or [Script as a gift].
Sometimes, the message from the “Lab” was so interesting that we started writing the script ourselves. And then gave the ready solution to the user,
What is the profit? As they say, “if you want to find a bug, be a bug.” And if you want to immerse yourself in a project, do as much as possible on it to check the performance and convenience. At the same time, the first solutions appeared on the uScript exchange with the use of uAPI - authorization through social networks and non-buggy output of pictures in the block of recommended materials.
3. Run an internal hackathon.
As you remember, we lacked testers. This is how the idea of ​​an internal hackathon for product verification appeared. For some of our developers, working with the API was a completely new and unusual task, so we had the opportunity to closely observe how people act with different skills.
Rinat did not usually communicate directly with users, but came to Kazan developers from the Kazan hackathon. He says, “with the exception of trolling, I was satisfied with the data collected in the process and criticism.”
4. Try to automate something.
My favorite feature for working with API was the semi-automatic creation of an application and a token.
Initially, this system appeared from my personal needs - at one moment I was tired of wasting time on signing requests while creating an application. Then I decided to write a semi-automatic mode - so that I do not have to think about anything, but I copied the data - and the application is ready. As it turned out, now people mostly use a semi-automatic solution.
5. Consult and change documentation.
Initially, we painted an example query in PHP and CURL.
It was. As it turned out in the process, no one used CURL.
But people asked, for example, in what format and which way to send the request. It became clear that the first implementation was not obvious to everyone.
We decided that there should be an additional module (written in a modern, OOP), where the signature for oauth will be automatically generated. Once I called this module - and then you just write the path and method of the request.
In parallel, I walked through our programmers and asked - the documentation for which APIs do they consider worthy role models? I looked at the recommended examples and compiled a new version of the documentation with their consideration:
And as a test before publication, I asked a colleague from our second office to check the compliance of the API work with the descriptions in the documentation for all methods.
6. Implement mobility.
Firstly, it will help you get good reviews and broaden your audience:
It has long been a dream of creating a mobile application for your site. I will study uAPI, to finally bring it to life
Secondly, it can be a valuable and new experience for the internal team. For example, on the hackathon, several of our guys wrote down this application:
Here you can see the application code.
7. Do not limit the desires of users.
Let's return to the question of fields. Since you can’t foresee all cases, in the end we made a query builder with which you can realize any bold fantasies on this topic:
Finally, we came to the conclusion that the API is a two-way traffic. Today, using uAPI, you can make authorization for any site, transfer materials from uCoz and vice versa, use only our database, but output data on a third-party site. The more user cases you take into the tool, the longer the implementation. But at the same time the area of ​​its application is growing.
PS
The whole process of rolling out and combat testing of the new version took us 14 months and 20 updates. Here is a visualization .
It happens, after the next update, they write to us: “When will you finish it already?” But the process is really very difficult to stop (we were not joking about the repair). Sometimes - for technical reasons: for example, when the system update requires a change in the API. And sometimes - for creative. For example, now we are thinking: when all the integrations with the modules are completed, why not work through the themes of changing the design and settings of the site via API?
Source: https://habr.com/ru/post/282385/
All Articles