Translation: What I learned by developing an API for the Microsoft cloud platform
Your attention is invited to the translation of the post about how the Windows Azure API was developed: about the difficulties, successful and unsuccessful decisions, conclusions. Next - the text of the author.
After talking with my work colleague on the REST API, I had the idea to tell about my experience in organizing and working the team that created the Windows Azure Service Management API. I was inspired to write this post by such great articles in the “what I learned” genre, like this one by Foursquare and this one by Daniel Jacobs from Netflix.
Warning: Everything told under the cut is my personal opinion. I'm not even sure that the other team members agree with me. And I know for sure that some of the thoughts expressed are rather contradictory.
The Windows Azure Service Management API (SMAPI) was created in 2008 and allowed the developer to control his Windows Azure account - deploy a new code, change settings, scale services, manage storage, etc. By the way, here is the documentation . I can’t name specific numbers, but I can say that the API handled a huge number of requests from different clients every day - development environments, utilities, automation systems, intruder bots, and many other sources. I was lucky to work on this API together with an amazing team (taking this opportunity to say hello to Zhe, Dipa, Frank!) And I am proud of what we managed to create.
')
What have I learned during this time?
By saying this, I risk incurring the wrath of many friends and colleagues. At the very beginning of work on the API, we spent weeks to create the most “clean” URL design. We all read the work of Fielding and for us it was the norm to wave this book during particularly heated discussions. But in the end I realized - we just wasted time.
And that's why. Developers who will use your API will not see HTTP requests and responses. If your API is at least as successful, then it will be made binding to programming languages ​​- by you or by the community. Of course, the ability to wake up at three o'clock in the morning and build a curl request from memory will make the life of the binding developers a little easier. As well as ignoring basic REST idioms (for example, state-changing GET requests) will cause developers to suffer. But if every second meeting slips into discussions whether to use PUT or POST, and this lasts for weeks - then you lose your time for academic discussions.
Proof of? Look at the Amazon API (not S3, but everything else that is there). All of these APIs are the ugliest RPC-style XML monsters. But you will not deny how successful this API is?
One of the best things we did while working on the API was that we started with code that uses this API, and looking at it, we made a REST interface. Since the API itself was not there yet, we simply wrote pseudocode in different programming languages ​​for important scripts. This allowed us to avoid a large number of flaws that were not obvious during the simple reading of the API specification. For example, we found that we are trying to force developers to create too many intermediate objects between meaningful API calls. We also found that users do not have the ability to send structures received via an API back without having to modify them. Without a preliminary pseudo-code check, we would not be able to detect all these flaws that would surface much later and fixing them would be much more expensive. Or even worse - we could make a release of such an API and be a long time using a bad design.
We introduced the rule that the developer does not offer API functions, in the comments to which there are not at least a few lines of pseudo-code to understand how they work. This rule also improved API design meetings, where developers could see usage examples instead of the usual parameter lists and return values.
A bad way to create an API focuses on the REST interface to the service internals. The resulting “API” is usually thrown to grateful users, through a wall high enough so as not to hear what they think about it (the translator's note is an ideological expression, “to throw over the wall”). We see around these many APIs created by Microsoft and other companies, and all of these APIs hurt developers.
Try to minimize the number of “entities” that your API gives out. Each concept that your API operates on increases the complexity of understanding for developers and has a very negative effect on the learning curve. We adhered to an aggressive policy of cutting out “entities” sticking out, even if you had to combine slightly different “entities” into one. We also aggressively reduced the number of operations available for each “entity”. All this led to a significant simplification of the API compared with the very first design.
One of the problems we encountered in the late stages of an API design is known as the “N + 1 problem”. This problem occurs when, in order to access child objects, you first need to get their list from the parent object, and then make a separate HTTP request for each one received. In our case, the most popular operation was to request Windows Azure services and then obtain their status. Since we were already preparing for the release, we didn’t have time to change the API design, so we did what I called the “huge crutch” - the “recursive” parameter for the parent object. Surprisingly, this crutch turned out to be a very effective and convenient solution - both for users of our API, and for our servers. Another similar thing is “partial responses”, something that is currently supported within GData . In each such case, the implementation was not the most successful, but the very idea of ​​honing the API for key usage scenarios helped us a lot. Do not be surprised if these “crutches” in our design save millions of API calls every day. And allow you to quickly create convenient clients to our platform.
One of the good ways to achieve such results is to develop customer prototypes in parallel with the development of the API itself. For example, we supported client versions for publishing assemblies, clients for monitoring, and clients for other popular scenarios. It's amazing how badly your freshly prepared API becomes as soon as you start writing code that uses it. As an API designer, it's best for you to see this before thankful users get to your API.
What we did really well was to take measurements. We have instrumented our API for removing metrics and very closely following the resulting numbers. Every day we knew what users were doing, what the most popular API calls were, the most common errors, and so on. This gave us an understanding of how our API is actually used (for example, it allowed us to say how many calls the N + 1 hack saved). With the release of new functionality, we tracked how it is used and quickly made adjustments if the harsh truth of life was different from the use we had intended. And, perhaps most importantly, we had the opportunity to track which errors users encounter and carefully study them. We often added or modified error messages so that users could understand what is happening while working with the system. For a long time, all this helped us improve the usability of the API.
APIs are changing, and it’s hard to predict what changes they will make. Sometimes the changes are small - you want to add a field to an object or another element to the list of elements. Other changes are more difficult - you want to change the authentication mechanism or stop supporting the old version of the API. In any case, you will be grateful to yourself if you take care of versioning from the very beginning. There are many ways to protect your API from change related issues. You can go our way: the client sends the supported API version in the header, and the server demonstrates the expected behavior for this version. Another way is to add the version number in the URL of the REST methods. In any case, adding any protection method to the very first version of the API will save you from a big headache afterwards.
We should also consider the “small changes” in the API. I think we could make Azure SMAPI better from the start. For example, API Foursquare uses an interesting way to indirectly expand structures:
This allows them to add new “types” without losing compatibility with existing customers who are not expecting such types. It’s a pity that we didn’t do something similar for the Azure API. This would seriously simplify our lives in case of minor changes and would eliminate the need to constantly increase the version number and break compatibility with old customers.
Sriram krishnan
me@sriramk.com
After talking with my work colleague on the REST API, I had the idea to tell about my experience in organizing and working the team that created the Windows Azure Service Management API. I was inspired to write this post by such great articles in the “what I learned” genre, like this one by Foursquare and this one by Daniel Jacobs from Netflix.
Warning: Everything told under the cut is my personal opinion. I'm not even sure that the other team members agree with me. And I know for sure that some of the thoughts expressed are rather contradictory.
Foreword
The Windows Azure Service Management API (SMAPI) was created in 2008 and allowed the developer to control his Windows Azure account - deploy a new code, change settings, scale services, manage storage, etc. By the way, here is the documentation . I can’t name specific numbers, but I can say that the API handled a huge number of requests from different clients every day - development environments, utilities, automation systems, intruder bots, and many other sources. I was lucky to work on this API together with an amazing team (taking this opportunity to say hello to Zhe, Dipa, Frank!) And I am proud of what we managed to create.
')
What have I learned during this time?
Naming a REST URI is irrelevant ... almost
By saying this, I risk incurring the wrath of many friends and colleagues. At the very beginning of work on the API, we spent weeks to create the most “clean” URL design. We all read the work of Fielding and for us it was the norm to wave this book during particularly heated discussions. But in the end I realized - we just wasted time.
URI + HTTP verb + wire format does not matter
And that's why. Developers who will use your API will not see HTTP requests and responses. If your API is at least as successful, then it will be made binding to programming languages ​​- by you or by the community. Of course, the ability to wake up at three o'clock in the morning and build a curl request from memory will make the life of the binding developers a little easier. As well as ignoring basic REST idioms (for example, state-changing GET requests) will cause developers to suffer. But if every second meeting slips into discussions whether to use PUT or POST, and this lasts for weeks - then you lose your time for academic discussions.
Proof of? Look at the Amazon API (not S3, but everything else that is there). All of these APIs are the ugliest RPC-style XML monsters. But you will not deny how successful this API is?
Start with how your API will be used.
One of the best things we did while working on the API was that we started with code that uses this API, and looking at it, we made a REST interface. Since the API itself was not there yet, we simply wrote pseudocode in different programming languages ​​for important scripts. This allowed us to avoid a large number of flaws that were not obvious during the simple reading of the API specification. For example, we found that we are trying to force developers to create too many intermediate objects between meaningful API calls. We also found that users do not have the ability to send structures received via an API back without having to modify them. Without a preliminary pseudo-code check, we would not be able to detect all these flaws that would surface much later and fixing them would be much more expensive. Or even worse - we could make a release of such an API and be a long time using a bad design.
We introduced the rule that the developer does not offer API functions, in the comments to which there are not at least a few lines of pseudo-code to understand how they work. This rule also improved API design meetings, where developers could see usage examples instead of the usual parameter lists and return values.
A bad way to create an API focuses on the REST interface to the service internals. The resulting “API” is usually thrown to grateful users, through a wall high enough so as not to hear what they think about it (the translator's note is an ideological expression, “to throw over the wall”). We see around these many APIs created by Microsoft and other companies, and all of these APIs hurt developers.
Do not make the developer think a lot
Try to minimize the number of “entities” that your API gives out. Each concept that your API operates on increases the complexity of understanding for developers and has a very negative effect on the learning curve. We adhered to an aggressive policy of cutting out “entities” sticking out, even if you had to combine slightly different “entities” into one. We also aggressively reduced the number of operations available for each “entity”. All this led to a significant simplification of the API compared with the very first design.
Make key actions quick and easy.
One of the problems we encountered in the late stages of an API design is known as the “N + 1 problem”. This problem occurs when, in order to access child objects, you first need to get their list from the parent object, and then make a separate HTTP request for each one received. In our case, the most popular operation was to request Windows Azure services and then obtain their status. Since we were already preparing for the release, we didn’t have time to change the API design, so we did what I called the “huge crutch” - the “recursive” parameter for the parent object. Surprisingly, this crutch turned out to be a very effective and convenient solution - both for users of our API, and for our servers. Another similar thing is “partial responses”, something that is currently supported within GData . In each such case, the implementation was not the most successful, but the very idea of ​​honing the API for key usage scenarios helped us a lot. Do not be surprised if these “crutches” in our design save millions of API calls every day. And allow you to quickly create convenient clients to our platform.
One of the good ways to achieve such results is to develop customer prototypes in parallel with the development of the API itself. For example, we supported client versions for publishing assemblies, clients for monitoring, and clients for other popular scenarios. It's amazing how badly your freshly prepared API becomes as soon as you start writing code that uses it. As an API designer, it's best for you to see this before thankful users get to your API.
Measure and log all
What we did really well was to take measurements. We have instrumented our API for removing metrics and very closely following the resulting numbers. Every day we knew what users were doing, what the most popular API calls were, the most common errors, and so on. This gave us an understanding of how our API is actually used (for example, it allowed us to say how many calls the N + 1 hack saved). With the release of new functionality, we tracked how it is used and quickly made adjustments if the harsh truth of life was different from the use we had intended. And, perhaps most importantly, we had the opportunity to track which errors users encounter and carefully study them. We often added or modified error messages so that users could understand what is happening while working with the system. For a long time, all this helped us improve the usability of the API.
Versioning and Extensibility
APIs are changing, and it’s hard to predict what changes they will make. Sometimes the changes are small - you want to add a field to an object or another element to the list of elements. Other changes are more difficult - you want to change the authentication mechanism or stop supporting the old version of the API. In any case, you will be grateful to yourself if you take care of versioning from the very beginning. There are many ways to protect your API from change related issues. You can go our way: the client sends the supported API version in the header, and the server demonstrates the expected behavior for this version. Another way is to add the version number in the URL of the REST methods. In any case, adding any protection method to the very first version of the API will save you from a big headache afterwards.
We should also consider the “small changes” in the API. I think we could make Azure SMAPI better from the start. For example, API Foursquare uses an interesting way to indirectly expand structures:
{ type: "special", item: { special: { ...specialdata... } } } This allows them to add new “types” without losing compatibility with existing customers who are not expecting such types. It’s a pity that we didn’t do something similar for the Azure API. This would seriously simplify our lives in case of minor changes and would eliminate the need to constantly increase the version number and break compatibility with old customers.
Sriram krishnan
me@sriramk.com
Source: https://habr.com/ru/post/273533/
All Articles