RESTful API for the server - doing it right (Part 2)
In the first part of the article, I briefly described the principles of RESTful and explained how to design the architecture of your server so that you can easily release new ones and stop supporting outdated versions of your API. In this part, I will briefly talk about HATEOAS and Hypermedia, and then talk about the role they can play in developing native applications for mobile devices. But the main topic of this article will be caching implementation (more precisely, server-side caching support). The target audience includes server software developers and, to some extent, iOS developers or other mobile platforms.
Currently, the HTTP API can be divided into
Here is a very good explanation of this from Jon Algermissen. Unfortunately, everyone who provides their API calls its RESTful service even though it is not.
So what is it, a real RESTful server? This is any hypertext / hypermedia based API. In other words, a third-party developer or client application should be able to obtain information about “other available resources” through the root URL API. In fact, this is the most important condition when implementing the RESTful API. In addition, a true RESTful server can only be considered one that adheres to the principle of HATEOAS.
')
Two basic principles to be followed when implementing HATEOAS are:
NOT!
Why? In the past, APIs were mostly written for use in web applications. These servers typically returned XHTML, and client applications ran in a browser. The browser in such a scheme is similar to your mobile client, which parses hyperlink resources, knows what a form is and how to present it to the user. In the case of a mobile application, when the response of your RESTful server contains a form from which you can send data to the server, you cannot (at least without additional efforts) convert it into native interface elements available on the platform. None of the platform providers, either Apple / Google or Microsoft, provide support for converting XHTML forms to UIViewController (on iOS) or Intent (on Android) or Silverlight Page (on Windows Phone). I recommend using hyperlink resources in response to GET requests and provide controller method calls (instead of forms) for all POST / PUT and DELETE requests. (For example: / friends / add or / venues / checkin) Controller method calls can be embedded in other answers (so that the client application or the developer can learn about them). Of course, this violates the principles of REST, but that's okay. It is better to make a quality product than to blindly follow standards that are of little use in practice. (This may lead to our API becoming HTTP-based Type 2).
Go to caching. Caching, as many believe, is basically a client task (or an intermediate proxy task). But you know what, at the cost of small efforts in developing the server-side, you can make your API fully meet the requirements of intermediate proxy caching? This means that you get free load balancing on their part. Everything you need is described in chapter 13 of the HTTP specification .
Not long ago, Art Taylor tweeted:
"I derived two new rules: 1) If your application slows down, add caching. 2) If the application is buggy, remove caching. Why is caching so difficult!"
But believe me, caching can be implemented without collisions with such problems. Two main principles that I recommend that you follow are:
In any client-server application, the server is a trustworthy source of information. When you download a resource (page or response) from the server API, the server sends to the client, among other things, some additional “hints” about how the client can cache the received resource. The server authoritatively indicates to the client when the cached information expires. These hints can be sent both programmatically and through server configuration. The expiration model is usually implemented through server configuration, while the validity model requires software implementation by the server-side developer. It is the developer who must decide when to use validity, and when the validity period is based on the type of resource being returned. The expiration model is usually used when the server can uniquely determine how long a particular resource will be valid. The validity model is used for all other cases. Later I will show you how to implement both of these models during server development. As soon as you deal with both, I will show when to use each of them.
Let's look at a common caching configuration. If you use nginx, you probably have something similar in the config:
nginx translates these settings into the appropriate HTTP header. In this case, the server sends the “Expires” or “Cache-Control: max-age = n” in the header for all images and expects the client to cache them for 7 days. This means that you will not need to request the same data for the next 7 days. Each of the common browsers (and intermediate proxies) takes this header into account and works as expected. Unfortunately, most of the Open Source image caching frameworks for iOS, including the popular SDWebImage, use a built-in caching mechanism that simply deletes images after n days. The problem is that such frameworks do not correspond to the model of validity and your client application using these frameworks has to resort to non-standard solutions (hacks). I will give an example showing what could go wrong here. Let's return to our “new Facebook”. When your user uploads an avatar to the server, he thinks that the changes will be reflected in all views. Some clever developers clear the local cache after successfully calling update-profile-image. (This means that all controllers must download the image from the server for a new one). Everything works great, you have reported to the project manager and in each submission the most recent profile picture is now displayed. However, this does not completely solve the problem. A new avatar of the user will be seen by his friends only after 7 days. Absolutely unacceptable. So how to solve it? As I said, you must accept the statement that only a server can be a source of reliable data. Do not use dishonest client tricks to update the cache by prematurely ending the expiration of the cached content.
Both Facebook and Twitter solve the problem of outdated profile images (after a new image has been uploaded) using the validity model. In the validity model, the server sends a unique resource identifier to the client and the client caches both the identifier and the response. In terms of HTTP, this unique identifier is called ETag. When you make a second request to the same resource, you must send it to the ETag. The server uses this identifier to check whether the resource you requested has changed since the last time it was accessed (remember, the server is the only reliable source). If the resource has really changed, it sends the latest copy. Otherwise, it sends 304 Not Modified. The cache validity model requires additional efforts from the developer in the development of both client and server parts. I will describe both of them further.
In fact, under iOS, if you use MKNetworkKit it does all the work automatically. But for developers for Android and Windows Phone, I will describe in detail how this should be implemented.
Cache validity model uses ETag and Last-Modified HTTP headers. Client side implementation is simpler than server side. If you receive an ETag with a resource, when you make a second request to receive it, send the ETag in the “IF-NONE-MATCH” header field. Similarly, if you received a “Last-Modified” resource, send it to the “IF-MODOFIED-SINCE” field of the header in subsequent requests. The server, on its part, decides when to use “ETag”, and when “Last-Modified”.
The implementation of the lifetime model is simple. Simply calculate the expiration date based on the header fields, “Expires” or “Cache-Control: max-age-n” and clear the cache when this date arrives.
Using ETag
ETag is usually calculated on a server using hashing algorithms. (Most high-level server languages ​​such as Java / C # / Scala have object hashing facilities.) Before generating a response, the server must calculate the object hash and add it to the ETag header field. Now, if the client actually sent an IF-NONE-MATCH in the request and this ETag is equal to what you calculated, send 304 Not Modified. Otherwise, create a response and send it with a new ETag.
Use Last-Modified
Implementing Last-Modified is not entirely straightforward. Let's imagine that our API has a call that returns a list of friends.
When you use ETag, you calculate the hash of an array of friends. When using Last-Modified, you must send the last modified date of this resource. Since this resource is a list, this date should be the date when you last added a new friend. This requires the organization developer to store the date of the last data change for each user in the database. A bit harder than ETag, but gives a big advantage in terms of performance.
When a client requests a resource for the first time, you send a complete list of friends. Subsequent requests from the client will now have the “IF-MODIFIED-SINCE” field in the header. Your server code should only send a list of friends added after the specified date. The access code to the database before the modification was something like this:
after modification it became so:
If the request does not return records, send 304 Not Modified. Thus, if a user has 300 friends and only two of them have been added recently, then the answer will contain only two entries. The time it takes for the server to process the request and the resources it consumes is significantly reduced.
Of course, this is a very simplified code. A developer will have a headache when you decide to support the removal or blocking of friends. The server must be able to send hints, using which the client will have the opportunity to tell which friends were added and which were deleted. This technique requires additional efforts in the development of the server part.
So. It was a difficult topic. Now I will try to summarize and derive the basic rules for the use of a particular caching model.
Another way (easier to implement, but a bit hacker) is to use the “URL error”. When there is an avatar URL in the response, it is necessary to make part of it dynamic. So instead of submitting a URL like
to make
The hash should be changed when the user changes the avatar. A call that sends a list of friends will now send modified URLs for users who have changed their avatars. Thus, changes in profile images will be distributed almost instantly!
If your server and client applications meet practically well-established caching standards, your iOS app and your product will simply “fly” at all.
In this article I gave a simple explanation of such standards, which the vast majority of developers do not adhere to.
At this point I finish the second part of the article. The next and last will describe the exchange of information about errors and their proper processing, as well as the internationalization of your application.
REST API Design Rulebook
HTTP API, REST and HATEOAS
Currently, the HTTP API can be divided into
- Web services
- RPC URI Tunneling
- HTTP-based Type 1
- HTTP-based Type 2
- REST
Here is a very good explanation of this from Jon Algermissen. Unfortunately, everyone who provides their API calls its RESTful service even though it is not.
So what is it, a real RESTful server? This is any hypertext / hypermedia based API. In other words, a third-party developer or client application should be able to obtain information about “other available resources” through the root URL API. In fact, this is the most important condition when implementing the RESTful API. In addition, a true RESTful server can only be considered one that adheres to the principle of HATEOAS.
')
HATEOAS - Engine of Application State
Two basic principles to be followed when implementing HATEOAS are:
- Service only hypermedia resources. Hyperlinked resources are those that contain only content and hyperlinks to other hyperlinked resources. JSON (application / json) is NOT a hyperlink resource. (On the other hand, there are many RESTful, HATEOAS servers that support JSON) However, you can add additional fields to JSON, thus causing it to act as a hyperlink resource. (For example: the href field for the link to the corresponding thumbnail image)
- Single entry point for client application. From the API home page, subsequent GET calls should be made up as links to the corresponding URLs, and subsequent “POST”, “PUT” or “DELETE” requests in the form of forms.
Does HATEOAS use in your new API?
NOT!
Why? In the past, APIs were mostly written for use in web applications. These servers typically returned XHTML, and client applications ran in a browser. The browser in such a scheme is similar to your mobile client, which parses hyperlink resources, knows what a form is and how to present it to the user. In the case of a mobile application, when the response of your RESTful server contains a form from which you can send data to the server, you cannot (at least without additional efforts) convert it into native interface elements available on the platform. None of the platform providers, either Apple / Google or Microsoft, provide support for converting XHTML forms to UIViewController (on iOS) or Intent (on Android) or Silverlight Page (on Windows Phone). I recommend using hyperlink resources in response to GET requests and provide controller method calls (instead of forms) for all POST / PUT and DELETE requests. (For example: / friends / add or / venues / checkin) Controller method calls can be embedded in other answers (so that the client application or the developer can learn about them). Of course, this violates the principles of REST, but that's okay. It is better to make a quality product than to blindly follow standards that are of little use in practice. (This may lead to our API becoming HTTP-based Type 2).
Caching
Go to caching. Caching, as many believe, is basically a client task (or an intermediate proxy task). But you know what, at the cost of small efforts in developing the server-side, you can make your API fully meet the requirements of intermediate proxy caching? This means that you get free load balancing on their part. Everything you need is described in chapter 13 of the HTTP specification .
Not long ago, Art Taylor tweeted:
"I derived two new rules: 1) If your application slows down, add caching. 2) If the application is buggy, remove caching. Why is caching so difficult!"
But believe me, caching can be implemented without collisions with such problems. Two main principles that I recommend that you follow are:
- Do not try to do non-standard caching schemes in the client application.
- Understand the basic caching principles described in the HTTP 1.1 RFC specification. It describes two models of caching. Model of validity and model of validity (validity).
In any client-server application, the server is a trustworthy source of information. When you download a resource (page or response) from the server API, the server sends to the client, among other things, some additional “hints” about how the client can cache the received resource. The server authoritatively indicates to the client when the cached information expires. These hints can be sent both programmatically and through server configuration. The expiration model is usually implemented through server configuration, while the validity model requires software implementation by the server-side developer. It is the developer who must decide when to use validity, and when the validity period is based on the type of resource being returned. The expiration model is usually used when the server can uniquely determine how long a particular resource will be valid. The validity model is used for all other cases. Later I will show you how to implement both of these models during server development. As soon as you deal with both, I will show when to use each of them.
Expiration Model
Let's look at a common caching configuration. If you use nginx, you probably have something similar in the config:
location ~ \.(jpg|gif|png|ico|jpeg|css|swf)$ { expires 7d; } nginx translates these settings into the appropriate HTTP header. In this case, the server sends the “Expires” or “Cache-Control: max-age = n” in the header for all images and expects the client to cache them for 7 days. This means that you will not need to request the same data for the next 7 days. Each of the common browsers (and intermediate proxies) takes this header into account and works as expected. Unfortunately, most of the Open Source image caching frameworks for iOS, including the popular SDWebImage, use a built-in caching mechanism that simply deletes images after n days. The problem is that such frameworks do not correspond to the model of validity and your client application using these frameworks has to resort to non-standard solutions (hacks). I will give an example showing what could go wrong here. Let's return to our “new Facebook”. When your user uploads an avatar to the server, he thinks that the changes will be reflected in all views. Some clever developers clear the local cache after successfully calling update-profile-image. (This means that all controllers must download the image from the server for a new one). Everything works great, you have reported to the project manager and in each submission the most recent profile picture is now displayed. However, this does not completely solve the problem. A new avatar of the user will be seen by his friends only after 7 days. Absolutely unacceptable. So how to solve it? As I said, you must accept the statement that only a server can be a source of reliable data. Do not use dishonest client tricks to update the cache by prematurely ending the expiration of the cached content.
Validity model
Both Facebook and Twitter solve the problem of outdated profile images (after a new image has been uploaded) using the validity model. In the validity model, the server sends a unique resource identifier to the client and the client caches both the identifier and the response. In terms of HTTP, this unique identifier is called ETag. When you make a second request to the same resource, you must send it to the ETag. The server uses this identifier to check whether the resource you requested has changed since the last time it was accessed (remember, the server is the only reliable source). If the resource has really changed, it sends the latest copy. Otherwise, it sends 304 Not Modified. The cache validity model requires additional efforts from the developer in the development of both client and server parts. I will describe both of them further.
Customer support
In fact, under iOS, if you use MKNetworkKit it does all the work automatically. But for developers for Android and Windows Phone, I will describe in detail how this should be implemented.
Cache validity model uses ETag and Last-Modified HTTP headers. Client side implementation is simpler than server side. If you receive an ETag with a resource, when you make a second request to receive it, send the ETag in the “IF-NONE-MATCH” header field. Similarly, if you received a “Last-Modified” resource, send it to the “IF-MODOFIED-SINCE” field of the header in subsequent requests. The server, on its part, decides when to use “ETag”, and when “Last-Modified”.
The implementation of the lifetime model is simple. Simply calculate the expiration date based on the header fields, “Expires” or “Cache-Control: max-age-n” and clear the cache when this date arrives.
Server side implementation
Using ETag
ETag is usually calculated on a server using hashing algorithms. (Most high-level server languages ​​such as Java / C # / Scala have object hashing facilities.) Before generating a response, the server must calculate the object hash and add it to the ETag header field. Now, if the client actually sent an IF-NONE-MATCH in the request and this ETag is equal to what you calculated, send 304 Not Modified. Otherwise, create a response and send it with a new ETag.
Use Last-Modified
Implementing Last-Modified is not entirely straightforward. Let's imagine that our API has a call that returns a list of friends.
http://api.mynextfacebook.com/friends/ When you use ETag, you calculate the hash of an array of friends. When using Last-Modified, you must send the last modified date of this resource. Since this resource is a list, this date should be the date when you last added a new friend. This requires the organization developer to store the date of the last data change for each user in the database. A bit harder than ETag, but gives a big advantage in terms of performance.
When a client requests a resource for the first time, you send a complete list of friends. Subsequent requests from the client will now have the “IF-MODIFIED-SINCE” field in the header. Your server code should only send a list of friends added after the specified date. The access code to the database before the modification was something like this:
SELECT * FROM Friends; after modification it became so:
SELECT * FROM Friends WHERE friendedDate > IF-MODIFIED-SINCE; If the request does not return records, send 304 Not Modified. Thus, if a user has 300 friends and only two of them have been added recently, then the answer will contain only two entries. The time it takes for the server to process the request and the resources it consumes is significantly reduced.
Of course, this is a very simplified code. A developer will have a headache when you decide to support the removal or blocking of friends. The server must be able to send hints, using which the client will have the opportunity to tell which friends were added and which were deleted. This technique requires additional efforts in the development of the server part.
Select a caching model
So. It was a difficult topic. Now I will try to summarize and derive the basic rules for the use of a particular caching model.
- All static images must be serviced by expiration model.
- All data generated dynamically should be cached according to the validity model.
- If your dynamically configured resource is a list, you should use a validity model based on Last-Modified. (For example / friends). In other cases, a validation model based on ETag should be used. (For example /friends/firstname.lastname).
- Images or any other resources that can be modified by the user (such as an avatar) should also be cached according to the validity model using the ETag. Although these are images, they are not permanent, such as a company logo. In addition, you simply can not accurately calculate the duration of such resources.
Another way (easier to implement, but a bit hacker) is to use the “URL error”. When there is an avatar URL in the response, it is necessary to make part of it dynamic. So instead of submitting a URL like
http://images.mynextfacebook.com/person/firstname.lastname/avatar to make
http://images.mynextfacebook.com/person/firstname.lastname/avatar/<> The hash should be changed when the user changes the avatar. A call that sends a list of friends will now send modified URLs for users who have changed their avatars. Thus, changes in profile images will be distributed almost instantly!
If your server and client applications meet practically well-established caching standards, your iOS app and your product will simply “fly” at all.
In this article I gave a simple explanation of such standards, which the vast majority of developers do not adhere to.
At this point I finish the second part of the article. The next and last will describe the exchange of information about errors and their proper processing, as well as the internationalization of your application.
I recommend reading
REST API Design Rulebook
Source: https://habr.com/ru/post/144259/
All Articles