Hello, my name is Alexander Zelenin and I am a web developer.
Information is the basis of any application or service.



More than 10 years ago, I spoke with the owner of the poker room, and he showed me a page that brought about $ 10,000 a day . It was a completely banal page. She had neither styles nor graphics. Solid text, broken headings, sections and links. I just did not fit in my head - well, how can this bring such money?
')
The secret is that “this is” was one of the first comprehensive guides to playing online poker. The page had a PageRank 10/10 (or 9, not an essence), and in search results it was the first thing that came across.

The purpose of your application, whatever it is, is to convey (receive, process) some information to the user.





I think you already caught the idea. Examples can be given endlessly (here’s another one: they don’t go to Wikipedia. Moreover, some of the information from Wikipedia is displayed immediately in the search results, without even opening the site itself), and if you think that this is not applicable in your case - write in the comments (or mail / personal), and I will explain why it is still applicable.

So: whatever you do, information will always be primary. Users will surely find and contact you with good quality information.

I will tell you how to organize work with information so that it is:
1. Scalable - replication, sharding, etc. configured without interfering with the application.
2. Convenient for users - easy to document, understand how to use.
3. Convenient for your developers - rapid prototyping, optimization capabilities only necessary.

This approach does not make sense to you if you have a small project with a small number of components and developers.



Table of contents:

1. Information Consumers
2. How to work with information (API)
3. API inner layer
4. Outer layer API
5. Heavy query optimization
6. Users
7. Scaling
8. Caching
9. Versioning
10. Total

Information Consumers

Information consumers can be divided into two categories - internal and external.

Internal - these are your products and services. The difference is that for "their" APIs it can provide much more extensive functionality with fewer restrictions. For example, the same Google maps on their own domain could work with the use of webgl, and as a result, much faster, and embedded - not (at the moment the situation could change, did not check).

External - end users or products not owned by your company. For example, for Google Maps, you are an external user. Usually, access to information from outside is severely limited, and special access keys are often required.

How to work with information?

To work with information, we will provide a web API. We will implement 2 layers (external and internal). The layer implies at least a separate application.

Why do you need an API?

The API allows you to provide data in a platform-independent form. It is not always known how and where data will be used, and API development is a good way to state “we have information - contact us”.

All code examples are just one of many implementations. This will make it possible to use the data regardless of the ways in which the final product is implemented (including offline applications, provided at least one-time access to the network).

The first step is to describe the models and data collections. If the application is implemented in Javascript (nodejs on the server), it will be possible to use the same models on both clients and servers.

A model is a description of an entity (for example, a music track): its fields, their properties, ways to access and provide information. The model can duplicate the database schema, but can also expand it with additional information. Moreover, a model can contain information from several tables / collections and represent it as one entity. On the server, the model should be extended with descriptions for working with tables, access to servers, and so on. On the client, the model is extended by data access addresses.

When accessing data, the model may also contain additional meta-information about the request itself (execution time, position of the record in the database, links), virtual fields (for example, if path is stored in the database - relative path to the file, you can add a virtual url field that will be calculated "on the fly").

As an example, I will give a code describing a certain music service.

The examples will be in Javascript, but everything described applies to any language. I did things like this in php, python, and c ++. Everything needs to vary depending on the size of the project.

Model.extend('Track', { //   attributes: { id: 'integer', //     title: 'string', url: 'string', // /   duration: 'integer', album: 'app.model.Album.model', //     artist: 'app.model.Artist.model' } }) 

A collection is a collection of entities (usually of the same nature, that is, for example, music tracks). The data set may also contain additional data related to the set itself. The number of selected tracks, the number of remaining (not selected) tracks, the page number, the number of pages can be represented as meta-information. A virtual field can be the total duration of all tracks.

 Model.List.extend('Track.List', { //   attributes: { duration: 'virtual' //  ,     } }, { duration: function(tracks) { //     return _.reduce(tracks, function(totalDuration, track) { return (track.duration || 0) + totalDuration; }) } }) 

API inner layer

This layer will be available only to our products.

Since our models already contain a lot of information, we can provide access to the data using the minimum amount of code.



On this layer, we have maximum query flexibility.

Outer layer API

The outer layer is directly accessible to end customers. It can be a website or just an API for third-party developers.

On the outer layer, we do NOT provide such flexibility as on the inner layer, but only give access to the basic features: the main query parameter, indent, number, etc. And all this with restrictions.

In part, he is a proxy to the internal API with a number of important additions.

Just an example:

 app.get('/api/track/:id', ..., function(req, res) { return app.model.Track.findOne({id: req.params.id}); }) 

Instead of "..." we do the necessary rights checks, modify the request, define the format. Data is returned in the same format and way as requested.
Those. for http and json, the data will come back. For socket and xml, the answer will be through the socket and in xml.

In this way, we can fully control what is available from the outside, how, under what conditions, and so on.

Heavy query optimization

Before that, we described working with the database as abstract as possible, and, of course, such requests will be executed much slower than the optimized ones. With the first step, we discover (using a profiler or some other convenient method) that one of the queries is slow.

Suppose we noticed that the query is slowly running, in which the track is selected along with information about the album and the musician. To optimize, we need to create a new internal method:

 Model.extend('Track', { 'GET /track/{id}/withAdditionalData': function() { var track = ...; //       //      return track; } }) 

and change the call on the outer layer to the inner one for the presented one. Everything. For the final client, nothing has changed, and the cache, the paths, and the data received are the same, but everything has become faster to work.

Users

The main task when working with users is to check their rights.

As soon as the user is authorized (the method is not important: a cookie, a key, another option), we make one request to the inner layer, verifying identity and obtaining permissions. Further checks we will do on the outer layer.

Scaling

We get great advantages at the scaling stage.

We can run both the outer and inner API layers in any number of instances, resolving the load with the help of balancers. Due to this separation, we can run multiple applications with the outer layer as close as possible to the final client, having received our own CDN network with data caching.

Databases are extended in classical ways, depending on the task.

Caching

For the inner layer, we cache the results of database queries, and on the outer layer, the results obtained from the inner layer. The end client can also have caching.

In one of the previous examples there was the line “cache: 1800” - it can provide a cache both on the outer layer, memorizing the result on the server for half an hour, and on the client, adding the result, for example, in localStorage (well, or another client storage).

Versioning

With the development of your project, new methods will appear, and any old methods will go away. To indicate the versions I recommend, undoubtedly, Semantic Versioning . We are particularly interested in changing the major version of the API, without backward compatibility. API access paths can be divided simply: / api / {version}

You can organize files and support for different versions in several ways, for example:
1. Make folders v1, v2 and place in them all the code relating to them. When modifying one of the API versions, the other is not affected.
2. Different repositories function as different applications.

Total

1. We have full control over the movement of data at all stages.
2. The developers are happy, they don’t have to wait for the API team to implement a super-tricky method to get data in a specific format.
3. Developers are happy, they can only optimize what slows down.
4. Clients are satisfied - the API is more stable and does not change the path due to the fact that some query was slow.
5. Customers are satisfied - the access speed is increased due to the location of the server as close as possible.

I would be happy to add to the article additional sections on your request.
Also, if you want to clarify the code somewhere - I will write and attach, ask.
Perhaps this is the beginning of a large cycle of articles on various topics ( or a start has already been made ). Material has accumulated a lot, but it is not systematized.