What we have learned by developing the backend

The development of Parallels Access has required the creation of a geographically distributed service that allows you to securely establish communication between computers and mobile clients of users at different points of the globe. The team that works on it wants to share the experience gained in the form of quotes to ease the plight of those who are only planning to create their client / server product, and plunge into nostalgia of professionals who have a dozen successful projects behind them:

1. For each public address that could potentially be covered by a CDN, start the second for direct use by services. Otherwise, after turning on the “knife switch in the CDN”, you will learn a lot about yourself as a phenomenon.
2. Enable the log slow queries option on the DBMS. There is always an engineer who will “charge” the query past the indices. Or an administrator who configured the backup incorrectly.
3. Virtual machines in the clouds - consumables. Maximize server deployment. Design the service so that the loss of one server is painless.
4. Virtualization as a student dormitory: a party at one (increased CPU or IO activity) can haunt the entire floor.
5. Use proven technology. Any cool thing contains a lot of pitfalls, which are often found already in production.
6. NoSql can turn from a carriage into a pumpkin at the first need to produce a join.
7. The Backend API should be not only easy-to-develop authors, but also easy-to-use clients who are distinguished by their diversity (web / mobile / desktop) and are in the habit of showing different data from one version to another on the same screens.
8. Engineer, remember: the task is completed when your code has benefited the user and the profit to the customer. A commit is just the first step on this path.
9. Running services as root is a security hole. As a last resort, start from under root, switch using setuid ().
10. Excessive logging can adversely affect performance. Learn to change the log level on the fly, which will allow you to investigate problems at the moment they appear.
11. SSL can be used not only for encryption. Transition from keys to certificates will allow to organize the authorization and authentication of components in the infrastructure.
12. Linux system administrators will thank you if the configs are in / etc / myapp, logs in / var / log / myapp, etc. In other words, store files in the directories generally accepted for OS.
13. Any log file could potentially grow out of control. At the design stage, plan the life cycle of the log files and the data they contain.
14. Set up monitoring of all components using a convenient service. Next, get ready to improve it, so as not to wake up at night from minor positives.
15. Create a calendar with expiration dates for certificates, subscriptions, keys. Otherwise, “something will stop working” is completely unexpected.
16. Before improving performance, learn how to choose the right metrics and what throughput and latency are.
17. The service must choose the amount of data that it can correctly process. Otherwise the request
    
    
    
    update users set halyava_end_date = '2016-01-01'
    
    Rows matched: 300000 Changed: 300000 Warnings: 0
    
    
    
    performed in early December, may turn out to be a surprise:
    
    ')
    
    2016-01-01 00:00:00 Mail service is trying to send 300,000 email ...
    
    2016-01-01 00:00:01 Out of memory error
    
    
    
    
18. Client applications must correctly handle the situation when the service is overloaded or unavailable. Each subsequent attempt to connect, choose with increasing interval with the elements of randomness. Otherwise, the rise after the upgrade / fall will be difficult.
19. The application server has no history of migrations; it’s better to be able to roll it reliably and quickly from scratch.
20. Specify the version of third-party libraries. Changing a minor version can break everything. Do not use libraries from public repositories automatically.
21. If you do not know where to look for the error, look for it first in serializers, then in deserializers.
22. While you can use the unix timestamp for storing and processing dates, it is better to use the unix timestamp.
23. The speed of work is a property of the product, which can be costly, but it should not be necessary.
24. Distributed storage systems are either inconsistent, or inhibited, or both.
25. Any queue is a monitoring tool. The number of items in the queue, the speed of their receipt and processing can clarify what is happening in the system.
26. Prematurely getting rid of copy-paste code is a premature optimization.
27. If you do an extra operation, then no matter how fast you do it.

In the comments are ready to answer more specific questions. In addition, all this week I will be leading Twitter backend developers @backendsecret - http://bit.ly/20mJ9GP - where I will also talk about useful things and answer questions.