Erlang in wargaming
The Erlang language is not widely known, although it is a rather interesting technology, very strong in its niche. The last few years, Erlang is increasingly used in web development.
The attitude towards Erlang is controversial - there are both staunch supporters and harsh critics. However, few people use it in development. So there are very few specialists who can praise or scold this language based on their own experience of the application.
In the company Wargaming Erlang used for more than 3 years. There are about a dozen developers and several projects working under heavy loads. And in this article we will share our experience and vision of Erlang.
')
In Wargaming, a microservice architecture is used, where different services interact with each other using both the HTTP protocol and AMQP. Therefore, RabbitMQ servers are actively used. You could say it all started with RabbitMQ. And although the programmers did not know about Erlang yet, the architects and DevOps had already gained experience, figuring out why RabbitMQ had fallen somewhere, which can be learned from its logs and crash dump; and what you need to do so that it does not fall :)
I must say that at that time our architects had long been sympathetic to Erlang, but did not implement it as a development language until there were good reasons for this.
It's no secret that our game servers run on the Big World engine, where game logic is written in Python (Python was and remains the main development language in the company). Initially, an in-game chat worked through these servers. But at some point it was impossible to waste server resources (which were not enough anyway) for the chat.
Chat World of Tanks is closely related to the game, supports many specific functions. Therefore, he needed his own decision. It, of course, had to be clustered, since one server cannot be scaled to the existing and planned loads in the future. It is quite logical that Erlang was chosen for such a project. Then the first Erlang-developer came to the company.
In addition to game servers, we have a lot of web services (such as registration and authorization of players, payments, etc.). Initially, it was all written in Python / Django, and as a result of the rapid growth in the number of players on the web, problems began.
Then we started using Twisted . But both the developers and the architects considered it too complicated, and the development on it was too expensive. As a replacement, the architects suggested using Erlang, but the management had reasonable doubts as to where to get enough Erlang developers, because this is not the most popular technology.
Meanwhile, prototypes written in Erlang showed good performance in benchmarks, while Twisted continued to hurt. And the architects went further than the prototypes, made and launched one small web service on Erlang. It was written in a couple of weeks by the efforts of one architect and was successful. Then there was the second project, the third. The company began to search for Erlang developers, and our Pythonists easily mastered the new language (for some it was easier than dealing with Twisted).
Today, there are a dozen developers in the company who can be called Erlang-developers - they write most of the code (or even all) to Erlang. Some of them have been writing Erlang for several years, while others have just switched from Python.
We have a lot of work on the web. The number of games that Wargaming releases is growing. We are moving from specialized (for a specific game) services to universal. Also new projects start, many of which are written on Erlang. In vacancies, we often look for Python-developers (the main programming language in our company), but we mention that knowledge of Erlang will be a plus :)
As a result, we managed to build a team of strong Erlang developers. Among them, for example, Sergey Prokhorov, whose patch was accepted by the Erlang / OTP team. So now in the Erlang virtual machine there is some Sergey code.
Having a lot of Erlang-developers within the company with different experience, I managed to collect a wide range of opinions about Erlang. It is interesting to note the difference in the perception of language between experienced and novice developers.
Experienced are generally positive about Erlang. And this is not surprising, otherwise they would not have written on it for several years and would not have become experienced :) Beginners are treated differently. They have more criticism and discontent. This is also not surprising, because in comparison with Python, Erlang is noticeably poorer in terms of expressiveness, and in infrastructure, and in the wealth of libraries.
In general, the developers gladly criticized Erlang, and they found more minuses than advantages. This is not surprising, because you quickly get used to the advantages of technology, take them for granted, and the disadvantages continue to irritate and attract attention. However, in our company, the developers themselves choose the language for projects (of course, from the list approved by architects :)), this decision is not imposed from above. And since they write on Erlang, it means that it is not so bad. To date, our web uses Python 2.7, Python 3.5 and Erlang. Architects do not mind trying other options: the Go language, one of the JVM languages, something else - but so far there were only those who wanted it on Elixir, and more on that below.
Surely, many have heard that Erlang has a low entry threshold, and a developer after 2 weeks of acquaintance with a language can write code for production. It is true, but not all. The language is really simple and easy to learn, but there are pitfalls.
First, the "children's problems" on which many inexperienced developers stumble, and experienced, having learned to solve them, forget. The classic example is to get a deadlock in gen_server. It happens to almost everyone. A kind of ritual of entering the club :) From the same sandbox problem of understanding that the lines do not need to use, and use only binary. This rule is easy to follow in your project, but, unfortunately, in third-party libraries it can be different, and you need to be ready for this.
Secondly, there are not at all childish problems. And the first of them is a lack of information. Erlang has written a few good books. Most of them describe the language as such, and not its use for solving applied problems. Many important questions are not described at all, neither in books nor in documentation. What tools to use to build the project? How to monitor a working node? How to detect mailbox overflow? What about other memory leak options? How to find bottlenecks that prevent system scaling? Which libraries to use? They, on the one hand, are few. On the other hand, popular libraries have many forks. And it is not clear what the forks are, how they differ, how they are supported. Those who are looking for a driver for working with MySQL understand this problem well :) And this we have not yet taken on distributed systems - a separate world with a lot of nuances. All this can only be learned from another experienced developer. Therefore, it is important that they are always in the team.
There are several languages that are compiled into Erlang bytecode and run on the Erlang virtual machine. The most popular of them is Elixir .
A few years ago, Ruby / Rails developers noticed Erlang. They were looking for a solution to performance problems specific to Rails. Performance problems were resolved, but they did not like everything else in Erlang. And not surprisingly, from a powerful high-level language with a rich infrastructure, they fell into a harsh low-level world.
Ruby / Rails developers quickly realized that they did not want to live this way, and began to build a more comfortable environment for themselves. The result of their efforts was the Elixir language and the Phoenix web framework, each of which has an active community. For 3 years, they did more for web development than Erlang did in 10 years.
Opinions about Elixir, as well as about Erlang, opinions are contradictory. Some consider it easy to read, concise and powerful. Others - difficult to read and thoughtless. Conservative Erlang developers refer to Elixir cool. And those who come to Erlang from Ruby or Python accept Elixir with curiosity.
The community is occupied by infrastructure, libraries and frameworks, and the core language is developed by a small team. There is a risk that if this team loses funding, and no one replaces it, the language will no longer develop and be supported.
In the Wargaming web, a prototype of one of the future services is done on Elixir. Let's see how he will show himself.
This is the opinion of our Erlang developers. What do you think? What exactly does attract you in the language, and what prevents you from working and simply spoils the blood? Share in the comments! Tell us about your experience with Erlang and implemented projects.
Thanks for attention,
Yuri Zhloba
Web developer at Wargaming
The attitude towards Erlang is controversial - there are both staunch supporters and harsh critics. However, few people use it in development. So there are very few specialists who can praise or scold this language based on their own experience of the application.
In the company Wargaming Erlang used for more than 3 years. There are about a dozen developers and several projects working under heavy loads. And in this article we will share our experience and vision of Erlang.
')
How it all began
In Wargaming, a microservice architecture is used, where different services interact with each other using both the HTTP protocol and AMQP. Therefore, RabbitMQ servers are actively used. You could say it all started with RabbitMQ. And although the programmers did not know about Erlang yet, the architects and DevOps had already gained experience, figuring out why RabbitMQ had fallen somewhere, which can be learned from its logs and crash dump; and what you need to do so that it does not fall :)
I must say that at that time our architects had long been sympathetic to Erlang, but did not implement it as a development language until there were good reasons for this.
It's no secret that our game servers run on the Big World engine, where game logic is written in Python (Python was and remains the main development language in the company). Initially, an in-game chat worked through these servers. But at some point it was impossible to waste server resources (which were not enough anyway) for the chat.
Chat World of Tanks is closely related to the game, supports many specific functions. Therefore, he needed his own decision. It, of course, had to be clustered, since one server cannot be scaled to the existing and planned loads in the future. It is quite logical that Erlang was chosen for such a project. Then the first Erlang-developer came to the company.
How it went on
In addition to game servers, we have a lot of web services (such as registration and authorization of players, payments, etc.). Initially, it was all written in Python / Django, and as a result of the rapid growth in the number of players on the web, problems began.
Then we started using Twisted . But both the developers and the architects considered it too complicated, and the development on it was too expensive. As a replacement, the architects suggested using Erlang, but the management had reasonable doubts as to where to get enough Erlang developers, because this is not the most popular technology.
Meanwhile, prototypes written in Erlang showed good performance in benchmarks, while Twisted continued to hurt. And the architects went further than the prototypes, made and launched one small web service on Erlang. It was written in a couple of weeks by the efforts of one architect and was successful. Then there was the second project, the third. The company began to search for Erlang developers, and our Pythonists easily mastered the new language (for some it was easier than dealing with Twisted).
Today, there are a dozen developers in the company who can be called Erlang-developers - they write most of the code (or even all) to Erlang. Some of them have been writing Erlang for several years, while others have just switched from Python.
We have a lot of work on the web. The number of games that Wargaming releases is growing. We are moving from specialized (for a specific game) services to universal. Also new projects start, many of which are written on Erlang. In vacancies, we often look for Python-developers (the main programming language in our company), but we mention that knowledge of Erlang will be a plus :)
As a result, we managed to build a team of strong Erlang developers. Among them, for example, Sergey Prokhorov, whose patch was accepted by the Erlang / OTP team. So now in the Erlang virtual machine there is some Sergey code.
Developer Opinions
Having a lot of Erlang-developers within the company with different experience, I managed to collect a wide range of opinions about Erlang. It is interesting to note the difference in the perception of language between experienced and novice developers.
Experienced are generally positive about Erlang. And this is not surprising, otherwise they would not have written on it for several years and would not have become experienced :) Beginners are treated differently. They have more criticism and discontent. This is also not surprising, because in comparison with Python, Erlang is noticeably poorer in terms of expressiveness, and in infrastructure, and in the wealth of libraries.
Let's start with the general points with which all our developers agree:
- Erlang shows good web performance, measured by requests per second. To achieve similar performance in Python for longer, harder and more expensive.
- Erlang language is highly specialized. You should not try to solve any problems on it.
Pros Erlang, general opinion:
- A good model of multithreading, which is convenient to use. Actor model and asynchrony make it easy to write efficient multi-threaded code.
- Over 20 years of use in loaded projects, standards and templates have been formed that are compiled into an OTP framework. It is easier to build complex, but stable and reliable systems with them.
- Erlang can use all available kernels. There is no need to run a separate service instance for each kernel.
- Low threshold of entry. The language is simple, it is learned quickly (however there are some pitfalls, which will be described below).
Pros that novice developers see:
- A closed ecosystem, the project does not need anything from the outside world.
- You do not need an external in-memory storage (Redis, memcached), you can keep the cache directly in the RAM of the service.
- You do not need to run multiple instances of the service for each kernel, under the external supervisor, with its own configuration. One service can use all the kernels, and has its own flexible supervisor system.
- Fewer service instances - less communication between them over HTTP or AMQP. It is much more convenient to communicate between processes within a single node.
- Easier to develop distributed systems. Erlang has its own tools for building clusters and communication between nodes.
Pros that experienced developers see
- Functional paradigm, immiable data structures.
- It is convenient to monitor the service in real time, search for and fix problems right on the running node.
- You can change the configuration without restarting the service.
Cons, general opinion
- There are few libraries, even fewer good ones. Much need to write yourself or modify.
- Limited language syntax. Sometimes there is a lack of more convenient syntax. For example, the let construct to introduce scopes would not hurt. Bothers the need to create variables with numeric suffixes: Var1, Var2, Var3.
- The language is conservative and develops very slowly. For example, the type of map has been implemented for several years (and has not been completely completed).
- The infrastructure for the web is slowly developing, there is no web framework.
- The language is focused on lower-level application than what is familiar to Python / Django or Ruby / Rails developers.
Cons that novice developers see
- Lines are implemented terribly, there are very few tools for working with strings. Experienced developers do not use strings at all, and therefore have forgotten about such a problem :)
- Weak standard library, few features. This is especially noticeable in the background of Python. Inconsistencies in standard modules (function names, order of arguments, return values).
- Functional paradigm, immiable data structures. For an experienced developer, plus, for a beginner who is not familiar with the paradigm, is a minus. This requires separate study.
- Hard to read error messages.
Cons that experienced developers see
- There are no good tools for managing dependencies and building packages. Rebar3 and hex - the movement in the right direction, but they are still raw and not widely accepted.
- Releases are difficult and do not solve those problems. For example, they solve the hot update task, and this is not needed by the web developer. It is more correct to build the service so that it painlessly relives the stopping of one node.
- Syntax of configs is not convenient to DevOps. Honestly, it is not convenient for the developers themselves. It is better to describe the configuration in simpler formats, for example, ini.
Some more individual opinions
- A Python developer is easier to learn Erlang than Twisted, despite the fact that it is a completely different language. Projects on Erlang are simpler, are developed faster and behave more reliably than similar ones on Twisted.
- The Erlang developer community is small and not very active. Each team lives in its project. Few people exchange experience and fumble knowledge to colleagues from other companies. The usual situation is when each developer writes his own set of libraries, which is used only in his project.
- There are large open source projects like Cowboy or Riak , but there are no projects around which an active community would form.
- Erlang virtual machine is good, but the language itself is too conservative, does not strive for development. Therefore, active development is now underway in the framework of new languages for this virtual machine, such as, for example, Elixir.
- Erlang and Python complement each other well, compensating for weaknesses. Correctly combine them together in one project. Business logic, ORM, work with the database is best done in Python, and multi-threaded data processing, communication between services, background processes - in Erlang.
- Erlang has a terrible syntax that makes you suffer :)
What is the bottom line?
In general, the developers gladly criticized Erlang, and they found more minuses than advantages. This is not surprising, because you quickly get used to the advantages of technology, take them for granted, and the disadvantages continue to irritate and attract attention. However, in our company, the developers themselves choose the language for projects (of course, from the list approved by architects :)), this decision is not imposed from above. And since they write on Erlang, it means that it is not so bad. To date, our web uses Python 2.7, Python 3.5 and Erlang. Architects do not mind trying other options: the Go language, one of the JVM languages, something else - but so far there were only those who wanted it on Elixir, and more on that below.
Insidious entry threshold
Surely, many have heard that Erlang has a low entry threshold, and a developer after 2 weeks of acquaintance with a language can write code for production. It is true, but not all. The language is really simple and easy to learn, but there are pitfalls.
First, the "children's problems" on which many inexperienced developers stumble, and experienced, having learned to solve them, forget. The classic example is to get a deadlock in gen_server. It happens to almost everyone. A kind of ritual of entering the club :) From the same sandbox problem of understanding that the lines do not need to use, and use only binary. This rule is easy to follow in your project, but, unfortunately, in third-party libraries it can be different, and you need to be ready for this.
Secondly, there are not at all childish problems. And the first of them is a lack of information. Erlang has written a few good books. Most of them describe the language as such, and not its use for solving applied problems. Many important questions are not described at all, neither in books nor in documentation. What tools to use to build the project? How to monitor a working node? How to detect mailbox overflow? What about other memory leak options? How to find bottlenecks that prevent system scaling? Which libraries to use? They, on the one hand, are few. On the other hand, popular libraries have many forks. And it is not clear what the forks are, how they differ, how they are supported. Those who are looking for a driver for working with MySQL understand this problem well :) And this we have not yet taken on distributed systems - a separate world with a lot of nuances. All this can only be learned from another experienced developer. Therefore, it is important that they are always in the team.
A few words about Elixir
There are several languages that are compiled into Erlang bytecode and run on the Erlang virtual machine. The most popular of them is Elixir .
A few years ago, Ruby / Rails developers noticed Erlang. They were looking for a solution to performance problems specific to Rails. Performance problems were resolved, but they did not like everything else in Erlang. And not surprisingly, from a powerful high-level language with a rich infrastructure, they fell into a harsh low-level world.
Ruby / Rails developers quickly realized that they did not want to live this way, and began to build a more comfortable environment for themselves. The result of their efforts was the Elixir language and the Phoenix web framework, each of which has an active community. For 3 years, they did more for web development than Erlang did in 10 years.
Opinions about Elixir, as well as about Erlang, opinions are contradictory. Some consider it easy to read, concise and powerful. Others - difficult to read and thoughtless. Conservative Erlang developers refer to Elixir cool. And those who come to Erlang from Ruby or Python accept Elixir with curiosity.
Pros of the language:
- Rich standard library;
- High-level web framework Phoenix ;
- Good infrastructure: build tools, package manager, libraries;
- Good documentation;
- Active community.
Minuses:
- The language is young, there is no experience in the industry;
- Knowledge of Erlang is still needed, too close a connection between these languages;
- Low bus factor.
The community is occupied by infrastructure, libraries and frameworks, and the core language is developed by a small team. There is a risk that if this team loses funding, and no one replaces it, the language will no longer develop and be supported.
In the Wargaming web, a prototype of one of the future services is done on Elixir. Let's see how he will show himself.
This is the opinion of our Erlang developers. What do you think? What exactly does attract you in the language, and what prevents you from working and simply spoils the blood? Share in the comments! Tell us about your experience with Erlang and implemented projects.
Thanks for attention,
Yuri Zhloba
Web developer at Wargaming
Source: https://habr.com/ru/post/279621/
All Articles