Geekly Articles each Day
📜 ⬆️ ⬇️

Ten years of Erlang programming


I joined the Erlang community about 10 years ago, in the middle of the first phase of the HYIP. We were told that Erlang is the future of competitiveness and concurrency. Implementing them in this language is the easiest and fastest, and you still get free distribution. At that time, the future seemed incredible. The virtual machine recently received SMP support, but in order to actually use all the processors, you had to run multiple virtual machines on one computer.

I want to reflect on the past decade. In this article I will talk about the HYIP phases in relation to Erlang, about the ladder of ideas in the language and its possible influence on the distribution of the language, about the changes I have gone through in these 10 years. And in conclusion, I will share my thoughts on the fact that Erlang has yet to be brought into the community of programmers as a whole.

HYIP Phase


The HYIP cycle consists of phases in the life cycle of a product or technology. This is a marketing concept, not a scientific one, but it often helps in describing the essence of what is happening. Most of all I am interested in the HYIP phase , a kind of gold rush covering the communities of programmers. Perhaps you have come across this more than once, and all these phases are connected with some killer app that everyone rushes to use.

From the examples, it immediately comes to mind Ruby on Rails, How to Build a Blog Engine in 15 minutes (the phrase “Look at everything that I don’t do!” Is still funny) or Go with Kubernetes (they worked together before, but then there was a direct surge). This also includes Elixir from Phoenix.
')
During the HYIP phase, there is an incredible influx of newbies who want to see what all the fuss is about. Someone remains, most go. You can stay for months or years, in rare cases you find a home for decades. However, the overwhelming majority are a constant stream of serial people who jump from technology to technology, looking for opportunities to gain benefits by the fact that they are the first to use a framework, a language or a set of tools.

So most often you need to make one real “killer app”, and people will reach into your ecosystem. Killer app boosts the hype. If you write it, they will come to you. And if you can keep a small share of them and keep them active, then you will have a living community for the foreseeable future. This, to some extent, resembles the idea of rain following the plow :

The Lord is driving the plow. ... Following this miraculous adaptation, which is man’s sole dominion over nature, clouds shed abundant rains ... [the plow] is a tool separating civilization from barbarism, it turns the desert into farms and gardens. ... In short, the rain follows the plow.

The essence of the theory was that human habitation and agriculture have an irreversible effect on the climate of arid and semi-arid areas, making these regions more humid. This theory was widely adopted in the 1870s as an excuse for settling the Great Plains - a region formerly known as the “Great American Desert”. In the same years, this theory was used to justify the expansion of wheat sowing on marginal lands in South Australia.

If we can run a large project, then developers will appear, and it will become self-sufficient. I think this idea is clearly erroneous, mainly because during the largest phase of the HYIP, Erlang had dozens of killer applications, but the community is still small. Here are examples of these applications:


Many of these products came out during the first run of the book by Joe Armstrong Programming Erlang . As a result, there was an explosion in the popularity of the language, and Erlang gained many admirers. Even the fact that Hacker News forced all the discussions on the Innards of Erlang had a significant impact. However, few people remained faithful to this language.

Now I believe that killer applications are created by people who form the initial phase of the HYIP, and not vice versa. There is always a small part of people who, before others, seek out interesting technologies, decide whether they like them, then create something, and if a killer app turns out, then it stimulates the development of an even more powerful phase of HYIP. People will be engaged in the cult of cargo, and successful projects will produce imitators. Also, the standard situation is the “wheel inventions” phase, when everyone spends his time re-inventing already existing things, and a wave of messages about “implementing X, but in Y” appears.

However, it is not enough just to create a killer app. It is curious that all these products, like RabbitMQ and Ejabberd, despite their popularity, have a much larger user community than the developer community. Thousands and thousands of corporations using these products do not necessarily make a significant contribution to the Erlang community.

This is partly due to the fact that most Erlang killer applications are used in specialized infrastructures. You have created a highly reliable component in the form of a black box that everyone uses, and if it works well, then no one will look inside. Several dozen developers have provided the basis for thousands of other products and services. A specialized infrastructure, by definition, does not need a large number of people to ensure its widespread impact. It always has few developers and a small community compared to technologies that are closer to the final product, for example, web frameworks, or even more generalized infrastructures that are used in small projects for deployment, suitable for any company.

But even despite all this, it is obvious that Erlang missed many people who went through it during the HYIP phase.

Ideas ladder


I will not rant about what could or should have been done. Instead, I want to talk about popular learning patterns that I have met in the Erlang community over the years I have been teaching this language and programming on it. I observe the same patterns today in the Elixir community, and this may indicate a similar future for him.

I have a theory that such a technical topic as a programming language (and its ecosystem) has several levels of complexity, on which different concepts are located. I used this theory in the project Learn You Some Erlang . She is represented in the diagram, which I called the Nine Circles of Earl .

Of course, this is an ironic idea, I do not think that the study of any technology is endless suffering (at least it should not be so). I just liked the pun. Simply put, there is always some kind of “main” path or sequence of topics that you learn when studying technology - this is the “ladder of ideas”, and the higher the concept on this ladder is, the more valuable it is and the harder it is to achieve, the fewer people learn it.

What a ladder of ideas in Erlang might look like:

  1. Functional programming.
  2. Isolated processes and competitiveness.
  3. Reliable competitiveness (links, monitors, timeouts).
  4. Principles of OTP and other system abstractions.
  5. How to structure the system according to the principles of OTP.
  6. How to collect releases and work with them (deploy).
  7. How to prevent the system from falling and working with it.

If you only get acquainted with Erlang and bought a book for beginners, then you probably spend the first days on the first rung of the ladder: acquaintance with functional programming, immutability (immobility), recursion and other similar concepts. Sooner or later you will get to the competition and parallelism, processes and messaging. Immediately after them, you will begin to explore links and monitors, handling failures and what makes Erlang what it is. During the big phase of the HYIP, the second and third steps contained opportunities that for most observers looked just unbelievable. If you needed to learn something that you would use in all future projects, then you could choose one of these Erlang features.

It was possible to climb to other steps later, but only if you stick to the step training program. In particular, OTP (fourth stage) can be described as “what it is all about.” Competitiveness and functional programming are good, but the common framework introduced using OTP was truly unique and needed to be used. Over time, many will work with him, find that they can create cool abstractions, but they will not like the approach to structuring.

In fact, to create applications like Ejabberd it was enough to overcome the fourth step. At that time, the ecosystem was reminiscent of the Wild West, and the knowledge of OTP was owned by experts working at Ericsson, as well as the most motivated self-taught people. Most never reached the fifth step, unless they sent something to production and did not encounter problems after that, or if they were not looking for a better solution. The achievement of the sixth stage was rare until 2015-2016, and then Relx emerged , simplifying the assembly of releases and their mastering. Almost no one got to the seventh stage, and a lot of people think that they should not perform a hot update of the Erlang node, and, ideally, they will never need to use SSH to debug production.

In practice, not everyone goes up the ladder of ideas in the same order, its steps in some books are interchanged (for example, in Erlang and OTP in Action ). There is nothing wrong with that, the ladder is invented only as an illustration.

The size of communities changes in waves. During the phases of the HYIP, the size for some time may increase by tens or hundreds of times, and all these people are curious and go. Most participants sit on the first rung of the stairs and rarely leave it behind. Some climb to the second step, even less - to the third, and so on, until you get to a narrow circle of experts at the very top step.

I think it was easy to climb the first three steps of Erlang. To climb to the fourth one had to be engaged in development for several years, and indeed to feel the need for further study. The fifth step is already very difficult. The Erlang toolkit and ecosystem were poor. Community members tolerated this barren environment, and were indifferent to the plight of newcomers. To make the article shorter (well, long, but not absurdly long), see my presentation on the language ecosystem at the Erlang User Conference:


If you write on Elixir, then you probably see at what level of this fictional staircase you are standing, and you can already imagine how much the community is distributed over it. Many of the participants are fluent in Phoenix alone, but rarely rise above the fourth step, and many get stuck in the third and below. And often this is enough. I do not blame, but only share the observation. Like anyone who has climbed many steps (and probably there are a few more steps in front of me, like a virtual machine patch or something else), I see that these people don't know much. But honestly, all this information may be useless to them. And this is good.

What I want to say: I believe that we, as a community, are probably nagging under the bough, making the ladder of ideas very difficult to master above the basic level. Some topics need to be studied slowly, and in some topics the blind lead the blind, because Erlang is so small that we simply have too few people who can share all the necessary experiences. Today it’s easier, and if you don’t come to Erlang during the HYIP phase, you’ll probably get help, because far fewer people ask for it at the same time.

I would like to think, if tomorrow the second phase of the HYIP begins in the world of Erlang, then we will be able to accept newcomers better than during the times of the big wave, when I came too. And I hope this experience, coupled with the closer cooperation of the Erlang and Elixir communities, will double our chances of successfully expanding the use of these languages.

What changed


Erlang does not float alcoholized in a flask, waiting to be pulled out into the light. He is constantly evolving. This is partly due to the competition and wishes from the Elixir community, which, fortunately, expects much more from their tools than what Erlang users are used to. And partly the development is due to the actual needs of the industry and the wishes of the academic community.

I think someone will be happy with these changes, which have occurred since 2009 or even earlier:


If you want to know more, read the full list of changes in releases . In short, I can say that if the OTP team at Ericsson worked on 13-16 releases a little less intensively (and now version 22!), Then their efforts to use Erlang in the main product would be obvious. However, everything changes outside of Ericsson. The Erlang and Elixir communities, as well as Erlang VM development participants from other languages, have joined together to create the Erlang Ecosystem Foundation , with working groups that now help coordinate and solve the problems of assembly and packaging tools, improve observability ( security, etc.

If you, like me, were part of a big initial HYIP phase, but, unlike me, you didn’t linger on this language, because a lot of things looked unusable or confusing, then you might want to give Erlang a second chance. The convenience and language ecosystem has improved dramatically.

Where does Erlang go?


Perhaps, large killer applications no longer appeared, as it was in 2007-2009, but this does not mean that there are no promising projects. Erlang is still deeply integrated into the infrastructure of many corporations, and most of its original killer applications are still in use. In addition, we have many interesting new players, as each BEAM Conf conference shows. I myself am very passionate about concepts like property-based testing (Property-Based Testing), and for Erlang and Elixir some of the best frameworks in the world have been created. And despite all this, we are not in the HYIP phase. At least now.

Will there be another such phase? Maybe yes, maybe not. You can say that the next phase of HYIP was Elixir. Ecosystems have much in common, so that the lessons learned in one of them are applied in the other. Both ecosystems have more similarities than differences. Perhaps we are waiting for some Renaissance. Personally, I do not care anymore. I like small communities, I feel comfortable in them. To love Erlang, I do not need the geometric growth of followers, let their number simply be stable.

In addition, the size of the community has never prevented the spread of Erlang around the world. As far as I know him, the language has always been in such a position that there are not enough tasks for all Erlang programmers, and not enough programmers for all Erlang tasks: there are many of those and others, but geographically they do not always coincide. The best are corporations and workers who enter remote labor markets.

And if before Erlang could not easily penetrate the web application market, now with minimal modifications the entire Elixir market is open to it.

It is probably not very important for the overall picture of the world whether you use a language like Erlang or not. Although it seems to me that it is undervalued and could be applied more widely. The main advantage of Erlang is not to use it. The main advantage in learning the basics of designing reliable systems, as well as in the practical application of the knowledge gained.

Over the years constantly heard issues related to the study. How to learn to design protocols? What advise to read about the creation of distributed systems? How to work, but to do something very reliable and fault-tolerant? How to understand that my architecture is modular and my abstractions do not flow? What is good error handling? How to know that optimization is premature? What does “make something declarative” mean?

We love short and digestible solutions, like recipe books and best practices, but the real answers are a variation of "I have learned in recent years." Honestly, in my career, nothing compares to being in the Erlang community and absorbing the experience of its veterans. The community is small but rich in all other criteria. For several years, I moved from junior to senior roles, performed around the world, looking for ways to transfer my experience, and for the most part, it's all thanks to the community.

Perhaps, I still can’t write a blog engine for 15 minutes (to tell the truth, I’m generally slowly developing), but I very quickly became much stronger as a developer and system architect. Again, I was not always attracted by the use of systems, but by their creation and bringing to working condition. Everyone has their own motivation.

I can not imagine that I would have received as much in another community. These 10 years have been awesome. It is curious that the Erlang community is still small and its potential has not been revealed. This means that there are many opportunities to participate in everything, to communicate one on one with people who are full of wisdom and want to share it, and find their place.

Ps. Thanks for the translation mail ru group; Erlang community in Telegram (Evgeny M., Sergey Ivanov, Vladimir Sekisov, Greg)

Source: https://habr.com/ru/post/460115/

More articles:


All Articles