Why do we need our own development department for 200 people in Leroy Merlin?
Hello! I am Valery Laptev, Head of LM Development in Russia. For two years I had to raise a huge department, and it was quite an interesting experience.
The fact is that "Leroy Merlin" is in many countries. The parent company - in France, is called ADEO. They write code for France, Italy, Spain and Russia. We have different business models: if we keep the minimum prices on the Russian market (below all competitors in the monitoring), then in Europe everything is different. In fact, the differences between the sea - from the characteristics of the locale and ending with other legislation. There are features of the infrastructure of Russia (the same very large delays to Khabarovsk) and another life cycle of the order. All this gives rise to such a hell of code, consisting of huge blocks of IF:
')
Two years ago we had 60 stores and many, many hotelok on features. They rolled in about six months and not always right. The last straw after the pile of features rejected by low priority was the request to create a string field in the order, so that we could only parse it later. It was necessary for the features of delivery in Russia, since the country is larger than other countries where LM is present. We were refused and this, more precisely, was told that it would be somewhere in seven to eight months.
The semi-annual cycle of the slow head company did not suit us. Naturally, we offered to write our code, give it back to review and wait for implementation ... True, nothing good came of it.
The story is very simple: despite our dozens of stores, we are not the first in the world. There are needs of France (where the parent organization), there are needs of other countries. They are prioritized by possible profit or cost reduction for a group of companies and are performed in that order. That is, our features are almost never done, unless they are very lucky and in some kind of sprint someone will finish their part earlier and go to disassemble the bottom of the backlog.
The second peculiarity is that when rolling any feature into the master branch (here in this Legacy code on the IF-IF-IF) you need to check how it behaves in other countries. Beg quoting 441869 with Bash:
In general, when we introduce something for the Russian banks, somewhere in Brazil, someone can ricochet.
This means a very large test coverage, long pre-release procedures, and an unwillingness to support code that is growing rapidly. Therefore, the fewer features - the better. But you stay there.
The position as a whole is very clear, and I would have done just that in the place of the head company. Because it is rational.
The first approach was on the forehead: we offered to open the code to us to make pull-requests there. It was supposed that there would be about 10 developers who would quickly code business-critical functions, give to the French, those who would test according to their own procedure, and everything would be fine.
Actually, these people were already there: we were engaged in cutting out extra business logic that we received from other countries, such as various cumulative discounts and unusual actions (which are simply impossible with our business model) and put dummies on it.
The French in ADEO wanted a release cycle for rolling and installing new versions through themselves. They accepted our proposal for one branch for experiments.
Gave access, began to take the code on the review. It turned out, it is still slow. Rollauts for several months is not the case. Well, won a few weeks, but still the process did not fit.
Then, for six months, an important feature for us in the content part (managing data about the product that the client sees) did not come out. We sat for six months without release. So they didn’t have the feature, they didn’t pass the tests, they didn’t realize exactly what we needed. As a result, they sat on the key system for a long time without updates. There were NodeJS + PostgreSQL + Couchbase + Elasticsearch + Angular on the front. In the code, due to a number of archaeological layers, things like misuse of the SQL database and non-relational database were encountered. In one of the places, a huge piece of master data was taken, inserted into one field of the SQL database, and then split into entities in the NoSQL database. On one page of the site with the display of goods accounted for dozens and hundreds of such requests. With further reading of this legacy, the hair on different parts of the body began to move. We understood what features of the layers of Legacy the head department is facing.
The first idea was to make features together. In place, that is, in France. The four of us went to France and began to sit next to our colleagues in order to sort things out together and do what we needed.
Did not work Anyway, everything was very slow.
The second idea was to fork all that already exists and gradually finish it. We sat down by an architectural committee, evaluated the prospects, calculated approximate development plans, and realized that we need to choose another method. Specifically - yes, do fork, but not to develop the existing code, but in order to break the monolith into parts and install microservices where it is needed.
That is, we planned to rewrite entire blocks of logic in the form of our code. And for this, they began to switch part of the services to what could be done with us. Began to recruit developers. Then we completely followed the stack of the head company - Java, Spring and everything, instead of Couchbase, it was Mongo (this is a similar NoSQL base).
As the project progressed, they realized that there was a lot to do in their own way (because it’s faster and easier not to support the legacy we don’t need, in particular), and began to expand on other technologies as well. Then there were Java 7, Wildfly (formerly JBoss) and SVN. We asked why they are not switching to Java 8, GIT and Tomcat. It turned out they would not mind, but after a couple of years. In the meantime, family, write on the old stack. So we word for word and decided to separate completely. Business figured out what the question is, what are the pros and cons, and fully supported.
Almost immediately, 30 percent of the code was thrown out, and many microservices were written around. From what they did not touch, this is almost the entire core of business process transactions for money.
Naturally, the first thing we thought about was how to delimit the areas of development. About this, I would have told you separately, but in general terms the scheme is this:
The horizontal is the business area. For example, everything related to customer relations (the first green cell) is one area, and there is a set of applications from one department. Such a separation by the objective function creates several duplicates of the code in different places and requires a good corporate bus (again, a separate story), but it allows you to clearly find the ends and solve the problem before the result. Looking at it in almost three years, I can say that the architecture was chosen correctly, but knowing what I know now, I would make a couple of changes.
Now we have come to the conclusion that in the general structure there are many feature-teams that make up large product teams. At the same time, the product team includes not only the developers themselves, but also designers and business representatives. Since the ultimate goal of any IT department in a company is either to increase the speed of business development or to reduce costs through automation, we need business representatives within these teams. Retail is all about IT. There is not a single process that could be called “non-IT”.
A feature team is a small group of people (usually an analyst, a tester, a developer, a beck and a developer of a front, an ops). The analyst usually plays the role of the owner of the product or a person from the business comes to this place. The owner has backlog, priority, tasks. He dictates their design. The developer chooses an implementation option himself, discussing in the team what will be done and how. We do not have timblids to assess the task, the code and the decision. Everybody does everything. Usually, more experienced team members give opinions that many are willing to rely on. But anyone can make a decision. Of course, there are conflicts when two developers cannot agree on the implementation of the feature. If the conflict turned to the individual - need an escalation to the head. But the most part is the choice of implementation solution. Then everyone stands and draws near the board until they find an option that suits everyone. Usually it turns out quickly, but there were cases when they argued all day. When a dispute comes to a standstill, they often call the referee - a person from another team whose opinion everyone trusts. He explains the essence of the problem, he decides. In theory, even a john or a trainee can participate in this dispute in theory, but I know only a couple of such cases — usually the juna have a mentor whom they listen to.
An example of a feature-team: we have a service platform that allows us to buy a service from a contractor along with the product. For example, bought a door - you can immediately put the installation in the basket so that the independent master comes and does it according to our standard. We will check and pay, we will give a guarantee. So, this team makes a product, for this it writes IT decisions and make business decisions, change processes in stores. Agree with contractors. There - the owner of the product from the business, the shopkeeper, architect, developers, analysts and testers. They immediately use our corporate API platform to deliver all the data back and forth and write microservice. Such an approach — immediately, operating companies and business people in conjunction with developers and a small team — allows you to make a product very quickly. But I think it would be better for them to tell later on how this looks from the inside.
Initially, we did not want to do individual testers: there was a trend that a developer should either follow the TDD methodology, or test his code to the end. In fact, it was not very effective. At first everything went well: I wrote - you answer. You need tests to make your application work properly. But then, the more tasks and applications became, the more difficult it was. Some applications died off, some went on the prod and did not change for months. It became hard to write tests, support them and so on. Analysts in teams changed. Relatively recently, they agreed that they were wrong: testers are needed. But developers don't stop writing tests and - sometimes - do TDD. We realized for ourselves that the tests that check the functionality (the application works correctly) and the tests that check that the application works in problematic situations are different people who should do it. And for the second, testers are needed, since they cover possible strange cases not only with unit tests.
Now purely developers 60 people - back, front, full steaks. There are also analysts, testers and support. And plus an additional seven devops. But still, 200 planned people from the headline are not recruited, so we are now looking for new people, because the field of work is huge. Jobs are on My circle , if that. That is, from the development we now have 74 of about 200.
Given that we have a lot of independent teams only in Russia, and there are still teams that are sawing something similar in many other countries, we are moving in the direction of the intersource . This is very similar to open source within a limited group of people.
Within the ADEO group of all divisions by country, all code lies in the cloud Githab. The project is made out according to the same rules. Restrictions on style, technology and tools there. When you have open source and clear design rules - any developer on the stack can contribute. In order to log you into the code, you need to read the dock, clone the repository, make an edit, send a pull-request.
Currently, Brazil, Russia and France are very actively using this common base.
We are now trying to transfer all the code of contractors (and we have a lot of them in different directions) to transfer to InnerSource. In the analysis, we created a map on two axes: the technical difficulty of transferring to the inner spacer mode along one axis, and the second axis, to what extent the translation to the inner spacers is generally useful. If the code is unique and is needed only in one place - perhaps you should not even touch it. But if a lot of teams use this site - just need it.
All of this increases the overall development culture. And it speeds up the work of several teams, because you can write a merge-request to any feature you need. His team is the owner team, and the one who is a contributor is testing. One of the important conditions is the presence of autotests and descriptions of any code in the repository.
When started, there was nothing at all. In parallel with the developers recruited devops. Now devops services are either provided as a service (to those who need it), or the product team has its own OPS, which already determines what and how.
The assembly is done in Jenkins, the code is run in Sonar (more precisely, already SonarQube). Sonar did not pass - there is no release. Testers write autotests, code owners - functional tests. The database is given as a service by infrastructure builders. Full load testing is done in rare cases, since the structures of the test base and the main one are different: on the preprode, we have chaotic data, not real impersonal (this is one step in the future), so you need to roll gently and not all at once.
Soon we have to go to Kubernetes (and some of the new products like the marketplace are initially there), as soon as we understand the transition plan and agree on all the details in the infrastructure.
My colleagues and I will continue to talk about how much of the work is arranged, because almost everywhere you can go on forever. Well, you can follow our reality show (because everything is just being built and changing quickly) and make bets whether we screw it up or not.
The fact is that "Leroy Merlin" is in many countries. The parent company - in France, is called ADEO. They write code for France, Italy, Spain and Russia. We have different business models: if we keep the minimum prices on the Russian market (below all competitors in the monitoring), then in Europe everything is different. In fact, the differences between the sea - from the characteristics of the locale and ending with other legislation. There are features of the infrastructure of Russia (the same very large delays to Khabarovsk) and another life cycle of the order. All this gives rise to such a hell of code, consisting of huge blocks of IF:
')
Two years ago we had 60 stores and many, many hotelok on features. They rolled in about six months and not always right. The last straw after the pile of features rejected by low priority was the request to create a string field in the order, so that we could only parse it later. It was necessary for the features of delivery in Russia, since the country is larger than other countries where LM is present. We were refused and this, more precisely, was told that it would be somewhere in seven to eight months.
The semi-annual cycle of the slow head company did not suit us. Naturally, we offered to write our code, give it back to review and wait for implementation ... True, nothing good came of it.
Why features were implemented slowly?
The story is very simple: despite our dozens of stores, we are not the first in the world. There are needs of France (where the parent organization), there are needs of other countries. They are prioritized by possible profit or cost reduction for a group of companies and are performed in that order. That is, our features are almost never done, unless they are very lucky and in some kind of sprint someone will finish their part earlier and go to disassemble the bottom of the backlog.
The second peculiarity is that when rolling any feature into the master branch (here in this Legacy code on the IF-IF-IF) you need to check how it behaves in other countries. Beg quoting 441869 with Bash:
Programmer: Well, imagine that you are a writer and support the project "War and Peace". You have TK - to write the chapter as Natasha Rostova walked in the rain in the park. You write - “it was raining”, you save, the error message flies: “Natasha Rostova died, the continuation is impossible”. Why did she die? You start to understand. It turns out that Pierre Bezukhov has slippery shoes, he fell, his gun hit the ground and shot at the pole, and the bullet from the post ricocheted at Natasha. What to do? To charge the gun with idle? Change shoes? We decided to remove the pillar. We receive the message: "Lieutenant Rzhevsky died." It turns out that in the next chapter he leans against a pillar, which is no longer ...
In general, when we introduce something for the Russian banks, somewhere in Brazil, someone can ricochet.
This means a very large test coverage, long pre-release procedures, and an unwillingness to support code that is growing rapidly. Therefore, the fewer features - the better. But you stay there.
The position as a whole is very clear, and I would have done just that in the place of the head company. Because it is rational.
Solution to the problem
The first approach was on the forehead: we offered to open the code to us to make pull-requests there. It was supposed that there would be about 10 developers who would quickly code business-critical functions, give to the French, those who would test according to their own procedure, and everything would be fine.
Actually, these people were already there: we were engaged in cutting out extra business logic that we received from other countries, such as various cumulative discounts and unusual actions (which are simply impossible with our business model) and put dummies on it.
The French in ADEO wanted a release cycle for rolling and installing new versions through themselves. They accepted our proposal for one branch for experiments.
Gave access, began to take the code on the review. It turned out, it is still slow. Rollauts for several months is not the case. Well, won a few weeks, but still the process did not fit.
Then, for six months, an important feature for us in the content part (managing data about the product that the client sees) did not come out. We sat for six months without release. So they didn’t have the feature, they didn’t pass the tests, they didn’t realize exactly what we needed. As a result, they sat on the key system for a long time without updates. There were NodeJS + PostgreSQL + Couchbase + Elasticsearch + Angular on the front. In the code, due to a number of archaeological layers, things like misuse of the SQL database and non-relational database were encountered. In one of the places, a huge piece of master data was taken, inserted into one field of the SQL database, and then split into entities in the NoSQL database. On one page of the site with the display of goods accounted for dozens and hundreds of such requests. With further reading of this legacy, the hair on different parts of the body began to move. We understood what features of the layers of Legacy the head department is facing.
Own development
The first idea was to make features together. In place, that is, in France. The four of us went to France and began to sit next to our colleagues in order to sort things out together and do what we needed.
Did not work Anyway, everything was very slow.
The second idea was to fork all that already exists and gradually finish it. We sat down by an architectural committee, evaluated the prospects, calculated approximate development plans, and realized that we need to choose another method. Specifically - yes, do fork, but not to develop the existing code, but in order to break the monolith into parts and install microservices where it is needed.
That is, we planned to rewrite entire blocks of logic in the form of our code. And for this, they began to switch part of the services to what could be done with us. Began to recruit developers. Then we completely followed the stack of the head company - Java, Spring and everything, instead of Couchbase, it was Mongo (this is a similar NoSQL base).
As the project progressed, they realized that there was a lot to do in their own way (because it’s faster and easier not to support the legacy we don’t need, in particular), and began to expand on other technologies as well. Then there were Java 7, Wildfly (formerly JBoss) and SVN. We asked why they are not switching to Java 8, GIT and Tomcat. It turned out they would not mind, but after a couple of years. In the meantime, family, write on the old stack. So we word for word and decided to separate completely. Business figured out what the question is, what are the pros and cons, and fully supported.
Almost immediately, 30 percent of the code was thrown out, and many microservices were written around. From what they did not touch, this is almost the entire core of business process transactions for money.
Naturally, the first thing we thought about was how to delimit the areas of development. About this, I would have told you separately, but in general terms the scheme is this:
The horizontal is the business area. For example, everything related to customer relations (the first green cell) is one area, and there is a set of applications from one department. Such a separation by the objective function creates several duplicates of the code in different places and requires a good corporate bus (again, a separate story), but it allows you to clearly find the ends and solve the problem before the result. Looking at it in almost three years, I can say that the architecture was chosen correctly, but knowing what I know now, I would make a couple of changes.
Now we have come to the conclusion that in the general structure there are many feature-teams that make up large product teams. At the same time, the product team includes not only the developers themselves, but also designers and business representatives. Since the ultimate goal of any IT department in a company is either to increase the speed of business development or to reduce costs through automation, we need business representatives within these teams. Retail is all about IT. There is not a single process that could be called “non-IT”.
A feature team is a small group of people (usually an analyst, a tester, a developer, a beck and a developer of a front, an ops). The analyst usually plays the role of the owner of the product or a person from the business comes to this place. The owner has backlog, priority, tasks. He dictates their design. The developer chooses an implementation option himself, discussing in the team what will be done and how. We do not have timblids to assess the task, the code and the decision. Everybody does everything. Usually, more experienced team members give opinions that many are willing to rely on. But anyone can make a decision. Of course, there are conflicts when two developers cannot agree on the implementation of the feature. If the conflict turned to the individual - need an escalation to the head. But the most part is the choice of implementation solution. Then everyone stands and draws near the board until they find an option that suits everyone. Usually it turns out quickly, but there were cases when they argued all day. When a dispute comes to a standstill, they often call the referee - a person from another team whose opinion everyone trusts. He explains the essence of the problem, he decides. In theory, even a john or a trainee can participate in this dispute in theory, but I know only a couple of such cases — usually the juna have a mentor whom they listen to.
An example of a feature-team: we have a service platform that allows us to buy a service from a contractor along with the product. For example, bought a door - you can immediately put the installation in the basket so that the independent master comes and does it according to our standard. We will check and pay, we will give a guarantee. So, this team makes a product, for this it writes IT decisions and make business decisions, change processes in stores. Agree with contractors. There - the owner of the product from the business, the shopkeeper, architect, developers, analysts and testers. They immediately use our corporate API platform to deliver all the data back and forth and write microservice. Such an approach — immediately, operating companies and business people in conjunction with developers and a small team — allows you to make a product very quickly. But I think it would be better for them to tell later on how this looks from the inside.
Initially, we did not want to do individual testers: there was a trend that a developer should either follow the TDD methodology, or test his code to the end. In fact, it was not very effective. At first everything went well: I wrote - you answer. You need tests to make your application work properly. But then, the more tasks and applications became, the more difficult it was. Some applications died off, some went on the prod and did not change for months. It became hard to write tests, support them and so on. Analysts in teams changed. Relatively recently, they agreed that they were wrong: testers are needed. But developers don't stop writing tests and - sometimes - do TDD. We realized for ourselves that the tests that check the functionality (the application works correctly) and the tests that check that the application works in problematic situations are different people who should do it. And for the second, testers are needed, since they cover possible strange cases not only with unit tests.
Now purely developers 60 people - back, front, full steaks. There are also analysts, testers and support. And plus an additional seven devops. But still, 200 planned people from the headline are not recruited, so we are now looking for new people, because the field of work is huge. Jobs are on My circle , if that. That is, from the development we now have 74 of about 200.
InnerSource
Given that we have a lot of independent teams only in Russia, and there are still teams that are sawing something similar in many other countries, we are moving in the direction of the intersource . This is very similar to open source within a limited group of people.
Within the ADEO group of all divisions by country, all code lies in the cloud Githab. The project is made out according to the same rules. Restrictions on style, technology and tools there. When you have open source and clear design rules - any developer on the stack can contribute. In order to log you into the code, you need to read the dock, clone the repository, make an edit, send a pull-request.
Currently, Brazil, Russia and France are very actively using this common base.
We are now trying to transfer all the code of contractors (and we have a lot of them in different directions) to transfer to InnerSource. In the analysis, we created a map on two axes: the technical difficulty of transferring to the inner spacer mode along one axis, and the second axis, to what extent the translation to the inner spacers is generally useful. If the code is unique and is needed only in one place - perhaps you should not even touch it. But if a lot of teams use this site - just need it.
All of this increases the overall development culture. And it speeds up the work of several teams, because you can write a merge-request to any feature you need. His team is the owner team, and the one who is a contributor is testing. One of the important conditions is the presence of autotests and descriptions of any code in the repository.
Some more nuances
When started, there was nothing at all. In parallel with the developers recruited devops. Now devops services are either provided as a service (to those who need it), or the product team has its own OPS, which already determines what and how.
The assembly is done in Jenkins, the code is run in Sonar (more precisely, already SonarQube). Sonar did not pass - there is no release. Testers write autotests, code owners - functional tests. The database is given as a service by infrastructure builders. Full load testing is done in rare cases, since the structures of the test base and the main one are different: on the preprode, we have chaotic data, not real impersonal (this is one step in the future), so you need to roll gently and not all at once.
Soon we have to go to Kubernetes (and some of the new products like the marketplace are initially there), as soon as we understand the transition plan and agree on all the details in the infrastructure.
My colleagues and I will continue to talk about how much of the work is arranged, because almost everywhere you can go on forever. Well, you can follow our reality show (because everything is just being built and changing quickly) and make bets whether we screw it up or not.
Source: https://habr.com/ru/post/455710/
All Articles