Tribes, guilds, build train and no TDD: how does mobile development work in Uber, Spotify, Odnoklassniki and Avito

On the eve of AppsConf 2018, we interviewed specialists from large companies about the features and processes of large teams involved in the development of mobile applications. What approaches to work are used, what pitfalls await rowers entering the huge galley. Whether the foreign origin of the company imposes an imprint on the workflow - read about it all under the cut.

---

Philip Uvarov, Android developer in Spotify. The company has been working for the last six months - in one of the platform teams that support other developers in Spotify. He lives in Sweden. Before Spotify, he worked in a Swedish startup, and even earlier in Avito.

Artem Olkov, Lead IOS Developer in Odnoklassniki. He is currently leading the iOS development of one of the products. In addition to the actual development, he is responsible for the architecture, design, distribution of tasks, coordination with the design, backend API, etc. In total, Odnoklassniki now has about 60 mobile engineers, who are divided into smaller teams. The team Artem - 11 people.
')
Maxim Efimov, Senior software engineer in Uber. In the company works two years, is engaged in Android-development. Included in the “rider payments” team, which handles everything related to payments in the Uber passenger application. Prior to that, he developed for Android in other companies - mainly to order, and even earlier - he wrote in C ++ (server solutions for SCADA systems). In Uber, within the division that deals with payments, there are several more similar teams for other applications, as well as infrastructure teams - a total of several dozen teams.

Evgeny Suvorov, head of mobile unit development at Avito: Mobile applications development has been around for eight years. I tried games, freelance, worked in outsourcing companies, in a large company, and then switched to product development.

---

Let's start with the features. What is the difference between the work of teams with a large staff of developers in the company?

Artyom Olkov (Odnoklassniki): In my opinion, the peculiarities are connected not with the scale of the company, but with how the processes are arranged inside it and what role the team plays in these processes. Roughly speaking, if your team makes a mobile platform on which other applications or services of the company are based, 1000 requests from different corners will arrive to it and without a good product manager, development will drown. If the team makes a stand-alone service without complex integrations, the process will look much simpler.

Maxim Efimov (Uber): In my opinion, the most characteristic feature is the speed of work.

Small teams work much faster in principle. But at the same time, large teams of the final gross product of the developer, of course, more, because a whole group of people is doing about the same thing. And from this follows a different view on how projects are generally implemented.

In small companies, we often fit into the terms, conditionally, up to a day or a week. We can calculate and plan everything in advance. In large teams, this is difficult to do, because everything is tied to a large number of people. Therefore, the approach to planning is somewhat different. Projects are made with certain tolerances in terms, and the focus is on the quality and the features themselves that we do. And only after that - the need to fit into the timeline.

Another interesting point: how teams agree with each other. If the project employs five to ten people, they can be easily assembled in a negotiation room and, after spending two to three hours, solve all the issues. And you can go on to do the project. But when hundreds of people are involved in the project, this will not work. We in Uber have certain communication mechanisms that allow teams not to interfere with each other, to effectively unite, etc. In small companies, in principle, this does not exist.

Philip Uvarov (Spotify) : I think the main feature is that I don’t know all Android developers by sight. And we have very much shared responsibilities. This allows you to always be in the context of what you are doing, and quickly enough to deliver products in its direction.

How does your team sync with others within the company?

Yevgeny Suvorov (Avito): Our teams are connected by one mobile application - Avito. All of them contribute to this product, that is, we have a synchronization point in the code base and functionality.

Philip Uvarov (Spotify) : We have cross-functional teams that deal with specific issues (for example, like us, developing analytics for mobile clients), are united into one big department - we call it “tribe” (tribe). As a rule, teams within one tribe are closely connected with each other, they have an active exchange of experience. Plus, of course, we have clients - these are other developers, so we support the solutions that we create for them.

Maxim Efimov (Uber) : We have a small team, each no more than 20 people. They are integrated into structural units that are responsible for large areas, for example, a mobile application. At the same time, the teams themselves are quite autonomous, because if everything is reduced to a single control system with direct subordination, we will get a very inefficient system.

The general product idea is delivered to individual teams through product managers or owners. There is a separate structure that unites these people and helps to form an understanding of how and where we are going. At some upper level, strategic goals such as concern for passenger safety can be defined. But the details are solved one to two levels lower. For example, we have certain security arrangements in countries where people mostly use cash to pay.

Artem Olkov (Odnoklassniki): We are engaged in a separate service. So let's say we are a startup inside a big company. Understandably, we live in the same infrastructure. In all other respects, we are strongly separated. Even as part of the integration, we often use the public API of our own tools.

Are there any problems typical for big teams? What do you have to face?

Evgeny Suvorov (Avito) : In my opinion, in a large team, processes are the first to suffer.

All guys are usually experienced, even juniors are able to solve technical problems. But problems with processes can easily slow everyone down, so processes are better automated.

And that means high-quality Continuous Integration, Continuous Delivery, test automation.

Philip Uvarov (Spotify) : I think the biggest problem is the spread of knowledge within the company. I may not have an idea of ​​what is happening in some distant teams. And the same is true in the opposite direction: many teams have no idea that we have a team of analytics in Spotify. This is where the scale is felt.

The second point is the speed of making innovative decisions: the adaptation of the same Kotlin and other new technologies. It's one thing to come and tell the five developers: “From now on, we write on Kotlin,” but it's quite another to say this to 100 developers. There is a huge infrastructure that needs to be translated, to somehow support these innovations (CI, etc.).

Artem Olkov (Odnoklassniki): Yes, there are really two problems: the transfer of knowledge and the coordination of actions, including modernization.

Do large companies have any proven ways to solve these problems?

Philip Uvarov (Spotify) : To solve the first problem - sharing of knowledge - we have such a thing as guilds. This is a combination of developers by function, for example, the Android guild, which every two weeks holds some events: presentations, lunches, where you can discuss pressing problems or share something, etc. We also have, again, guilds , are internal conferences. Plus there are mailings, etc. A simple human problem remains: it's difficult to keep up with all this.

In a separate line I want to highlight the internal training (advanced training). We have our own Data University, where you can learn to a data engineer. Now colleagues are thinking about creating the same scheme for mobile learning.

Artem Olkov (Classmates): We do not have pronounced guilds.

Anyway, the guys united by one task intersect at different meetings - they know each other, communicate in a smoking room or at lunch. There are separate chatki, for example, purely for iOS-nicks. Inside the team, of course, the issue is resolved easier - we all sit behind one "daisy".

If you have any questions, raise your hand, wipe the one you need - and he will answer you. To transfer knowledge, we use 15-minute morning rallies, where everyone tells how, what, why and why they did. There is still a certain layer of documentation, where the main points are put.

The second problem - the coordination of actions - is solved by competent management.

Maxim Efimov (Uber): In fact, I don’t see a problem in the same sharing of knowledge. I see that the sharing mechanism itself is different. In small teams, this is simply done in any way. Gathered - talked. And we have convenient processes that allow us to organize everything so that it works. By the way, I will talk about them in my speech at AppsConf 2018. The idea is that in our company almost all of the development is fairly transparent. People from any team can look at what other developers are doing, and use some of this for themselves.

Yevgeny Suvorov (Avito): We also organize meetings 2 times a week. We love automation, and this task is also automated. There is a process when, during the week, people cover topics that they would like to talk about at a general meeting of iOS or Android developers. And if there is a subpoena, we are going. As part of the meetings, the food teams tell about the features that they implemented in the product, because otherwise it is difficult to keep track of everything that happens.

We met from the very beginning, but it was with the growth of the company that these meetings became most relevant.

Plus there are chat rooms where you can discuss any specific issues.

By what principle does it make sense to divide a lot of developers in a company - by platform, by functionality or in some other way?

Artem Olkov (Classmates): It still depends on what you do and how you earn money.

I can, in theory, present the structure of an outsourcing company in which the division, for example, by platform will work. But for a grocery company, I can hardly imagine a different form of division, except by functional teams.

Because if you gather in a bunch of all iOS nicknames, throw tasks into them, without having a very short bridge of communication with design, backend or testing, you will have to spend too much time for coordination.

Philip Uvarov (Spotify) : We are all divided by product. For example, our team is engaged in analytics and it includes both iOS, backend developers, and many others. In my opinion, this is a very reasonable division of teams, which also leads to certain problems, but it works well on such a scale.

Maxim Efimov (Uber): It seems to me that the idea of ​​dividing teams into platforms - iOS, Android, backend - on a large scale will not work very well. It rather strongly separates developers from each other and as a result, synchronization of, for example, two mobile applications for different platforms will cost much more. And the profit from the fact that people in a team see only people from their platform, it seems to me, not so much. Yes, with the sharing of knowledge is easier, but is it worth it? It seems to me no.

Plus, it seems interesting to me that some commands do basic things that everyone else uses, for example, some buttons, lists, text input fields.

Yevgeny Suvorov (Avito): I agree . Such a structure is quite successful and we just recently introduced it in our Avito, at least in the grocery part of the business.

Our large team became, probably, when we had five developers, since self-organization was difficult at this number. It became more difficult for children to cut one feature, they had to somehow separate them, plant them in different corners, according to different features. Disagreements began in architectural and other issues, and communication became more complicated.

At one point, Avito had two big teams — iOS and Android development, 15 people each. And at this stage, we began to break up into grocery teams: the “customer experience”, “seller experience”, messenger, delivery, etc. were distinguished. This resolved the issue of priorities. Previously, many project managers came to the teams with requests for features, and for them each of these features had priority number one. As a result, we have 20 projects and their cross-cutting priorities are not clear. Higher people had to manage it manually. After the selection of multi-functional teams, each of which has its own development pipeline, its own priorities and resources, everything has become easier.

At the same time, we still have small platform teams that make, as we call it, “horizontal” solutions that roll out to all product teams.

Do teams often reorganize?

Philip Uvarov (Spotify) : Some movements occur every week. In our company, the teams are small and autonomous. If you wish, you can very easily go from one to another. How often this happens depends on the team and the direction in which you work. Where I work, it is not pronounced. But Spotify is famous for the fact that we work on agile and in many respects things like OKR, etc., have come from us. Therefore, the company's management is not afraid to change priorities, if it suddenly realizes that something is not working. We just switch to something else.

Maxim Efimov (Uber): We had large-scale reforms related primarily to the very rapid growth of the Amsterdam office. During the year, the number of employees almost doubled. Those teams, where people were recruited, became very large and it was difficult for one manager to manage such a structure. In this regard, the teams were simply divided into several "subcommands", each of which was engaged in a narrower sphere. I think this is a natural process.

Do you agree with the idea that in a large team, so that the quality of the code does not suffer, it is worth avoiding hiring over-qualified juniors and seniors?

Philip Uvarov (Spotify) : I think neither one nor the other. Every year Spotify recruits quite a few university graduates through summer practice (some of the people after the practice receive an invitation to work). Similarly, we easily take people with several PhDs.

Technical qualification is cool, but it can be taught. If a person does not know how to work in a team, then it will be extremely difficult. Therefore, in such companies almost more attention is paid to software skills.

And so that the quality does not suffer, we need good interviews that will allow selecting developers who are not below a certain basic level.

Maxim Efimov (Uber): We are repelled by a slightly different principle. We have the desired mix of experienced developers and juna. Just so that there is no situation when there is a crowd of June, and there is no one to help them and explain how to work. Therefore, we try to keep a balance.

I do not see the point of sticking to the principle of “not gaining too many seniors”. The quest for the seigneur is a rather difficult quest. There are many companies on the market, and the competition for developers is serious. People with experience successfully work in almost any place, so it’s unrealistic to refuse qualified personnel today and then recruit them tomorrow. These people will not sit and wait until we want to invite them. And in my practice, if a person has good competencies, it is not a problem to find a place for him in a company. But we must build on the specific situation. We need to make a decision depending on what ambitions each person in the team has, to look at upcoming projects, whether we can pull them out on our own, who we need. That is, it is necessary to go down on low-level planning.

Yevgeny Suvorov (Avito): In my opinion, when recruiting seniors, you should not be afraid that they have their own king in their heads or that they will not obey.

In our company, developers themselves offer ways to solve various problems. And seniors often have these quality solutions. They have experience, so problems can be solved more quickly with their participation. There is another problem - your tasks may not seem interesting or ambitious to the seniors.

For juniors, too, must be approached individually. When we were still one team, an intuitive feeling periodically appeared that the team could take out another jun. And at that moment it was possible to take a great ambitious guy who quickly grew to a quality engineer. In a team with well-established processes, learning is really fast. Yes, and the juna are different - some come after the university with burning eyes, but they do not know how, while others have seven years of experience backend, they just have just switched to mobile applications.

That is, we are not afraid of juniors, but we are guided by the feelings of the team whether it needs it or not.

Planning, synchronization, task distribution

How much effort does planning and synchronization of developers take within a large company (even if in small teams)?

Philip Uvarov (Spotify) : This is just a plus for dividing teams by product, not by function: we only plan our small product within the company and we often don’t care what the rest million developers do there.

Artem Olkov (Odnoklassniki): Here I can tell only about our specific team. We have the beginning of development, and it gives certain concessions - allows you to be freer. At the moment we only have daily 15-minute rallies on the current status and hourly closing of the previous / planning the next sprint once a week. Everything else is decided in a personal manner.

Maxim Efimov (Uber): In our case, not very big. Perhaps, times in 1,5 - 2 more than it took me when I worked at an outsourcing.

True, there are some processes in the company, such as code review, which inhibit development. Roughly speaking, until a team responsible for its piece of code looks at your commit, it may take just some time. But I don’t think it refers to a sync fee. It’s rather about how the whole scheme is set up on a low level - who is reviewing whom, how is waiting organized, etc.

Evgeny Suvorov (Avito): We initially solved the synchronization problem with frequent releases. As a result, now synchronization itself takes a bit. Everything is almost automated.

Do I need to distribute tasks in a special way so that the quality of the code does not suffer?

Philip Uvarov (Spotify) : In large teams, it makes sense to distribute the product by area of ​​responsibility. Thus, on tasks there will always be people who approach them responsibly, because they also live with it later. They do not change the context, i.e. there is no situation when all developers are involved in absolutely everything (Peter wrote 10 lines here, Vasily added 5 more below and as a result there is a mess that cannot be maintained later).

Moreover, if you work within a small “product within a product,” you should more or less understand how this whole system works from the backend to the client.

Evgeny Suvorov (Avito): As for quality, it is better to rely on testing and code review. For example, in my team now the guys with the mobile development background are sawing microservices on the backend in three languages ​​- Python, PHP, Go - which must withstand the load of Avito. But communications, test automation, etc. provide at the entrance of the junior, and the output - high-quality piece.

Can you remember one or two of the most interesting tasks that your team has been facing lately?

Artem Olkov (Classmates): As I said, we are making a new product. The most interesting task for me personally, as for the person who leads him, is his planning and design: to build architecture, think through what the user will have and what will not, and how the developers live with it.

Evgeny Suvorov (Avito): The most interesting thing we encountered in mobile applications was modularization - breaking our code base into separate modules that can develop independently without interfering with each other.

For example, viewing ads and the delivery screen in principle are not related to each other. But in the code base they were not separated, and when the developers did something, their colleagues were blocked. My team was just assembled around this division task. It took us about six months and it was interesting technically in the first place.

On backend microservices originated a long time ago, and in mobile applications this trend appeared quite recently - maybe a year or two ago. And then in large foreign teams, in Russia there are so few.

Technology stack

Have you encountered any revolutionary changes in the technological stack in your company / team recently? And in general, how do such processes take place?

Philip Uvarov (Spotify) : We rarely practice this. There are experiments on the introduction of new technologies, in particular, with the Gradle relocation to Bazel or with Teamcity on our internal system, but no more. I think that such revolutions are impossible by definition.

Maxim Efimov (Uber): I had such experience before Uber, but here it is a constantly ongoing process. There are a few people, including me, who favor Kotlin. There are certain things we need to figure out how to work. But overall this is possible. We are working on it now. We do not have to sit and wait until the "most important developer" says: "Now you write on Kotlin." Kotlin is already used in part of our codebase, but still in process.

On the whole, global changes in architecture are taking place, and they also come from our team. The initiative from people who want to solve some kind of personal pain, we have quite acceptable. If everything is done correctly, these changes are accepted.

Evgeny Suvorov (Avito): We have two platform teams that saw tools for internal use. Company growth and modularization have changed the way these teams work. They had to communicate a lot in order to understand what end users (developers) need.

In terms of technological change, we have Technology Radar. Often, developers say: “Let's inject this thing instead of the one that we have been doing for 5 years, because it is morally obsolete. ” To deal with these proposals, we have brought Technology Radar, in which the one who has the initiative has to climb into a small bureaucracy, do some research, express in digital metrics that innovation will speed us up, but something will slow us down. That there are some risks regarding the new technology. This will be reviewed by several colleagues and, if all is well, the technology will be applied. True, with Technology Radar offers become less.

Is it harder to carry out such reforms with the growth of the company?

Maxim Efimov (Uber): More difficult, but not by much. Suppose there are 10 people in a small company and someone alone wants to change something. To do this, he needs to negotiate with nine people. If you have 100 developers, this does not mean that you will need to come to an agreement with 99. Of course, not everyone will be actively involved in this, it is not as difficult as it could be.

Which of the latest technologies do you personally find most interesting from the point of view of application in large teams (even if they do not work in your company)?

Evgeny Suvorov (Avito): Probably, the key ones are microservices, division by domain areas, splitting into teams that have their own priorities. In terms of technology - everywhere its features.

For large teams, you need to be a little retrograde and not get into the alpha-beta version of the tools, because you never know what this will lead to.

Although there are exceptions. Due to the module partitioning, the Android project suffered - the IDE took the project model for a very long time and only the new early adopted preview versions helped. There were experimental ticks, with which everything became steeper to work.

Philip Uvarov (Spotify) : I think Kotlin is one of the super-promising things. It seems to me that soon everyone will forget about Java on Android, especially if Kotlin gets a little better with builds and we can go to it. As far as I know, many big teams - like Facebook - have long opposed Kotlin just because there are huge projects that have been going for three years without these problems, and Kotlin will meet for six years.

From the last one personally seems to me quite interesting Flutter.

Artyom Olkov (Odnoklassniki): I believe that the best technology stack in the context of iOS development is native, which Apple provides. Additional frameworks are often not needed. There are interesting tools, but they are built by very large teams only for their own needs. Outside of these teams, they often add just more pain.

There are a couple of very interesting approaches that are now in trend, and some of them are deservedly so. I want to mention specifically Unidirectional Data Flow.

What do you think about the gradual transition to Swift?

Artem Olkov (Classmates): This is a complex topic. Now we have started a new project and we are all on Swift, but there are 150 lines on Objective-C, because for some reason, we were unable, with the help of Swift, to do relatively briefly what we needed.

If you have everything on Objective-C and it suits you completely, is there any point in switching to Swift? For many guys I know, the transfer of the code base to Swift is HR-PR history, because there are a lot of new developers on the market who simply don’t know Objective-C and don’t want to learn it.

Evgeny Suvorov (Avito): This is our case. We got into Swift at one time and he slowed down our build time. In general, there were a lot of drawbacks, but for us one of the key points was that this is such a hype thing we can hire many developers for. In general, it justified itself to a certain extent. Swift is developing now, but Objective-C might have been faster, more convenient and fewer problems.

The idea of ​​long-term support can be a reason for the transition (there should be more developers in the future)?

Artem Olkov (Classmates): This question is not so easy to answer. Apple is still in a position where they can say that Swift is no longer supported, and give it to the development of open source. The probability of such a development of events is very small today, but it is not zero. Objective-C is still getting improvements.

Personally, my opinion (in Odnoklassniki they may not agree with him) - if you just take objective facts - in five to seven years both languages ​​will live, and developers will just need to be able to work with that and with that. It is not difficult if you understand the basic concepts. Knowing one, learning the other is not such a big problem. To date, iOS development is no longer about knowledge of the language, but about knowledge of the framework on which you work.

Development and Testing

When do you write tests? TDD or after writing the code?

Philip Uvarov (Spotify) : To be honest, given the launch of builds on Android, I don’t believe that you can effectively write tests before the code. We usually write tests in parallel, along with development.

Artem Olkov (Classmates): It all depends on what tests we are talking about. Unit tests on the conscience of the developer team.

Specifically, in our team, we cover Unit-tests with only certain layers, in which we need to be sure (we see problems on other layers right away). If these are integration tests (API, UI, etc.), testers are responsible for them as they become available.

Maxim Efimov (Uber): I know, part of our backend loves TDD, but I don’t know such people among mobile developers. , , — .

(Avito): . , , -. UI- unit- . , — , UI-.

. — . TDD.

TDD . Android- , . AppsConf 2018 TDD.

, .

- ?

(Spotify) : , . 100% - unit- .

(Uber): , . , . .

- «» ?

(Spotify) : — .

, - (proof of concept ), , . , , , .

(Avito): , , QA-, , backend, frontend . . , , QA.

?

(Spotify) : , code review — . CI: , code style — AppsConf 2018.

: pull request, code style, ; , . , git-. — , : , .

(Uber): Code review — , . , . .

(Avito): code review- -. , , -, , , . -.

, . . , . , , (« ?»). , , .

. , . , , . . , - code review. , .

?

(Spotify) : , .

(): , , . , «» «». — , . — — . . - .

. , Android- .

(Uber): , , — . .

build train, . - — . .

. , : . , .

(Avito): — 12 . , . , - ( , -).

-, . . . — . . , — , . , - . .

. , , . . . , , , . .

. AppsConf 2018ApplsConf 2018 . , .