“For us, it makes no sense to use Retrofit”: about Android development in Sberbank Online

How many Russian applications on Google Play say “50 000 000+ installations”? Obviously, each such case is a unique story with its own specifics, so it would be interesting to talk to the developers. And when such an application also has a rating of 4.6, this increases interest.

Vladimir Tebloev is one of the people working on the Sberbank Online Android application. In the spring, when Sberbank Technologies participated in our Mobius conference, he delivered a report there, and now we decided to ask Vladimir about the features of his work.

')
- First, tell us what exactly you do?

- I, in the Sberbank Online application, are engaged in the Dialogs service, which allows users to transfer money in one click and see the entire transfer history in full view. The service is available to all users of the application - now it is 37 million people.

I have been working at SberTech since the summer of 2016 - then, within the framework of the application, there was still no division into separate teams. And later, when, within the framework of the transition to agile, we began to assign different commands to individual modules of the application, the Dialogs command was one of the first, and since then I have been in it.

- All the words “Sberbank” and “mobile development” are associated with Sberbank Online. But in such a large company, there is probably also an internal mobile development? Is it different from the outside?

- Yes, there are also applications for internal use. I have nothing to do with them, but I know that React Native is actively used there. Internal development has its own requirements: there is no strict design review and heaped animation, development proceeds faster with the use of a cross-platform solution.

When the expertise grows up, it will be possible to apply it on the “combat” application. Although the fact that Sberbank Online will be able to actively use cross-platform development, I doubt it. There are many difficulties, and when you have tens of millions of users, even a rare problem can touch so many people.

- And how does this “even a rare problem affect many people” affect your work? Do you have to deal with some exotic problems that smaller applications may have under the radar?

- Sometimes there are problems on some "special" devices. On one custom, but widespread firmware, it was strongly “fired”, and we had to sort it out for a long time. It turned out that the problem is in the driver of the motherboard of the device itself - he tried to emulate libraries under ARMv5, although in the project they were only under ARMv7.

When there are a lot of users and the price of an error is high, this leads to the fact that you need to roll out everything “by a little”, carefully watching the reports. If something sharply increases there, we immediately stop the rolling and make a hotfix. In addition to rolling out “for a given percentage of users,” we also geographically roll out everything in parts: Sberbank has the concept of a “territorial bank”, and features can gradually be rolled out by region.



- As long as any hipster startup can put on a high MinSdkVersion and say “everyone else is not our audience,” you have a different situation, you cannot wave a hand at people. What is your MinSdkVersion value now?

- Now it is 16, and we raised it literally last year from 14. We look at the number of clients with a certain SDK, and we can raise the version if it becomes less than 5%. So far we have a lot of users on Android 4.4 KitKat, about 16% - we need to support them.

- Is there something from the new versions, what are you staring at now and think “As soon as we increase MinSdkVersion, will we use it right away?”

- Of course, I would like to raise our minimum API to Android 5.0 in order to take full advantage of such innovations, such as transition animation, which will work adequately and everywhere. But, in principle, this does not apply to writing functionality, business logic, so this is not critical. In general, the animations are worked out by our designers, that is, it can be implemented manually. So this question is not critical, it concerns the comfort, “peace of mind” of the developer.

There are some cases in which we check the version, for example, SSL pinning. It works differently in different versions of Android, so we are implementing two versions of the code for Android devices "up to 4.4" and "from 4.4".

Of course, I would like to “just develop under Android P and not think about anything” - but this is always the case, this is not going anywhere.

- To mentioned SSL pinning. Obviously, security issues are very important for the bank. How does this affect you? How is your work different from working on a non-banking application?

- It has a very strict approach to the user's personal data. Any leakage is a huge risk. We have a security department that tests our application before each release. If there are comments, work on them goes to the team responsible for the functionality with the detected vulnerability.

In small companies, I think, there is often no security department that will pentest the application. If any shoals are found, they can pop up on w3bsit3-dns.com or a similar resource.

Also security is the fact that we have an antivirus application. Some users are dissatisfied with its presence, but the introduction of antivirus has given us a very noticeable reduction in fraud. For example, in our SMS bank, where you can write a text message “transfer to X card Y amount”, you managed to reduce the fraud level in this direction to a minimum with antivirus.

- Also for security reasons, banks limit the functionality in the case of rooted smartphones. What is Sberbank Online for rooted devices prohibited?

- In June, we abandoned limited functionality for owners of devices with root-rights. Now all Sberbank Online users on Android have full functionality available. At the same time, the protection remains at the same level thanks to the fraud monitoring system.

- And in the name of security, it was necessary to limit in some way not users, but themselves as developers, refusing what they would have used otherwise?

- When in 2015 we wanted to implement Retrofit, he had problems with obfuscation, he worked crookedly with the standard obfuscator. Our security department pointed to this vulnerability, fraught with cyber attacks on the bank and the risks of breaking the code, as the API sticks out. Then we abandoned Retrofit and still do not use it. As far as I know, now there are problems with the standard obfuscator already fixed. But we have since written our own HTTP client, it works and satisfies everyone, many wrappers have already been written for it for different commands working with different servers. Sense to change it to Retrofit anymore.



- The inevitable question: what do you have with Kotlin?

- We are going in his direction, but slowly. The difficulty is that many Android developers of different levels are working on the application at once, someone knows Kotlin perfectly, someone does not. In general, there are no insurmountable obstacles for the implementation, but now we have a shortage of viewers for looking at the Kotlin-code. If tomorrow we all start writing abruptly at Kotlin, then people will “tear up” in requests. In addition, in the case of Kotlin, there are problems with the static code analyzer that is used in our pipeline.

So Kotlin is implemented in small steps: for example, we write tests on Kotlin and use data classes (so we save time not to write tests for getters, setters, equals (), hashCode (), etc.).

Now it is running around slowly, and in the next step we want to write our DSL for testing at Kotlin. And in parallel, we want to raise the level of Kotlin knowledge in the company: for example, with the help of metaps.

- In the case of the "Dialogues" you are messaging, but not a direct equivalent of WhatsApp. And therefore it is interesting: how much are other people's solutions useful to you, do you climb into the code of open-source messengers?

- It was useful when we wanted to add emoticons. We had a question how to make a panel with them, and in the open source we saw an option where everything is easily solved by a pop-up above the keyboard. Then everything converges in height, and it turns out seamlessly for the user.

But in general, looking at other people's decisions is not always good, it is more efficient to form your own, taking into account the experience of others. For example, it is better not to look at Telegram at all, because of the huge size of classes in the source code of their Android application is not easy to understand. We are trying to go our own way, especially since the interaction with the server can be different: in the same Telegram it is MTProto, we have the usual WebSockets.

- I am a lazy interviewer, so I decided to just take a list of things with which you are connected with work, and ask about each item “tell me exactly how things are going with this.”

First point: you are engaged in the "application module architecture". About the fact that the application is divided into modules, already said - and what else can you say about the architecture?

- It develops iteratively with us, versioning is underway, now we have already reached the 17th version.

On the 16th, Clean Architecture was implemented. They agreed on who was responsible for what (presentation, domain, data layer), which entities and where should be used, where the converters should be - in general, they painted all the architectural issues and implemented them.

Implemented as follows: all new features had to be written on our new architecture. If in a pull request something deviates from a given norm, then such a pull request is left for revision. But at the same time they did not immediately rush to saw through all the old functionality, because it can cause many problems.

For the presentation layer, we chose the MVP standard, but some teams use MVVM here. In the presentation layer, we are not limited by anything. For example, we sawed our chat on MVI - more precisely, on its interesting implementation of MVI, which is completely different from what was written by the developer Mosby.

Then we switched to the 17th version of the architecture and implemented RxJava, which resulted in architectural changes. If we use strict definitions, now our architecture has turned out to be hexagonal, from Clean we have forged. But they are similar in that both work on the principles of SOLID, so one into the other pretty smoothly flows. We are working on it now.

In future versions of the architecture, we want to abandon the Moxy framework used to implement MVP, because it causes some difficulties. The project is large, it uses annotation processing, and when changes are made to the "lower level" modules, the build time is large. And we strive to make life easier for our developers.

- The second item is the "optimization of work and memory consumption." How acute is this question, do you have to constantly think about it for the sake of users with old devices?

- This issue is in focus of platform teams, they are developing tools that feature teams use. The need to do this, rather, arises from the need of one of the teams. For example, in our team “Dialogues” in the early stages of development, the chat worked very slowly. Then I had to roll up my sleeves, start with the profiler, see where the bottlenecks are in the application, and deal with the reasons for their occurrence.

In terms of optimization, for example, we abandoned the PNG and gradually clean them out of the project in order to use only the vector. This year is planned to optimize the dependency graph in Dagger to speed up the cold start of the application.

- Let us turn to the questions of testing: how is it going with you?

- I can only tell about our team, in others this process can be built differently.

Our team initially had one tester. Subsequently, he was bored just to test. And he began to ask us to help deal with writing unit tests. We showed him how to write tests on the database, on entities, on parsing - and so he unloaded us, took off some of the work. This is good: it is interesting to him and to us.

Over time, we came to the conclusion that we need to automate the regression, we need to write UI tests. The first time I and my partner were working on the UI tests, and later the quality department joined us - our testers who had tested the backend in the past. They know Java, and now they are connected to our project to automate the entire regression. We sat down together and reviewed the solutions that are: Appium, Espresso, Selenium.

We stopped at Espresso and began to develop approaches together. To facilitate testing, we developed our own framework, something like Kakao. We started this work in the beginning of 2017, and now we have a large framework, and most of the tests are collected as a constructor, because there are many gamers and action games written for various situations.

Now our testers are actively asking us to teach them how to write UI tests, because it is easier to write a test once than to “poke” the same actions on five devices. But, of course, you are not automating everything, and some cases still need to be manually checked.

As for developers, our team takes a bi-weekly retrospective. On one of them, we came to the conclusion that developers should conduct at least alpha testing after they wrote a feature. In order not to get out of some very basic bugs like "application crashes at startup." Thus, the developers also joined the testing. When we are preparing a major release and we need to quickly test the feature, everyone sits down on the regression and together pass the regression tests. When a bug is detected, the developers disconnect from recourse, quickly fix, and a new one.



- The next item: code review. Do you have any specificity, or “like everyone else”?

- There is a specificity caused by the number of developers. When there are ten mobile developers in a company, two or three people can review everything. How to revise the code of hundreds of people? We have developed a "revision matrix". We selected 20-30 people, about whom we know for sure that they can be a good expert, leave feedback and resolve controversial points in the comments. They took these people and divided all the teams between them.

Why the matrix? This is done to ensure that all reviewers have the same load. How is the review? We need a team of at least three appruva. The first is from someone from the team. The second is from someone from the outside, from a team that does not handle this functionality. And the third appruv - from someone from the adjacent team. In our case there are several adjacent commands, and they all watch our code. Well and, accordingly, all builds should be built: unit-tests and UI-tests should pass without problems. Thus, we have a code review.

- The next item is the refactoring of the Legacy code. How systematically does it take place: exactly scheduled tasks, or “did it take to make changes to the old code - did they factor at the same time”?

- In general, we have a kind of “scout principle”: if you touched something old - be kind enough to do it right, you are now a coauthor. But planned refactoring is also there. For example, for the “Dialogues” it was necessary to refactor two directions: the contact book that we use and the translations. The contact book was taken out, cleaned, rewritten the entire database on the Room, carried in a separate module. And our payments were written a long time ago with the help of RoboSpice, if you still remember this, and it hurt us. It must be said, cutting it out turned out to be an unpleasant task, because there were a lot of strings on it. And it was necessary to clean out finely so as not to break the rest of the functionality.

- Back in Sbertech you were involved in the training of programmers. What does training inside the company look like?

- Since September, it is planned to conduct such a program as retraining of internal employees. Now we have already defined a range of topics on Java and Android. For example, we have javascript, javista and analysts who want to retrain on Android. For them, such a school will be organized, where there will be purposeful study, according to the schedule and with lectures.

And now we have regular meetings. The choice of topics for them is not the same as at conferences, where something new and high-pass is absolutely necessary. For example, if we know that developers have problems with something, then it is important to tell about it. From recent - one of our developers talked about vector graphics. Not just about a specific library that beautifully draws vectors on Android, but started with how vector graphics work, and then went to the private. They told both about Room, and about Java concurrency, with which many developers have problems, and about Dagger 2.

Last year we had a school of Android development, and we hired those who successfully passed it. Such people should not be immediately connected to any projects and left to cook on their own. Therefore, a mentor is assigned to each newcomer employee and also to the junior, who will guide him, develop and assist him. This is internal training.

- Interviews: do they take place “like everyone else”, or is there a specificity?

- I used to think that “like everyone else,” but in the end it turns out that they are still a bit special. In my experience, there are three common approaches on the market. The first is asking for three or four topics and evaluating them exclusively. For example, I came to the company in the position of an Android developer, and they consider me as a person who should know perfectly well the algorithms and synchronization in Java, and at the same time do not appreciate what I do in the super library.This may be due to the fact that the company needs a person who must perfectly know some narrow part of the framework or language. The second is when the interviewer is careless, almost a conversation for life within 30-40 minutes. Here, rather, it is a matter of competence and experience of the interviewer. The third is when at the interview they talk about the company's problems and try to get some kind of solution on the spot. The disadvantage of this approach is that the decision may not coincide with the opinion of the person who is asking this question. In my opinion, such approaches are found in about half the cases.

As for us, we have worked out a method for considering a candidate in four broad areas: OOP, OOD (Object Oriented Design, Architecture), Java Core, and Android SDK. Methodically, question by question, we go through all the topics. If the candidate as a whole is confidently answering the topic, gradually we begin to go deep into, ask more specific questions. Figuratively, it looks like a tree: we have a root, from where we go into each topic, and we can go five or seven steps deep. Then the candidate is evaluated in the aggregate for all the questions passed. If the interview passes quickly, then we start asking for libraries, for example, Dagger 2, RxJava. If there was enough time for this, then according to Kotlin. Thus, the candidate is evaluated as a whole. If a person does not understand any one topic, but knows the other well, this does not meanthat he is a bad programmer. This means that in a certain period of time he should pull up this topic.

- Among your work tasks there is “research and review of new technologies”. Here I would like to ask for a specific example of some technology that has been reviewed.

- The last large library is RxJava, we considered how it can affect our project. Tested in local branches, then implemented in one non-critical module to see how it behaves in production. After all this, they took it as a standard and determined everyone to write new functionality on it.

Of the unsuccessful examples, we considered Retrofit, about which I have already spoken: a good library that solves its problems, but its time for our project has passed. Implementing it so that we have many ways of entering the network is a bad practice.

We also considered the TinyMachine library for the implementation of the state machine - the library is simple, not expandable, that is, it satisfies one command, but is not suitable for others. Therefore, it was abandoned, because if we were to “drag” the library, then only the one that suits everyone. As a result, we decided to write our own state machine, thankfully, this is not some kind of rocket science, which is hard to implement.

- And last: record keeping. In your case, when there are a lot of developers and it’s impossible to keep everything in your head, without accurate documentation, there’s nowhere else?

- Yes.The first kind of documentation is Java docks. We have a local meme "checking Prilutsky": no Java dock - pull-request does not pass ("Prilutsky" - in honor of one of the leaders in the Android-developers team: he wrote so often about all requests that documentation to the code, and without it the code will not pass into the common branch that such a meme was born). Now developers already understand that every public method, every class, every constructor - everything should be described by Java-docks. All code should be covered by docks, even tests. To make it clear what this test is written for, so that, for example, the questions “What kind of payment? Is it payment from instant messenger or payment from payments? What is paymentTest? ”.

In addition, we have documentation in Confluence. When I came here, material design guidelines were described in the cloud, and there were a couple of articles about how we work. Now all global things affecting everyone are necessarily described in Confluence. For example, we need to insert certificates for accessing the repository, and the one who did this writes an article so that they do not write to the chat a million times what to do in case of a non-working certificate. Another example: it was decided to implement RxJava and Confluence describes best practices - how well to do it, how not to do it, and a reference to the sample. The simplest example: how to locate methods in a class so that everything is standard.

These articles are gradually but regularly written. Now our Confluence has grown to 200 articles on various issues. Such a tool helps including newcomers developers. They study Confluence, get an idea of ​​the internal kitchen of the development, in case of any questions they can independently figure out and make a decision, not always involving their mentor.