"The development team is a self-sufficient organism": SEMrush about Java and freedom of choice

SEMrush stands out among Joker sponsors for several reasons. First, the direction of activity: the company is developing a platform for marketers. Secondly, with the participation of the company in the Java conference of its CTO, Oleg Krasnov is known for his love for C. Thirdly, for her this Joker will be the first.

Because of all this, it was just more interesting to ask about how the company is doing with Java, and about the approach to development in general, and about the expectations from the conference. That's who we talked to:
')

• Elena Danilenko, HR generalist
• Oleg Krasnov , CTO
• Anatoly Chichikov , Senior Java Developer

Elena Danilenko (HR generalist)

- SEMrush becomes Joker sponsor for the first time - what does the company expect from the first experience?

- In our company, most of the work issues are solved by personal communication. Therefore, one of our goals is to meet, tell about ourselves and talk with Java developers on topics of interest to us. In addition, we are very pleased to have the opportunity to participate in the support of such a significant event as the Joker conference. We are for more networking, good and different :)

- Sometimes IT-specialists dislike the concept of "marketing" - they are not frightened by the fact that you are doing a marketing platform?

- It is not the very concept of “marketing” that frightens off, but what is not very clear what exactly is behind this word. But we are not a marketing, but an IT company. We are not engaged in marketing, but in the development of a high-tech online platform for people who are professionals in this field or want to understand their position in the market.

And for us, it’s not the product area from which a potential colleague comes to us, but whether he agrees with our approach to work. We try to give development teams maximum freedom in decision making - and we work well with those people who are ready to be fully responsible for their product and the development process, working to improve the quality of both. We try to clarify this position as much as possible when communicating with candidates.

- SEMrush has offices in four countries, and where are the developers (Java is primarily interested in)?

- We are developing in St. Petersburg, Prague (Czech Republic) and Limassol (Cyprus). Most of all Java teams are in St. Petersburg: now there are seven of them. Next year we plan to increase this amount at least twice. Java developers also work in our Limassol office.

Oleg Krasnov (CTO)

“When a CTO of a company specializes in C, a Java developer who chooses a job may be worried that he will be in supporting roles in such a company. What do you have with this?

- The specificity of SEMrush is not in this or that language, but in the following: we consider that the development team is a self-sufficient developing organism. And the team can choose for itself what to use - including the development language.

This adds freedom, it adds responsibility, but it also makes the process of managing the situation more difficult. Management from the collection of reports and the use of the "stick" turns into persuasion and motivation. Communication of equals is to my liking.

I’ll add that our attitude to Java is like a good programming language with a large community and a developing ecosystem. Projects written in Java perfectly exist and develop, which gives us confidence that we want their number to grow.

- In 2014, you gave a talk "When to write your database." And in this connection, it is interesting: what is the approach to the question “take ready or make your own” in SEMrush?

The same as with the choice of language: the guys themselves decide which path to follow, to do something of their own, or to use a standard solution. Speaking historically, in a number of products there was a transition from standard solutions to self-written ones, because, as you know, a well-written private solution works better than the standard one.

If we talk about my personal experience presented in the report, then at the time of development there was neither a stable working Hadoop, nor Mongo, so I acted according to the circumstances :)

- In our recent material on Java hols vs C ++, you mentioned a product with more than 150 servers, and you want to understand: what is the scale of SEMrush? What else there are large products, how many total servers?

- Let's start with the "iron". Now we have about 440 real servers, as well as about 250 virtual machines. Technological stack of development, if we talk about languages ​​- from ANSI C to Node.js and, of course, Java.

Our largest projects are Backlinks, Domain & Phrase Analytics and Tracking. “Backlinks” is the largest and most loaded of them, more than 150 “machines”, its meaning is in building a database of backlinks of all Internet pages, a kind of own Google. Domain & Phrase Analytics is the oldest product of the company, more than 60 servers, search analytics on more than 30 local databases. Tracking is one of the most dynamically developing products, 28 servers, the dynamics of the positions of sites in search results.

These three products from the backend side are written in C / C ++, but we also have enough successful Java products - such as SEO Ideas and Site Audit, you can read more about them on our website .

Anatoly Chichikov (Senior Developer)

- What technologies / tools are used in the case of Java (both by you and SEMrush as a whole)?

- We welcome everything that facilitates the delivery of value to the user with minimal time. We also always try to keep up with the times and use the latest stable versions of languages ​​and libraries.

If we talk about the project to which I am involved, then from the Java framework it is, of course, Spring (Boot, Web, Data, AMQP). For text analysis use Lucene. In everyday life, we widely use Lombok, Guava, and the Java 8 stream. As for external storage media, these are MongoDB, Redis and RabbitMQ.

In other Java teams, technology may vary depending on the tasks, but often the kernel is Spring and Java 8. Someone stores large amounts of information in HDFS and uses Hadoop for processing. For someone, complex queries and transactions are important - their choice of PostgreSQL. There are commands that compromise the speed of insertion and complexity of queries — they use MongoDB, as we do.

Build and warm - Maven, Gradle, Fabric, Ansible. Many teams use Docker in production.

Tools - Intellij IDEA, TeamCity, YouTrack, GitLab. From interesting: our team quite successfully uses XRebel. And at the company level, we are now implementing the inter-command review code, trying Upsource.

- Is there any recent large-scale task that I would like to talk about?

- I think such a large-scale task I can call “SEO Ideas”: a project that our team started and has written for just over a year. His goal is to turn a bunch of incomprehensible to the ordinary marketer of metrics and data on competitors into concrete recommendations for action. People love simplicity, so while steadily growing.

This is a distributed stream processing system. There are concurrency, word processing tasks, working with external APIs and load balancing them. The server business logic is entirely in Java, the client part is written in JavaScript. In my opinion, Java as a language and as a platform confidently keeps hitting new HYIP technologies (NodeJS, Go, etc.) in terms of both performance and usability. The temptation or the need to change it we did not arise.

- In SEMrush, a lot has been done in C / C ++ - and you, as a Java developer, do not have the feeling “it would be better with Java”? :)

- The use of C / C ++ is a need for some projects, as these teams are faced with very demanding tasks. One of the most striking examples is the compilation of a reference graph and the ranking of the domains of the entire Internet. Using Java would cost us dearly, and the decision to use C allowed us to free up more resources for other equally useful products.

- Does Java fit well with the “self-organizing teams” approach?

- Yes, it is quite. In our case, the team fully takes responsibility for its experiments and product stability, and here Java and its mature libraries show themselves from the best side: we rarely shoot ourselves.

We also try to write, as far as possible, simple and understandable flat modules; preference is given to horizontal connections and constant readiness for changes. And here, too, Java usually fails us: it has excellent refactoring and dependency resolution tools.

- With the “diversity” of teams that independently choose languages ​​and technologies, do you have any difficulties with interoperability?

- No, there are no problems. The whole SEMrush is, in essence, a set of microservices, and we use JSON and REST as the “lingua franca” between them. In some cases, when it comes to a large amount of data or queries, binary solutions are possible, but these are exceptions rather than rules.

- Do you have some experience in Groovy and Scala - is this experience related to working at SEMrush, and are these languages ​​used in production there?

- I myself practically did not write in SEMrush in these languages. But in general, Groovy is used, albeit less actively, than much more. As far as I know, mostly QA loves him, they write beautiful tests on Spock. I myself had an interest in Groovy as a scripting language. In general, it is pleasant and concise, and I understand why people love him. However, it turned out that I write scripts mostly in Python, and I consider it to be the best tool when it smells like a chore.

And Scala, as far as I know, has not yet taken root. I myself had studied this language closely some time ago, and I also cannot say that we met characters. The language is not simple, it takes a lot of time to learn, and each programmer writes on it very differently. Looking through the Scala GitHub, you can sometimes feel the presence of an extraterrestrial mind. In my opinion, in an era of rapidly changing requirements and one-button interfaces, these are rather minuses.

I myself am now looking with interest at Kotlin, continuing to actively write the product in Java. What impresses you is that the language is simple, concise, safe and made by pragmatists for pragmatists. In our company, such tools are loved. I do not exclude that we will have a project on it, if a critical mass is formed that would like to try.

- Thanks for the answers, we will be waiting for you on Joker!