Big interview with creator Jenkins, Kohsuke Kawaguchi

Do you use Jenkins? Most likely, yes, because this is the most popular project of this class today. It was always interesting for me to talk to one of the developers and ask a couple of hard questions. Here we have not just a developer, but the creator of Jenkins himself - Koske Kawaguchi (Kohsuke Kawaguchi) .

As you know, Jenkins is an open source project with a MIT license. Most recently, the FOSDEM conference, the world's largest free software conference, was held. Free, open, with dozens of speakers from all over the world. This means that there can meet anyone - even the creator of Jenkins. With a small group of friends and colleagues from the JUG.ru Group, we arranged a sudden landing there and were able to record a good interview with the creator of Jenkins.

So, in our virtual studio Koske Kawaguchi (who will introduce himself and tell everything in detail just below), Ruslan Akhmetzyanov ARG89 from the JUG.ru Group and Kirill Tolkachev tolkkv from TsIAN , our regular speaker, guru Groovy, Gradle, Spring and the stack of technologies Netflix, whom You can know from the “Debriefing” podcast.

Ruslan: Hello. First, tell us a little about yourself and Jenkins?

Koske: My name is Koske, mostly I am known as the creator of Jenkins and the technical director of CloudBees. Jenkins is the tool by which developers automate the various stages of the software delivery process. I personally know several programmers from Russia who use Jenkins, and I will be glad to have the opportunity to meet others.

Kirill: I'll introduce myself too - I am a developer and also organize the Moscow Jenkins Area Meetup (JAM). I have extensive experience using Jenkins, especially Scripted / Declarative Pipeline. Can you tell us what the CTO of the Jenkins company is doing?

Koske: One of the tasks of my team is to interact with the community. The success of our company very much depends on the success of its open source project. Another important task is to teach the rest of the people in the company how the open source works. In general, we have a very interesting relationship between the company and the community: everyone learns something from each other, and we pay a lot of attention to this interaction.

You also asked what function the CTO performs. The fact is that this function has evolved as the company grows. At first, I was something like a lead programmer and did a lot of technical work. But as the company developed, other people gradually began to do this work. For example, Pipeline became a separate team. I periodically communicate with them in order to have an idea of ​​what they are doing, sometimes I give advice. But all decisions related to design and development, they make their own. In general, this evolution of my role was very interesting, I constantly have to adapt.

Cyril: If I understand you correctly, then your focus has shifted from technical issues to working with the community?

Koske: Yes, much of what I do is connected with this. There are things that can only say the founder of the company. Therefore, part of my activity is, relatively speaking, waving the flag and calling the community in a certain direction when a change of course is necessary, as it was, for example, last year. In addition, we together with the team are engaged in Jenkins propaganda: we explain what our goals are, what difficulties we are trying to overcome. In addition, when I have to make a speech about the general state of affairs, I arouse more confidence. So briefly my activity looks.

Kirill: And how many years ago did this departure from technical work towards organizational work begin? Can you tell Jenkins story and how your role has changed as the project grows?

Koske: The project appeared in 2004, and at first I was engaged in it in the evenings and on weekends, that is, it was a kind of hobby. But gradually the project grew. I spent quite a lot of time creating this platform on which other people could write their applications. Over time, an ecosystem and a developer community emerged, some of which were from Russia - for example, Oleg Nenashev (Twitter: @oleg_nenashev ), who wrote a very interesting subsystem on top of Jenkins. As the project gained popularity, the need for better user interaction and educating them became more acute. Therefore, I became more involved in these things. Finally, somewhere in 2010, Jenkins became so popular that I decided to devote all my time to him and set up a company. Since then, work on the organization of the company has been added to the work with the community. The company grew, and I began to ask myself the question - what kind of activity can no one perform except me? Both in the company and in the community there are a lot of capable people who are happy to take on additional responsibility. Gradually, they began to do much of what I used to do myself. So, in general, looked our way.

Kirill: Thank you. Today, Jenkins is the most popular CI system. How common is the commercial version of Jenkins compared to other commercial CI systems? For example, with Travis Enterprise or with systems that can work locally?

Koske: Jenkins Open-Source is a big project with a lot of users. CloudBees has a much smaller scale. Therefore, if we manage to turn even one percent of the Jenkins user base into CloudBees users, this will be a very large result. According to this principle, all companies based on open source products work. Another important question is how much we manage to help people whose problems our product has to solve. We calculate how effective our technical support is, and, if I remember the statistics correctly, in general, our services satisfy them.

Kirill: Do you mean those who use the paid version? Or those who have free, too?

Koske: And those and others. That is, if a person buys a product, he will be provided with support, but it can also be obtained without using proprietary software.

Kirill: So you play the role of an intermediary between the developers and your users, open source and commercial. Do I understand correctly that the tasks of the product vision also lie on you?

Koske: That's not entirely true, we have a separate product team. I am not a representative of the user, but I constantly have meetings with many people, users and not only, that is, I keep in touch with the community. For example, I have a heading "Reports from the front line" for other employees of the company. In these posts, I tell you how our users write software and what it means for Jenkins. So I'm trying to influence the decisions that the product team makes.

Kirill: Apparently, you play an important role in the company. Let's move on to more personal questions. What is your favorite Jenkins feature you could talk about from morning to evening?

Koske: Well, speaking from morning to evening will be difficult. I am now more interested in how the organization works - I think this is because of my function in it. When I was working alone, my attention was much more of an individual feature. Now I do not think so, so I find it difficult to answer your question.

Kirill: Then, perhaps, you remember some feature that you implemented years ago and which you were very proud of?

Koske: I can tell you about one of the recent significant features on which we worked - I think it will be more interesting for users. This is a project last year, Jenkins Configuration as Code . We saw that our users were gradually moving from managing Jenkins mouse clicks in the GUI to customizing through the configuration file in the git repository. The need for the Configuration as Code project was felt for a long time, and many temporary crutches were written to perform this function. But none of these undertakings were large enough. Finally, we managed to bring together those people from the community who were interested in this new project and create a fairly large and thoughtful tool. Many aspects of this project have turned out very well, and he has gained fairly wide popularity, which can not but rejoice.

We can also recall another project, Jenkins X, which is an evolution in a completely different direction. It was created specifically to work with Kubernetes. Thanks to this specialization, we can achieve smooth integration and hide the complexity arising from integration and due to the connection of continuous delivery processes. As a result, we allow to easily implement the best, in our opinion, practice. I think that thanks to this project, Jenkins will be available to a large number of people who perform quite standard actions and do not want to spend a lot of time on the configuration, who need everything to just work.

Ruslan: Do you use Jenkins when writing Jenkins?

Koske: Yes, the Jenkins project has its own Jenkins. In fact, we have a large instance that launches a review of pull-trials on each plugin and on the core. In addition, there is an instance for particularly important work related to security and vulnerabilities, since the development of security features occurs in a separate, proprietary repository. Finally, there is a third instance for even more important tasks, such as signing keys and things related to releases. In addition to these three, a separate copy works around the clock in my home.

Cyril: You mentioned Jenkins X. Jenkins himself looks like a Swiss knife, he can do everything with plugins. But Jenkins X, on the contrary, is highly specialized, it exists for Pipeline and working with Kubernetes. Which technical strategy do you and your company follow? Do you support only Jenkins and Jenkins X? Or, besides that, will there be another Jenkins XX for Mesos clusters, Jenkins XXX for Cloud Foundry, and so on?

Koske: I think a lot depends on the reaction of the community. Having a universal platform is definitely very important. The question is who exactly implements this flexibility. Today, users usually have direct access to it. This is convenient if you are a large company and are doing something unusual. But at the same time I know a lot of people who do not need this flexibility. Imagine that you came to the dining room and asked you to make a sandwich. You are offered a choice of seven types of bread, five types of butter, four types of cheese, and you need to decide on all the ingredients. But you don't need all this choice, you just need a tasty sandwich, and you don't want to figure out how to make it right. I know that many are exactly like Jenkins and want all the flexibility to stay on our side. Jenkins X is the first major step in this direction. If this project continues to be a success, I would gladly try to do something similar for firmware and IoT, mobile platforms, or machine learning. I think there are a lot of vertical markets in which people just need a tool that will allow them to quickly bring the product to production. You are from Russia, so most likely you know about JetBrains?

Kirill: Of course.

Koske: I think they follow a similar strategy. They have a very flexible foundation, on top of which are many more specialized add-ins, in which, in essence, the same elements are assembled in various ways. Users thank them for it, and I really like their approach.

Kirill: For many years I have been seeing the same picture in many communities, when I advise you to use certain plugins - by the way, now https://plugins.jenkins.io helps me a lot, there are all approved plugins. The problem is that people are trying for all cases to use one universal tool, which is often not suitable for a particular case. So now I usually recommend only one tool - Pipeline, this is the perfect tool that fits in all situations. But a new problem arises. People try to use Scripted Pipeline or Declarative Pipeline, not understanding how they are arranged inside. There is a problem with the infrastructure, with the establishment of correspondence between nodes, sharing data, there is unusual traffic between the agent and the master or problems with scaling on the master. Which tool, from your point of view, is better: one that indicates the only correct way to solve a problem, like Jenkins X? Or a tool like Scripted Pipeline, which asks you exactly what you meant?

Koske: I'm not sure I understood you correctly, but I'll try to answer. In Japanese, there is the word "Kata", I do not know how to accurately translate it. It is used, among other things, in judo. We are talking about strictly defined movements that students learn: for example, to raise a hand in a certain way in order to deflect a certain blow. I did a little bit of this, so I know the simplest ones. The task is to memorize a set of simple movements, a kind of patterns or best practices. But this is only the beginning. Knowing this base, you can continue to leave it correctly, if the situation requires it, without disturbing the basic movements themselves. You study your own space, your territory, but at the same time knowledge of the common base still helps you. If you don't know her, you will be asking yourself all the time - am I doing the right thing? Even if what you wrote works, you will still have doubts.

In general, your question reminded me of kata. I think with Jenkins we have a similar situation. Jenkins X provides the base, and as you begin to understand it deeper, you can, if necessary, move away from it with the help of plug-ins and other things. Of course, in Jenkins X you have to sacrifice flexibility in order to provide better interaction with Kubernetes. But the fact that Jenkins X pushes you to best practices does not mean that you are deprived of choice. By and large, this is true of any software. Just with Jenkins X, we had an important change in our way of thinking. Previously, we created only the key elements of the system - the platform and plug-ins, it was up to the user to assemble them. With Jenkins X, the community has moved to a new approach, and, in my opinion, this is a very important step.

Kirill: If you do not mind, I will have two more questions. What feature has brought you the most trouble over the past year? Each project has difficult periods - can you tell us how they looked from you?

Koske: We really did ruin our lives a few times. The problem is that as Jenkins grew, our project began to attract more and more attention from other companies and their security teams, who began to look for holes in Jenkins. Therefore, the number of vulnerability reports that come to us from outside has begun to grow exponentially. Sometimes it became a bit scary, especially if you consider that some of these requests came with a deadline, to which we had to close the vulnerability. This was especially difficult when it came to the features created by the community. In addition, when installing updates with fixed vulnerabilities, users sometimes have a problem with their work. We make every effort to prevent this from happening, because otherwise users will be afraid to install security updates, which is highly undesirable.

Kirill: Do you have some governing body that decides which features will be included in the release? Please tell us how such decisions are made and can users ask to include some features in the release? If they can, then who is given priority - CouldBees or open source Jenkins users? Finally, if you can, tell us about your plans for the next six months or a year.

Koske: There is a rather interesting point: CloudBees does not own Jenkins, they are two separate projects, each with its own decision-making structure. CloudBees is simply the biggest contributor to Jenkins, many CloudBees employees work in parallel in the community. For the past few years, we have been trying to create clearer Jenkins control structures that would do exactly what you have just asked. To this end, we created the Jenkins Enhancement Process ...

Cyril: Sorry to interrupt - is it shrinking like a JEP, just like in Java?

Kosque: Absolutely. Obviously, we did not invent this concept, but borrowed it from Python, Java and some other communities, because there she already recommended herself. The basic idea here is that the community should be able to express its opinion and come to a consensus before major work begins on some features. This is exactly what we are doing when CloudBees is going to create a new feature, so we have the opportunity to change the course depending on the response received. This is the planning element that we were able to incorporate into our work. Since we are an open source project, we cannot directly tell other project participants what to do. Sometimes they listen to our voice, sometimes not.

As for our plans for the next six months, most likely we will continue to work on many of our current initiatives, including Jenkins X. Some CloudBees employees and many third-party contributors work on it. I hope that Configuration as Code will become more popular among plugin developers. By and large, we have already created a foundation for it, so now you need to configure some plugins so that they can be properly connected through this system. Finally, if new JEPs arise, we will work on them.

Kirill: Well, let's hope that JEP is not patented :) Question to you as the author of Jenkins - whose idea was the Scripted Pipeline?

Koske: This is an interesting question. The fact is that many initiatives in the community often do not reach a critical mass and remain at the level of the experiment. Before we started working on Pipeline, in the same space, people had already tried to do something similar, and we had the opportunity to get acquainted with these attempts. So we were not, in fact, the inventors of Pipeline. One of these predecessors was the Build Flow plugin, which was written by one of the CloudBees employees before joining the company. I think the term “ecosystem” is very successful - everything really happens as in a real ecosystem, the technology created by someone is constantly evolving and changing.

Ruslan: Have you influenced the implementation of Scripted Pipeline?

Koske: Yes, on its original version.

Kirill: As we know, there are two ways to implement any DSL - statically and dynamically. Why was a dynamic approach chosen for Scripted Pipeline?

Koske: I'm afraid I do not quite understand what exactly you mean by static and dynamic approaches.

Kirill: With static DSL, we have some confidence in our code before execution. For example, with DSL in Java, you must know in advance all the APIs and interfaces. With a dynamic approach, we already carry out verification directly at execution. Even if the code is invalid, the machine will still try to execute it, it just throws an error if necessary. The declarative version of the pipeline allows you to eliminate many errors by narrowing the variability in the build script.

Koske: To be honest, it doesn't really matter to me exactly when the compilation happens. But I can tell you about the general installations that we followed when designing the Scripted Pipeline. At a certain stage, a significant number of Jenkins users were greatly hampered by the complexity of automation. It was necessary to ensure the correct interaction of many components. The code is compiled, tests are run, then you have to wait until the deployment is confirmed, and so on. In addition, even then there was a desire to write this process in the form of code - approximately then Infrastructure as Code was very popular. The developers wanted everything to be managed through a file in the project management system. From these two needs, Scripted Pipeline was born. However, its use required nontrivial knowledge and was not intuitive, so we wanted to make a more accessible Pipeline for everyone, not just for those who have to work with the Scripted Pipeline to implement complex orchestrations. Many needed clarity, convenience and accessibility, not Turing completeness. This is how the Declarative Pipeline came about.

Kirill: Today we have, firstly, Jenkins plugins, which usually have a static API with annotations and metaprogramming. Secondly, there are Scripted Pipeline plugins that announce additional DSLs. The Declarative Pipeline does about the same, there’s just less flexibility. What is easier to maintain - plugins for Pipeline or for regular Jenkins? I mean, by Jenkins.

Kosque: I would not want the idea to appear as if we are talking about two separate languages. It seems to me that so far we have not been able to find a language for Pipeline, which at the same time would be convenient for performing simple tasks, accessible to new users, but at the same time would have great flexibility. The freestyle job, on which Jenkins always stood, was originally very well designed. If you had to do something complicated in it, you could write a plug-in, and Freestyle Job then filled the space between these plug-ins. It's like a bar, which is obtained by pouring chocolate on the nuts. I think in Pipeline we need to apply a similar solution. We are already doing something in this direction - for example, the Shared Libraries feature. Yet it is clear that we still have much to do in this area. I have been talking about this to our product team and developers for a long time. Let's hope that I managed to influence them.

Cyril: My last question will be about the Jenkins life cycle. An important milestone for the project was the release of the Jenkins 2 version. Many features and many APIs in it had vulnerabilities. If you now look at the source code of Jenkins and its plug-ins, then we will face a very complex architecture, with a significant number of problems, which is extremely difficult to understand if the person himself was not part of the project’s history. From your point of view, would it be possible to break compatibility with previous versions and previous architectural solutions, if necessary?

Koske: It is important for the project that it continues to evolve. Indeed, as it grows, a certain technical debt accumulates. But I can not agree with your assessment of the complexity of Jenkins. We are talking about a project that writes about 800 people, there are 1600 plug-ins in it - with this in mind, it seems to me that it is very accessible. The situation is not as critical as you describe it. Nevertheless, it is really necessary to develop. Last year, I wrote a post titled “Transfer Shift,” in which I, figuratively speaking, waved a flag and urged us to revise some of our initial promises to users, including regarding strict adherence to backward compatibility, except for security-related cases. I hope that this topic will be discussed in the community, but so far I have not heard too much.

Ruslan: Will your backward compatibility policy be similar to that followed by Java? After all, it has backward compatibility with versions of 15 years old, and even 20 years ago. Or will there be something more like the situation with Python 2 and Python 3, which still do not work with each other? Are you ready to break backward compatibility to implement an important feature?

Koske: Honestly, I don’t think that it’s right to ask whether we’re going to be compatible as Java or Python. We need to think about what kind of software the user needs. One of the serious problems is that Jenkins plugins are created independently of each other, and it is unreasonably difficult to get them to work together. If, when solving this problem, we break user instances, then no one needs such a solution. And if, as a result, we get a system that works better and faster, then some compatibility violations may be justified. I think we have more flexibility in this regard than language developers, we do not promise to adhere to the same tight compatibility. We are thinking more about creating a new version, for example, Jenkins X. As long as the end result meets the requirements of direct compatibility, we no longer think about this issue.

Ruslan: Great, thanks a lot for the answers!

On April 5-6, Koske will speak at the JPoint conference with the report “Superpowers coming to your Jenkins” . In addition, it will be possible to talk with him in the discussion area after the report. You can learn more about JPoint and purchase tickets at the official conference site (do not forget that ticket prices will rise on March 1 ).