QIWI Server Party: DevOps beer is not a hindrance
Hello to all! The event, code-named QIWI Server Party, will take place very soon ! Meeting for back-end developers who care about business, not “effective workflows.”
There will be interesting reports and qualified speakers! For example, our colleague Sergey Chekansky will tell, "Why BigData is Useless!" Anton Dubenchuk will also share his experience with Kubernetes. Plus there will be a couple of interesting topics.
')
Well, of course, informal communication, beer and BBQ. For details, welcome under the cat!
This is our first event where we want to gather back-end developers for informal communication. The main goal is the exchange of knowledge in an informal setting. For our part, we promise four interesting speakers, two of whom want to introduce you right now.
The first is Anton Dubenchuk , the manager of the provider systems development group. Directions: microservice, Kubernetes, platform. We asked him a few questions.
Hello Anton. What is “provider integration systems”? Tell me more about your work? What are you and your team doing?
Yes, indeed, I am engaged in provider integration systems. QIWI has many different service providers with whom we need to integrate using different protocols. Either partners provide us with an interface that we implement on our side, or vice versa - we give them the necessary protocols and partners implement them on their side.
There are two types of integration: there are standard ones that simply consist in setting up access to the partner, and there are more complex ones for which the protocol needs to be implemented: what we call a gateway.
The purpose of all this work is to finally get the payment to the recipient.
In addition to connecting providers, our group is committed to implementing other, more complex, custom integrations with partners.
I heard that an important focus of your work is to build a microservice application architecture. Why is this approach preferred?
The decision to switch to microservice architecture was made not long ago - in early 2015. Up to this point, we had large systems and services that were strongly connected with each other. It was difficult to deal with them, to modify something, to release new functionality. After making changes, we and our colleagues had to be quite difficult to rebuild. And this is one of the reasons why we switched to microservices.
The second reason is that we needed to develop new products as quickly as possible, test ideas and convey them to users. The only solution we have seen is the transition to microservice architecture.
What technology stack do you and your team use? Particularly interesting to hear about how you apply Kubernetes.
If we talk about Kubernetes , then we now use it for testing. We recently introduced it in order to try and see how comfortable it would be to work with it. Later, with the experience gained in the test environment, we will implement it in the production environment.
As for applications, they are written mainly in Java. Databases - Oracle, PostgreSQL. Of course, we use Spring to build the insides of our applications.
We have also been developing an application server for quite some time, within which applications written in JavaScript can run. This is where we integrate with our partners.
Some of the approaches and practices that we have developed during our work, but which are not available in commonly used libraries, we took out in the form of our own libraries, which we can easily connect to various projects as needed.
Do you have a specific approach to development? Methodology?
From the point of view of project activities, we are closer to flexible development approaches. Initially, the project is discussed in conjunction with customers, lawyers, accounting, that is, with all the parties involved in the project. Developed preliminary decisions, ideas about the final version of the project. After this, the implementation process takes place.
In the process of implementation, we use iterations that take two weeks. We meet, discuss who did what, good luck and failure, plan what we will do in the next iteration.
If we talk from the development point about the tools and approaches that we use, then, of course, a version control system, code-review system, CI / CD, etc. are used.
Anton, what will be your report at the QIWI Server Party?
I plan to talk about our experience in implementing Kubernetes. I want to talk about why we decided to try it, what, in my opinion, the prospects and why the previous decision did not suit us.
Kubernetes is an open-source project. How actively is it developing, is there a large community? And what are his prospects?
In my opinion, the development he is experiencing sufficiently rapid. I observe how Kubernetes is actively and harmoniously developing - special groups have been formed in different areas of development, regular discussions are held, and pull requests are made. People gathered from different companies: Google, Microsoft, Red Hat, CoreOS, Huawei. At the same time, in the process of discussion, they are trying to develop solutions that on the one hand would be quite pragmatic, but on the other, they did not drive people and companies into too rigid limits. In the end, it turns out quite flexible system. Plus, given that initially the ideas that were introduced into Kubernetes, were borrowed from the practice of deploying applications to Google, they believe that the result will be a solution that will be able to occupy its niche.
The second of our speakers - Sergey Chekansky - project manager for machine learning and bigdate.
Sergey, tell us more about your work at QIWI? What is in the range of your tasks?
I do machine learning and big data analytics. The company has a huge number of data sources, combining which you can gain additional benefits. We merge, validate, structure data and build descriptive analytics on it or use machine learning models.
What technology stack do you and your team use?
For processing large data arrays we use Apache Spark and write mostly in Python, but separate modules in Scala. The tasks of loading, storing and retrieving information are solved by combinations of Apache Scoop, Kafka and own solutions. Storage rarely updated data is implemented in Hadoop, mainly in the format of parquet. Data that is frequently updated is written to Apache Kudu.
Do you have a specific approach to working with BigData? Methodology?
We do not have a methodology, but rather principles. Since the main tasks of an analytical nature, our principles are to implement the task so that the results can be repeated. If this is the result of some kind of analysis, then we are not doing one-time scripts, but something that can be repeated many times.
Sergey, I know that your report at the conference will be titled "Why Big Data is Useless." Tell me what are you going to talk about?
I will talk about how much potential we do not use in BigData technology, and why this happens.
If the back-end is your theme, wait! Admission is free, but you need to register so that we do not miss the number of seats and the traditional “admin” treats. By the same link you can find all the details.
Come, we will be glad to see you!
PS For those who do not have time to register or can not attend, a link to the broadcast , which will be available on September 6 from 18.00
There will be interesting reports and qualified speakers! For example, our colleague Sergey Chekansky will tell, "Why BigData is Useless!" Anton Dubenchuk will also share his experience with Kubernetes. Plus there will be a couple of interesting topics.
')
Well, of course, informal communication, beer and BBQ. For details, welcome under the cat!
This is our first event where we want to gather back-end developers for informal communication. The main goal is the exchange of knowledge in an informal setting. For our part, we promise four interesting speakers, two of whom want to introduce you right now.
The first is Anton Dubenchuk , the manager of the provider systems development group. Directions: microservice, Kubernetes, platform. We asked him a few questions.
Hello Anton. What is “provider integration systems”? Tell me more about your work? What are you and your team doing?
Yes, indeed, I am engaged in provider integration systems. QIWI has many different service providers with whom we need to integrate using different protocols. Either partners provide us with an interface that we implement on our side, or vice versa - we give them the necessary protocols and partners implement them on their side.
There are two types of integration: there are standard ones that simply consist in setting up access to the partner, and there are more complex ones for which the protocol needs to be implemented: what we call a gateway.
The purpose of all this work is to finally get the payment to the recipient.
In addition to connecting providers, our group is committed to implementing other, more complex, custom integrations with partners.
I heard that an important focus of your work is to build a microservice application architecture. Why is this approach preferred?
The decision to switch to microservice architecture was made not long ago - in early 2015. Up to this point, we had large systems and services that were strongly connected with each other. It was difficult to deal with them, to modify something, to release new functionality. After making changes, we and our colleagues had to be quite difficult to rebuild. And this is one of the reasons why we switched to microservices.
The second reason is that we needed to develop new products as quickly as possible, test ideas and convey them to users. The only solution we have seen is the transition to microservice architecture.
What technology stack do you and your team use? Particularly interesting to hear about how you apply Kubernetes.
If we talk about Kubernetes , then we now use it for testing. We recently introduced it in order to try and see how comfortable it would be to work with it. Later, with the experience gained in the test environment, we will implement it in the production environment.
As for applications, they are written mainly in Java. Databases - Oracle, PostgreSQL. Of course, we use Spring to build the insides of our applications.
We have also been developing an application server for quite some time, within which applications written in JavaScript can run. This is where we integrate with our partners.
Some of the approaches and practices that we have developed during our work, but which are not available in commonly used libraries, we took out in the form of our own libraries, which we can easily connect to various projects as needed.
Do you have a specific approach to development? Methodology?
From the point of view of project activities, we are closer to flexible development approaches. Initially, the project is discussed in conjunction with customers, lawyers, accounting, that is, with all the parties involved in the project. Developed preliminary decisions, ideas about the final version of the project. After this, the implementation process takes place.
In the process of implementation, we use iterations that take two weeks. We meet, discuss who did what, good luck and failure, plan what we will do in the next iteration.
If we talk from the development point about the tools and approaches that we use, then, of course, a version control system, code-review system, CI / CD, etc. are used.
Anton, what will be your report at the QIWI Server Party?
I plan to talk about our experience in implementing Kubernetes. I want to talk about why we decided to try it, what, in my opinion, the prospects and why the previous decision did not suit us.
Kubernetes is an open-source project. How actively is it developing, is there a large community? And what are his prospects?
In my opinion, the development he is experiencing sufficiently rapid. I observe how Kubernetes is actively and harmoniously developing - special groups have been formed in different areas of development, regular discussions are held, and pull requests are made. People gathered from different companies: Google, Microsoft, Red Hat, CoreOS, Huawei. At the same time, in the process of discussion, they are trying to develop solutions that on the one hand would be quite pragmatic, but on the other, they did not drive people and companies into too rigid limits. In the end, it turns out quite flexible system. Plus, given that initially the ideas that were introduced into Kubernetes, were borrowed from the practice of deploying applications to Google, they believe that the result will be a solution that will be able to occupy its niche.
The second of our speakers - Sergey Chekansky - project manager for machine learning and bigdate.
Sergey, tell us more about your work at QIWI? What is in the range of your tasks?
I do machine learning and big data analytics. The company has a huge number of data sources, combining which you can gain additional benefits. We merge, validate, structure data and build descriptive analytics on it or use machine learning models.
What technology stack do you and your team use?
For processing large data arrays we use Apache Spark and write mostly in Python, but separate modules in Scala. The tasks of loading, storing and retrieving information are solved by combinations of Apache Scoop, Kafka and own solutions. Storage rarely updated data is implemented in Hadoop, mainly in the format of parquet. Data that is frequently updated is written to Apache Kudu.
Do you have a specific approach to working with BigData? Methodology?
We do not have a methodology, but rather principles. Since the main tasks of an analytical nature, our principles are to implement the task so that the results can be repeated. If this is the result of some kind of analysis, then we are not doing one-time scripts, but something that can be repeated many times.
Sergey, I know that your report at the conference will be titled "Why Big Data is Useless." Tell me what are you going to talk about?
I will talk about how much potential we do not use in BigData technology, and why this happens.
September 6th is the best day of the calendar.
If the back-end is your theme, wait! Admission is free, but you need to register so that we do not miss the number of seats and the traditional “admin” treats. By the same link you can find all the details.
Come, we will be glad to see you!
PS For those who do not have time to register or can not attend, a link to the broadcast , which will be available on September 6 from 18.00
Source: https://habr.com/ru/post/337122/
All Articles