“There is no such problem in Tarantula as a strong degradation with time and under load” - Vasily Soshnikov
Today we are talking with Vasily Soshnikov , architect and developer of the domestic open source DBMS Tarantool .
Due to its ability to provide high bandwidth on OLTP-loads , Tarantula enjoys extraordinary popularity and demand among high-tech companies. Today, Tarantula is used by the largest domestic Web services, telecommunications operators and many other representatives of big business.
At PG Day'17 Russia, Vasily will conduct an intensive practical course on building a scalable distributed architecture for OLTP tasks using only two tools: Tarantool and nginx . All participants of the course will listen to a brief educational program on the architecture and internal structure of the Tarantula, after which they will plunge into a 3-hour practice , during which a fault-tolerant and scalable service will be built from scratch.
')
As part of today's conversation, Vasily told about the development of the Tarantula and its role in this process, shared examples of tasks that are solved with the help of Tarantool, spoke about the structure of the roadmap and the principles of the appearance of new “features” in it.
PG Day: Vasily, tell me briefly about yourself: who are you, what are you doing, what problems do you solve?
Vasily : I am a senior developer / architect. My responsibilities include design and development. I also sometimes intersect with the business, managerial and organizational components. It is difficult to characterize my work in one word. Now I am developing and promoting an open source project Tarantool. Also sometimes I help other third-party projects, for example Centrifugo (this is a message broker for the web).
Speaking of my passion is video streaming. I dealt with them once in my life, it was in the RBC company, I created the whole backend for RBC-TV. It was probably my most interesting project in terms of architecture and code.
PG Day: What exactly are you doing inside the Tarantool project?
Basil : Basically, planning and architectural solutions based on Tarantool itself, developing core parts and external modules. For example, from those developed by me: Tarantool Curl is the binding of libcurl in Tarantool; Tarantool MQTT - binding for MQTT; and the module for nginx is a reverse proxy module that converts the HTTP request flow into the Tarantula protocol and vice versa. As for the core, I recently took part in the transfer of an external project to the core. It was Tarantool Curl, HTTP is necessary for everyone in the database.
PG Day: How many people are employed in developing the Tarantula? Is there such a thing as a core team?
Vasily : Yes, of course. In fact, it is not only developers. This is also technical writers, a bit of marketing and some others. In total we are 21 people. These are those who sit in Moscow, in the office of Mail.ru, and develop Tarantula. People from our community make a big contribution. For example, they add new connectors for different languages, invent features and even implement them!
PG Day: Why is the Tarantula cool as a database? What makes it different from other products?
Vasily : The answer is simple - OLTP. In Tarantula there is no such problem as severe degradation with time and under load. Any disk database, and even some in-memory, is severely degraded. For example, the same Redis , when it is heavily loaded and at the same time makes a snapshot - we get x2 to the size of the memory and brakes, and this, I note, also occurs under load. Tarantool doesn’t have these drawbacks, plus it’s transactional.
Today there are few databases that actually work with predictable latency (response time) and predictable degradation. Every day I see how the Tarantula works in “prod”, and I know that it will work even in a year, even if the profile of the load changes.
PG Day: How is such predictability and stability achieved at high transaction loads achieved?
Vasily : In the internal architecture . In general, this will be my master class. Therefore, I do not want to tell now. If I tell now, it will not be very interesting to listeners. Who is interested, he will go on the road and will fall into several articles and reports by Konstantin Osipov. And it is better to come to the master class! I will draw a beautiful scheme, I will have a rationale on the slides why this works so fast and stably. So be patient a little more.
PG Day: Suddenly, but well, such an answer is accepted! As I understand it, do you plan to combine theory with practice in a master class?
Vasily : I have several sections planned. The first section is the shortest, without practice. It is about how the Tarantula is built from the inside , what it is and how to use it. We will also discuss the problems that exist in it; it is very important - during operation to be aware of all the problems! The second part - it will be a very light practice . We will launch Tarantool, see how it works, what the concept of the Application server is . The third part is the most difficult, we will create a full-fledged service, we will scale it up and, as the cherry on the cake, we will make it all fault tolerant. I summarize. Theories will be about 50 minutes, then only practice.
PG Day: Do participants need knowledge of some programming languages or something else? Any level of preparation to participate in the practical part?
Vasily : No, it will not be required, only laptops. The language of Lua is primitive, I will explain it to everyone. Configuration files can be taken from my github, I will give a link.
PG Day: It turns out a kind of plug-n-play: how to take the minimum number of tools and immediately start creating a working architecture.
Vasily : That's right. We will create a project that satisfies two requirements: predictable scaling and fault tolerance . Two technologies will be used: the Tarantula and the nginx (the reverse proxy module I spoke about).
PG Day: What features do you want to add to the Tarantula? Do something cool now that will appear in it soon?
Vasily : First I want to say that we have a completely open roadmap , which can be influenced by the community. Probably, SQL is one of the biggest features that is being developed right now, an alpha version is already available. Our plan: catch up with it to stable, and then start implementing distributed SQL and a full query planner.
PG Day: Does SQL mean standard SQL or some kind of?
Basil : We took the SQLite3 parser. SQL standard, if memory serves me, 2003. But, one must understand that there are some non-standard things in it.
PG Day: Are there any plans to make the Tarantula SQL built-in compatible with popular DBMSs?
Vasily : We had such thoughts, but, to be honest, we still do not see much value in this. Still, I want to keep some originality. If this is done, then for a specific customer, who will ask us to do it. But within the framework of the development of a common project, it is unlikely. Even in the roadmap you will not find it.
PG Day: Tarantula - is it something that should be taken at the start, when the launch of the product starts, or are there any examples of application in the already established infrastructure?
Vasily : For new projects, definitely yes. For old Tarantool can solve some problems. Let me remind you, OLTP is an ideal case for Tarantool, and this is or will be in all projects.
There was a great case recently: Tarantula was used as a read-only replica in the MySQL cluster , all of which made it possible to greatly unload the MySQL cluster and increase the response time. Created this company Mamba (Wamba).
Speaking of the community, colleagues from Mamba themselves developed replication mechanisms between these bases and laid them out in open source for general use. We learned about this to everyone when a person in our community chat started actively asking about replication.
We also have one client who replicated through Oracle GoldenGate in Tarantool . The scheme is the same: the entire OLTP load is transferred to Tarantool. This allowed them to save a lot of money and increase response time.
PG Day: If a company wants to bring your expertise to adapt Tarantool to their realities, what should be done?
Vasily : On a contract basis.
PG Day: There are examples of using the Tarantula for other purposes? Any recommendations on this?
Vasily : Yes. There are people who are trying to make very heavy analytics using one copy of Tarantool! Without even trying to build a multi-layered architecture that scales easily and flexibly. This is the most frequent anti-pattern that I have met.
A typical story looks like this: first, they make small requests, get high speed and rejoice; then they start to do very, very heavy queries, Tarantool works worse and this is quite logical, for OLAP you need a completely different architecture, and it must be thought out at the design stage. Tarantool may be an OLAP base, but this is not the best case for it.
PG Day: Do you have an idea to make Tarantula a product suitable for stable and fast OLAP? Or is it still contrary to his ideology?
Vasily : Thoughts are there, but this is not yet in the roadmap. We try to do what we are strong at. Namely - to process a large number of small transactions , it is quite understandable and large market. It is clear where to go. Starting a big development on OLAP without a specific customer is not the best way.
PG Day: Is the current roadmap formed from current requirements and real precedents?
Vasily : That's right. We have several different types of roadmap tasks. There are very small (roughly speaking, add some option) and rather large (some big problem to eliminate). There is another type - new development . All this is made up of the requirements of our customers, users and our own vision of the product.
PG Day: And at what point do you understand that there is some kind of feature that should get into the roadmap of the product?
Vasily : This is a difficult question. It happens quite differently. I can retell my recent experience.
There was such a task: to add the ability to return HTTP statuses and headers from Tarantool to the nginx module. The idea hung about a year, many voted for this idea, but my hands did not reach implementation. We got there when two customers appeared who say: “Guys, we have SOAP and REST everywhere, and we really need status codes”. Naturally, the feature immediately appeared in open source.
PG Day: We have already talked a little about the content of the master class. Who will be useful in the first place?
Vasily : I will speak about architecture , about programming - about what I am strong at. Therefore, first of all, architects, developers. In general, I invite everyone who wants to learn how to create new projects on the basis of the Tarantula or to apply them to existing ones.
PG Day: Thank you, Vasily! I want to thank you for the time you have given us. We look forward to our meeting in St. Petersburg.
Vasily : Mutually, Roman! Really looking forward to. For me it will be an interesting experience.
Due to its ability to provide high bandwidth on OLTP-loads , Tarantula enjoys extraordinary popularity and demand among high-tech companies. Today, Tarantula is used by the largest domestic Web services, telecommunications operators and many other representatives of big business.
At PG Day'17 Russia, Vasily will conduct an intensive practical course on building a scalable distributed architecture for OLTP tasks using only two tools: Tarantool and nginx . All participants of the course will listen to a brief educational program on the architecture and internal structure of the Tarantula, after which they will plunge into a 3-hour practice , during which a fault-tolerant and scalable service will be built from scratch.
')
As part of today's conversation, Vasily told about the development of the Tarantula and its role in this process, shared examples of tasks that are solved with the help of Tarantool, spoke about the structure of the roadmap and the principles of the appearance of new “features” in it.
PG Day: Vasily, tell me briefly about yourself: who are you, what are you doing, what problems do you solve?
Vasily : I am a senior developer / architect. My responsibilities include design and development. I also sometimes intersect with the business, managerial and organizational components. It is difficult to characterize my work in one word. Now I am developing and promoting an open source project Tarantool. Also sometimes I help other third-party projects, for example Centrifugo (this is a message broker for the web).
Speaking of my passion is video streaming. I dealt with them once in my life, it was in the RBC company, I created the whole backend for RBC-TV. It was probably my most interesting project in terms of architecture and code.
PG Day: What exactly are you doing inside the Tarantool project?
Basil : Basically, planning and architectural solutions based on Tarantool itself, developing core parts and external modules. For example, from those developed by me: Tarantool Curl is the binding of libcurl in Tarantool; Tarantool MQTT - binding for MQTT; and the module for nginx is a reverse proxy module that converts the HTTP request flow into the Tarantula protocol and vice versa. As for the core, I recently took part in the transfer of an external project to the core. It was Tarantool Curl, HTTP is necessary for everyone in the database.
PG Day: How many people are employed in developing the Tarantula? Is there such a thing as a core team?
Vasily : Yes, of course. In fact, it is not only developers. This is also technical writers, a bit of marketing and some others. In total we are 21 people. These are those who sit in Moscow, in the office of Mail.ru, and develop Tarantula. People from our community make a big contribution. For example, they add new connectors for different languages, invent features and even implement them!
PG Day: Why is the Tarantula cool as a database? What makes it different from other products?
Vasily : The answer is simple - OLTP. In Tarantula there is no such problem as severe degradation with time and under load. Any disk database, and even some in-memory, is severely degraded. For example, the same Redis , when it is heavily loaded and at the same time makes a snapshot - we get x2 to the size of the memory and brakes, and this, I note, also occurs under load. Tarantool doesn’t have these drawbacks, plus it’s transactional.
Today there are few databases that actually work with predictable latency (response time) and predictable degradation. Every day I see how the Tarantula works in “prod”, and I know that it will work even in a year, even if the profile of the load changes.
PG Day: How is such predictability and stability achieved at high transaction loads achieved?
Vasily : In the internal architecture . In general, this will be my master class. Therefore, I do not want to tell now. If I tell now, it will not be very interesting to listeners. Who is interested, he will go on the road and will fall into several articles and reports by Konstantin Osipov. And it is better to come to the master class! I will draw a beautiful scheme, I will have a rationale on the slides why this works so fast and stably. So be patient a little more.
PG Day: Suddenly, but well, such an answer is accepted! As I understand it, do you plan to combine theory with practice in a master class?
Vasily : I have several sections planned. The first section is the shortest, without practice. It is about how the Tarantula is built from the inside , what it is and how to use it. We will also discuss the problems that exist in it; it is very important - during operation to be aware of all the problems! The second part - it will be a very light practice . We will launch Tarantool, see how it works, what the concept of the Application server is . The third part is the most difficult, we will create a full-fledged service, we will scale it up and, as the cherry on the cake, we will make it all fault tolerant. I summarize. Theories will be about 50 minutes, then only practice.
PG Day: Do participants need knowledge of some programming languages or something else? Any level of preparation to participate in the practical part?
Vasily : No, it will not be required, only laptops. The language of Lua is primitive, I will explain it to everyone. Configuration files can be taken from my github, I will give a link.
PG Day: It turns out a kind of plug-n-play: how to take the minimum number of tools and immediately start creating a working architecture.
Vasily : That's right. We will create a project that satisfies two requirements: predictable scaling and fault tolerance . Two technologies will be used: the Tarantula and the nginx (the reverse proxy module I spoke about).
PG Day: What features do you want to add to the Tarantula? Do something cool now that will appear in it soon?
Vasily : First I want to say that we have a completely open roadmap , which can be influenced by the community. Probably, SQL is one of the biggest features that is being developed right now, an alpha version is already available. Our plan: catch up with it to stable, and then start implementing distributed SQL and a full query planner.
PG Day: Does SQL mean standard SQL or some kind of?
Basil : We took the SQLite3 parser. SQL standard, if memory serves me, 2003. But, one must understand that there are some non-standard things in it.
PG Day: Are there any plans to make the Tarantula SQL built-in compatible with popular DBMSs?
Vasily : We had such thoughts, but, to be honest, we still do not see much value in this. Still, I want to keep some originality. If this is done, then for a specific customer, who will ask us to do it. But within the framework of the development of a common project, it is unlikely. Even in the roadmap you will not find it.
PG Day: Tarantula - is it something that should be taken at the start, when the launch of the product starts, or are there any examples of application in the already established infrastructure?
Vasily : For new projects, definitely yes. For old Tarantool can solve some problems. Let me remind you, OLTP is an ideal case for Tarantool, and this is or will be in all projects.
There was a great case recently: Tarantula was used as a read-only replica in the MySQL cluster , all of which made it possible to greatly unload the MySQL cluster and increase the response time. Created this company Mamba (Wamba).
Speaking of the community, colleagues from Mamba themselves developed replication mechanisms between these bases and laid them out in open source for general use. We learned about this to everyone when a person in our community chat started actively asking about replication.
We also have one client who replicated through Oracle GoldenGate in Tarantool . The scheme is the same: the entire OLTP load is transferred to Tarantool. This allowed them to save a lot of money and increase response time.
PG Day: If a company wants to bring your expertise to adapt Tarantool to their realities, what should be done?
Vasily : On a contract basis.
PG Day: There are examples of using the Tarantula for other purposes? Any recommendations on this?
Vasily : Yes. There are people who are trying to make very heavy analytics using one copy of Tarantool! Without even trying to build a multi-layered architecture that scales easily and flexibly. This is the most frequent anti-pattern that I have met.
A typical story looks like this: first, they make small requests, get high speed and rejoice; then they start to do very, very heavy queries, Tarantool works worse and this is quite logical, for OLAP you need a completely different architecture, and it must be thought out at the design stage. Tarantool may be an OLAP base, but this is not the best case for it.
PG Day: Do you have an idea to make Tarantula a product suitable for stable and fast OLAP? Or is it still contrary to his ideology?
Vasily : Thoughts are there, but this is not yet in the roadmap. We try to do what we are strong at. Namely - to process a large number of small transactions , it is quite understandable and large market. It is clear where to go. Starting a big development on OLAP without a specific customer is not the best way.
PG Day: Is the current roadmap formed from current requirements and real precedents?
Vasily : That's right. We have several different types of roadmap tasks. There are very small (roughly speaking, add some option) and rather large (some big problem to eliminate). There is another type - new development . All this is made up of the requirements of our customers, users and our own vision of the product.
PG Day: And at what point do you understand that there is some kind of feature that should get into the roadmap of the product?
Vasily : This is a difficult question. It happens quite differently. I can retell my recent experience.
There was such a task: to add the ability to return HTTP statuses and headers from Tarantool to the nginx module. The idea hung about a year, many voted for this idea, but my hands did not reach implementation. We got there when two customers appeared who say: “Guys, we have SOAP and REST everywhere, and we really need status codes”. Naturally, the feature immediately appeared in open source.
PG Day: We have already talked a little about the content of the master class. Who will be useful in the first place?
Vasily : I will speak about architecture , about programming - about what I am strong at. Therefore, first of all, architects, developers. In general, I invite everyone who wants to learn how to create new projects on the basis of the Tarantula or to apply them to existing ones.
PG Day: Thank you, Vasily! I want to thank you for the time you have given us. We look forward to our meeting in St. Petersburg.
Vasily : Mutually, Roman! Really looking forward to. For me it will be an interesting experience.
Source: https://habr.com/ru/post/331646/
All Articles