GridGain on Highload: where to talk about distributed DBMS, In-Memory and open source

Creating new features is fun. But how to test them, if your product is a distributed system, and the actual operation is a set of cores and disks on different physical servers, network interaction, hardware failures and unpredictable user actions?

Let us tell you how testing of the new Apache Ignite functionality happens, what we do well and what is not very:
- why lean on integration tests, and not really complain about unit, mock and XP?
- testing multi-threaded and distributed algorithms
- creating a test plan and code review
- where do we most often pierce?

At one time, GridGain donated the eponymous product of the Apache Software Foundation - now it is known as Apache Ignite. Since then, GridGain engineers have combined two roles - employees of a profit-oriented commercial company, and independent developers of an open project. And of course, between these roles, a relationship of love and hate. At the meeting we will talk about how the development in Open Source helps and hinders the solution of business problems.

We will start, and you join.
From the open source side: Dmitry Pavlov, Apache Ignite Community Manager
On the business side: Stanislav Lukyanov from GridGain Customer Success

You can endlessly add functionality and fix bugs in the product, but all this does not make sense if the new version does not get to the user. And for the release of a new version, it is far from always enough just to run the build and tests. And for the release version - not the same as running the assembly.

What if you need to support many versions of the product? How to make the transition from one version of the user to another without any surprises? And what have open source?

Let's talk about which path the changes take from the user's letter to customer support until they receive the coveted version with the correction.

No matter how developed the technology, backup in a difficult moment continues to keep us nerves, and sometimes work. The GridGain platform runs on top of the Apache Ignite open source distributed system, which lacks the ability to back up data. To date, the maximum amount of data in the GridGain client is 200 terabytes at 160 nodes. Data is not only stored, but constantly updated with the provision of transactional guarantees.

The inability to create backups of a distributed system on a similar scale was a stumbling block for the practical use of our platform by big business. From the report you will learn how we managed to eliminate this gap.

We had to learn:
- make data backup without stopping the user;
- make the data in the backup of the distributed system consistent and transactionally complete;
- make the procedures for creating and restoring backup resistant to topology changes using a distributed state machine;
- implement incremental backups that take up an order of magnitude less space;
- restore old data backups created on a significantly different cluster topology.