Embedded JavaScript database with an eye on API compatibility with MongoDB
Some time ago we decided to write one project for Node.js that really needed to work with the NoSQL database, but at the same time not have any dependencies on external applications. As is usually the case, it all ended with the development of a new library.
Starting development two years ago, the desire to use the embedded database for a web application seemed very strange. In fact, why? Now that the node-webkit project has appeared, this is much easier to explain. Using the built-in database it is possible to develop a dual-purpose web application. Such an application can work both in the classical client-server scheme, and using node-webkit as a regular downloadable application. An important feature in both cases is that the database code is part of your application, which eliminates many compatibility and installation problems.
Looking around, we found several different projects, but for some reason we stopped at Alfred.js . Outwardly, and according to the first tests, he looked very cool and did what we needed. First, he did not load all the data into memory, but used file access. That is, it could potentially work with a relatively large amount of data, intelligently consuming memory. Secondly, he knew how to use indexes, which is very important. And thirdly, it seemed that it was very well developed, if only because replication was even provided there.
Passing the equator in development, we began to stumble over Alfred.js. Realistic data sets caused multiple errors including data corruption. At some point, instead of developing the application, we started fixing the database code. Having spent a lot of time with a variable result, we did not achieve anything but disappointment. Most importantly, the problems with the database completely stopped the development of the application itself, and this was the most frustrating.
')
At some point in time, it was decided to replace the database with MongoDB and at the same time try to develop our own database library, which could be used without changing anything in the application code. That is, such a database should have the functionality and API identical to the MongoDB driver for Node.js. It was initially clear that it is difficult to implement all the functionality of MongoDB. However, we knew for sure that everything was unnecessary. According to the well-known rule, we need only 20% of the possibilities, which will cover 80% of the needs. Thus, further development went in two directions. Part of the team took up the adaptation of the application to MongoDB, the other part was developing the database.
Oddly enough, the code adaptation under MongoDB was not so trivial. The fact that Alfred.js uses similar query syntax and methods is more likely to hurt than help. This experience helped us to understand that only complete identity is the way to go. Therefore, it was initially decided that all tests should work one-on-one with our new database, data and MongoDB.
Later, having implemented the basic functionality, an idea appeared to take ready-made tests from the Node.js project of the driver for MongoDB. It was a serious challenge that we accepted and fulfilled. Integrating ready-made tests, we learned a lot of new things about the driver itself and somewhere even became disillusioned. Borrowed tests in most cases do not check error codes, which greatly hindered. Among other things, the API turned out to be too flexible in our opinion. For example, the Collection.find function allows 7 different combinations of parameters, and each time it is called, dozens of lines of code deal only with trying to guess which option was used. However, all of this behavior was reproduced in detail down to the text of exceptions and possible errors.
At some point, both directions of development were destined to meet. On the one hand, we already had a well-functioning application using MongoDB. On the other hand, we got a database that implements the functionality we need and passes several hundred tests. The meeting did not go perfectly, but the revealed problems were insignificant. Soon, the application worked with the built-in database as well as with MongoDB, which continues to do so far.
We named our database TingoDB (TInymoNGODB) and published the source code on GitHub .
Starting development two years ago, the desire to use the embedded database for a web application seemed very strange. In fact, why? Now that the node-webkit project has appeared, this is much easier to explain. Using the built-in database it is possible to develop a dual-purpose web application. Such an application can work both in the classical client-server scheme, and using node-webkit as a regular downloadable application. An important feature in both cases is that the database code is part of your application, which eliminates many compatibility and installation problems.
Looking around, we found several different projects, but for some reason we stopped at Alfred.js . Outwardly, and according to the first tests, he looked very cool and did what we needed. First, he did not load all the data into memory, but used file access. That is, it could potentially work with a relatively large amount of data, intelligently consuming memory. Secondly, he knew how to use indexes, which is very important. And thirdly, it seemed that it was very well developed, if only because replication was even provided there.
Passing the equator in development, we began to stumble over Alfred.js. Realistic data sets caused multiple errors including data corruption. At some point, instead of developing the application, we started fixing the database code. Having spent a lot of time with a variable result, we did not achieve anything but disappointment. Most importantly, the problems with the database completely stopped the development of the application itself, and this was the most frustrating.
')
At some point in time, it was decided to replace the database with MongoDB and at the same time try to develop our own database library, which could be used without changing anything in the application code. That is, such a database should have the functionality and API identical to the MongoDB driver for Node.js. It was initially clear that it is difficult to implement all the functionality of MongoDB. However, we knew for sure that everything was unnecessary. According to the well-known rule, we need only 20% of the possibilities, which will cover 80% of the needs. Thus, further development went in two directions. Part of the team took up the adaptation of the application to MongoDB, the other part was developing the database.
Oddly enough, the code adaptation under MongoDB was not so trivial. The fact that Alfred.js uses similar query syntax and methods is more likely to hurt than help. This experience helped us to understand that only complete identity is the way to go. Therefore, it was initially decided that all tests should work one-on-one with our new database, data and MongoDB.
Later, having implemented the basic functionality, an idea appeared to take ready-made tests from the Node.js project of the driver for MongoDB. It was a serious challenge that we accepted and fulfilled. Integrating ready-made tests, we learned a lot of new things about the driver itself and somewhere even became disillusioned. Borrowed tests in most cases do not check error codes, which greatly hindered. Among other things, the API turned out to be too flexible in our opinion. For example, the Collection.find function allows 7 different combinations of parameters, and each time it is called, dozens of lines of code deal only with trying to guess which option was used. However, all of this behavior was reproduced in detail down to the text of exceptions and possible errors.
At some point, both directions of development were destined to meet. On the one hand, we already had a well-functioning application using MongoDB. On the other hand, we got a database that implements the functionality we need and passes several hundred tests. The meeting did not go perfectly, but the revealed problems were insignificant. Soon, the application worked with the built-in database as well as with MongoDB, which continues to do so far.
We named our database TingoDB (TInymoNGODB) and published the source code on GitHub .
Source: https://habr.com/ru/post/263619/
All Articles