NSNJSON. 道 (Final article)
道 - the way. In this final article on the NSNJSON format , I want to tell you about my path that led me to the invention of this format.
In the comments to my past articles ( “Complicated Simplified JSON” and “JSON for Fans of Brackets” ), questions about the meaning, complexity, convenience, and applicability of this format were repeatedly expressed. So, I hasten to congratulate all those who are not indifferent - you have waited!
Introduction
Task number 1. Presentation of documents on hierarchical data structures
Task number 2. Driver implementation
About NSNJSON format
Conclusion
It all started with my acquaintance with one interesting NoSQL DBMS, namely, InterSystems Caché . The company InterSystems maintains a blog on Habrahabr , in which you can read about the internal structure of this database. Let me just say that ESA uses InterSystems Caché in the GAIA project (thanks for the tsafin and morrison amendment).
')
In order to understand the further essence, it is necessary to know that the data in Caché is stored in globals (you can think of the global as a hierarchical data structure). You can read more about globals in the following articles:
As part of my master program, a project was launched to implement document-oriented NoSQL based on Caché. The theoretical research base of this project is the conceptual question of the speed, performance and reliability of such an implementation, compared with the existing solution. My favorite MongoDB was chosen as the reference document-oriented NoSQL.
So, what did I have in my beginning?
1. Document-oriented NoSQL MongoDB.
2. Hierarchical NoSQL Caché.
It would seem, so JSON documents can be called hierarchical, and therefore there is no difficulty in this task. However, thedevil is in the details , as it turned out, globals can store either a binary value, or a string or numeric value. More lists are supported. In other words, a leaf of a tree (the value of a global) can contain either a number or a string, or binary data or a list. I was at the very beginning of my journey and therefore refused to store JSON in binary form, preferring strings and numbers. In turn, such a solution required to figure out how to save JSON data having only strings and numbers available.
Recall that the JSON format defines 6 types: null , number , string , true , false , array , object . Thus, it was required to come up with an unambiguous data representation scheme for each JSON type, the name is available only to build hierarchies and use, as values for tree leaves, a string and a number. However, another requirement was put forward to the JSON data presentation scheme - unambiguity. This condition is the guarantor of unambiguous JSON data recovery from globals.
I will a little explain this moment.
Consider the JSON type null . One could suggest a simple scheme for it. Suppose if there are values of the JSON type null , then when saving this value to the global, you need to create a sheet and set the value to the empty string "". However, the very first counterexample comes very quickly - the scheme loses its uniqueness at the moment when we need to keep the value of the JSON string type equal to the empty string. In this regard, I decided to switch to another, rather simple scheme.
We can assume that this scheme and became the progenitor of the format NSNJSON.
Now that the JSON data presentation scheme is ready, the next step in my way was waiting for me. I needed to develop a driver for this document-oriented NoSQL DBMS.
The driver implementation consists of two stages:
To transfer data between my driver and the Caché driver, I chose a simple format - string. My driver received JSON, converted it to a string, and passed it to the Caché driver, who in turn passed this string to the stored code. The stored code parsed this line, and then applied the rules for JSON data representation on globals.
However, I was in for a surprise!
During development and debugging, I found out some interesting facts about the JSON parser I used in the stored Caché code:
Thus, I needed to solve the following problems:
The solution to these problems was the NSNJSON format that I developed .
So, for null values, the following view was suggested:
For true values, the following view was suggested:
For false values, the following view was suggested:
The problem with _ was solved by entering the field “n” .
So, for the _id field with a value of 213, the view would be:
Thus, this format solved all previously mentioned problems.
I decided to separate the format into a separate project and call NSNJSON ( N ot S o N ormal JSON ). And then I decided to share this format with the respected Habrakhabr community in my article “Complicated Simplified JSON” , and also, in my opinion, an interesting modification of this format, in which JSON data is presented using numbers, strings and arrays, in the article JSON for lovers of brace .
The NSNJSON project is published on GitHub .
Two drivers are implemented for it:
That concludes the final article on NSNJSON. I talked about the difficulties I encountered, and how I overcame them.
Lastly, I want to note once again that this was my (way), and I went through it this way. At each step, it would be possible to go differently, choosing another solution to the problems that arose, but this would not be my way ...
In the comments to my past articles ( “Complicated Simplified JSON” and “JSON for Fans of Brackets” ), questions about the meaning, complexity, convenience, and applicability of this format were repeatedly expressed. So, I hasten to congratulate all those who are not indifferent - you have waited!
Content
Introduction
Task number 1. Presentation of documents on hierarchical data structures
Task number 2. Driver implementation
About NSNJSON format
Conclusion
↑ Introduction
It all started with my acquaintance with one interesting NoSQL DBMS, namely, InterSystems Caché . The company InterSystems maintains a blog on Habrahabr , in which you can read about the internal structure of this database. Let me just say that ESA uses InterSystems Caché in the GAIA project (thanks for the tsafin and morrison amendment).
')
In order to understand the further essence, it is necessary to know that the data in Caché is stored in globals (you can think of the global as a hierarchical data structure). You can read more about globals in the following articles:
As part of my master program, a project was launched to implement document-oriented NoSQL based on Caché. The theoretical research base of this project is the conceptual question of the speed, performance and reliability of such an implementation, compared with the existing solution. My favorite MongoDB was chosen as the reference document-oriented NoSQL.
So, what did I have in my beginning?
1. Document-oriented NoSQL MongoDB.
2. Hierarchical NoSQL Caché.
↑ Task number 1. Presentation of documents on hierarchical data structures
It would seem, so JSON documents can be called hierarchical, and therefore there is no difficulty in this task. However, the
Recall that the JSON format defines 6 types: null , number , string , true , false , array , object . Thus, it was required to come up with an unambiguous data representation scheme for each JSON type, the name is available only to build hierarchies and use, as values for tree leaves, a string and a number. However, another requirement was put forward to the JSON data presentation scheme - unambiguity. This condition is the guarantor of unambiguous JSON data recovery from globals.
I will a little explain this moment.
Consider the JSON type null . One could suggest a simple scheme for it. Suppose if there are values of the JSON type null , then when saving this value to the global, you need to create a sheet and set the value to the empty string "". However, the very first counterexample comes very quickly - the scheme loses its uniqueness at the moment when we need to keep the value of the JSON string type equal to the empty string. In this regard, I decided to switch to another, rather simple scheme.
schematic description
Json null
JSON number ( value )
For example, for value
JSON string ( value )
For example, for value
Json true
Json false
Json array
For example, for an array
JSON object
For example, for an object
the presentation would be as follows:
JSON number ( value )
For example, for value
2015 JSON string ( value )
For example, for value
"R&D" the presentation would be as follows:Json true
Json false
Json array
For example, for an array
[ 2015, "R&D" ] the presentation would be as follows:JSON object
For example, for an object
{ "year": 2015, "section": "R&D" } the presentation would be as follows:
We can assume that this scheme and became the progenitor of the format NSNJSON.
Now that the JSON data presentation scheme is ready, the next step in my way was waiting for me. I needed to develop a driver for this document-oriented NoSQL DBMS.
↑ Task number 2. Driver implementation
The driver implementation consists of two stages:
- stored code development (in Caché, in Caché ObjectScript),
- developing a code that will access the Caché driver and transmit data to the stored code.
To transfer data between my driver and the Caché driver, I chose a simple format - string. My driver received JSON, converted it to a string, and passed it to the Caché driver, who in turn passed this string to the stored code. The stored code parsed this line, and then applied the rules for JSON data representation on globals.
However, I was in for a surprise!
During development and debugging, I found out some interesting facts about the JSON parser I used in the stored Caché code:
- JSON type null is translated to "" ,
- JSON type true is translated to 1 ,
- JSON type false is translated to 0 .
- fields starting with _ are ignored.
Thus, I needed to solve the following problems:
- storing type information
- saving fields starting with _ .
The solution to these problems was the NSNJSON format that I developed .
So, for null values, the following view was suggested:
{ "t": "null" } For true values, the following view was suggested:
{ "t": "boolean", "v": 1 } For false values, the following view was suggested:
{ "t": "boolean", "v": 0 } The problem with _ was solved by entering the field “n” .
So, for the _id field with a value of 213, the view would be:
{ "n": "_id", "t": "number", "v": 213 } Thus, this format solved all previously mentioned problems.
↑ About NSNJSON format
I decided to separate the format into a separate project and call NSNJSON ( N ot S o N ormal JSON ). And then I decided to share this format with the respected Habrakhabr community in my article “Complicated Simplified JSON” , and also, in my opinion, an interesting modification of this format, in which JSON data is presented using numbers, strings and arrays, in the article JSON for lovers of brace .
The NSNJSON project is published on GitHub .
Two drivers are implemented for it:
↑ Conclusion
That concludes the final article on NSNJSON. I talked about the difficulties I encountered, and how I overcame them.
Lastly, I want to note once again that this was my (way), and I went through it this way. At each step, it would be possible to go differently, choosing another solution to the problems that arose, but this would not be my way ...
Source: https://habr.com/ru/post/270031/
All Articles