How we were friends Neo4j and Meteor

Writing driver support graph database Neo4j for Meteor

In Meteor, any work with data is related to two-way reactivity. Currently, MongoDB and Redis built into Meteor (both drivers are developed in Meteor walls) have 100% reactivity, and some reactivity is implemented for MySQL and MSSQL (by third-party developers).

For the above databases, reactivity is implemented by observers who report where, how, when and what data has changed, so that the driver servicing the communication [data <-> presentation] knows what data and which Clients to update. Neo4j is devoid of any watchers and observers, but this did not stop us. How we came out of this situation and why we need Neo4j read under the cut.

Under node.js there is an official and eponymous npm-package from manufacturers Neo4j, which is replete with functionality, but in the first two lines of its documentation it is gently hinted that only the connection to the GraphDatabase and the query method works stably.

Why do we need Neo4j

I believe that each task should have its own tool, created and designed to perform this task at the highest level. Everyone knows that one nail can be, and sometimes it is necessary (if the time spent searching for or buying funds for a hammer is unjustifiably inflated) to score with a sneaker. But for 100 and even more than 100,000 nails, it would be advisable to get a hammer, and better nails. In our case, we need to store and retrieve relationship data between records. We continue to store the data in a denormalized form in MongoDB, but we store the relationships of this data in Neo4j.
')

How it all began: Connector

Initially it was assumed that by creating a global variable that holds an object of the type GraphDatabase , the functionality supplied in the npm-package will be enough for the tasks: at that time we wrote / read data to / from the database (without reactivity). This is how neo4jdriver was born - a package containing the globally available class Neo4j , which, when initialized, creates a connection to a database running locally or remotely. During initialization, you can pass a single url parameter.

Later there was a need for:

• reactivity;
• observer;
• isomorphism - the ability to perform requests from both the server and the client;
• to receive incoming data - by default Neo4j on any MATCH request throws out a bunch of garbage that weighs a lot.

This is where the fun began.

How it went: Reactivity

The second was the neo4jreactivity package, based on the principle of pseudo-reactivity, implemented through a layer in the form of MongoDB. Simply put, on any request in Neo4j, we return - Mongo \ Cursor , which in turn is the source of reactive data or, as it is commonly called in the Meteor community: REACTIVE DATA SOURCE.

Initially, everything seemed simple:

• We make some kind of caching collection in MongoDB, which contains the request, the request hash and the response from Neo4j;
• For the Client, we create a session in which we hold an array containing all the hashes of the requests to which you want to subscribe;
• All requests are passed through the Meteor.neo4j.query method, which creates a hash from a query to the database, receives a response from the database, writes it to the database and sends it to all Clients subscribed to this request hash;
• To launch requests from the Client, we do the Meteor-method, which eats! any request and executes it on the server.

At the time of release of one of the first driver versions, you could run absolutely any request on the Client, i.e. could change, retrieve or erase all data stored in Neo4j. This problem was solved by introducing the Meteor.neo4j.methods ({}) and Meteor.neo4j.call (methodName, opts, callback) methods , which work according to the standard Meteor.methods ({}) principle, for example:

if(Meteor.isServer){ Meteor.neo4j.methods({ 'GetUser': function(){ return 'MATCH (n:User {_id: {userId}}) RETURN n' } }); }; if(Meteor.isClient){ Meteor.neo4j.call('GetUser', {userId: 123}, function(error, data){ if(!error){ Session.set('theUser', data); } }); } 

The second thing we did was property Meteor.neo4j.allowClientQuery , which is true and false , and is false by default. This will allow developers at the time of developing and testing the application to work in the browser console, send data and verify the received data.

If for some reason you decide to leave the possibility of executing requests to Neo4j from the Client, then the following functionality is provided to limit the type of requests to Neo4j. You have two methods available: neo4j.set.allow and neo4j.set.deny . Both methods take a single parameter - an array of strings (array of strings). In addition, arrays are available to you: Meteor.neo4j.rules.allow , Meteor.neo4j.rules.deny and neo4j.rules.write , which contain the current rules, and the latter contains an array with write statements, which allows you to do this shortcut:

 if(Meteor.isClient){ Meteor.neo4j.set.deny(Meteor.neo4j.rules.write); } 

And prohibit all requests for recording by the Client. All methods described in the paragraph above are isomorphic. The client’s hack will not work, as the data is additionally checked for integrity on the Server side.

Watching the data: your Observer with blackjack and listener

It was later discovered that data change requests did not initiate data updates on Clients. Reactivity simply did not work until one of the Clients turned to the changed data and initiated an update in MongoDB, and as a result, on all Clients. This happened due to the fact that we did not have an observer who would monitor the changed data and initiate the launch of all queries related to the data that had changed.

Listener

We return to our ideal package called neo4jdriver , erase the entire project and write again:

• We leave the class structure and initialization of the class instance with the ability to transfer the url to the database;
• Create an array GraphDatabase.callbacks , storing callbacks that take two parameters - query and opts ;
• Add the GraphDatabase.listen (func) method, which accepts a function with two parameters, query and opts , all functions fall into the GraphDatabase.callbacks array;
• We reassign the query method built into the npm-package by adding to it the launch of all callbacks from the GraphDatabase.callbacks array.

Reactivity Observer:

First of all, we need to learn how to separate the request data from the query design; for this, the sensitivities parameter was entered. This parameter contains data that can be changed. Now the entry in the Neo4jCacheCollection collection looks like this:

 uid // Unique hashed ID of the query data // Parsed data returned from Graph query // Original Cypher query string sensitivities // Sensitive data, which contains a map of parameters, and hardcoded data into query opts // Original map of parameters for the Cypher query type // Type of query ('READ'|'WRITE') created // Creation time 

We connect observer and listener:

• Put a wiretap on all requests to Neo4j;
• We get the sensitivities of the current request;
• We find all requests for reading, which include sensitivities from the current request;
• Run the resampling for the resulting matches;
• Further necessary reactivity will provide us a layer in the form of MongoDB.

We managed to provide data updates when they change - on all Clients, in a very simple way.

We get only the data we need

The third problem was the data that came from Neo4j. In addition to the fields we requested, we also get a bunch of empty objects that are returned to us by the npm package. Empty objects weigh a lot and do not contain information, we have nothing to store them. For the separation of useful and requested data, the parseReturn method was written, which parsed the query in the database (Cypher query) and understood what data was requested and what fields the developer wanted to receive. After that, for each requested information, an object was created containing an array of nodes with their data and metadata. If relations of nodes are requested, each node contains relations object containing data in the form of the following parameters:

• extensions
• start
• end
• self
• type

We deliver updates to customers

We learned to update the data in MongoDB and monitor their changes in Neo4j, but the embedded objects in the returned data will not be updated by themselves. The functionality offered by the reactive-var package came to our rescue. For this, data on the Client upon receipt from the Neo4jCache collection are assigned and returned via ReactiveVar . On the Server, upon receipt from the collection, Neo4jCache will be returned from promise. On the server and the Client, it is enough to call the get () method to get the data reactively. For those who need to get Mongo \ Cursor there is a property cursor .

Example:

 /*   (  ) */ allUsers = Meteor.neo4j.query('MATCH (users:User) RETURN users'); /*     */ var users = allUsers.get().users; /*  Mongo\Cursor  */ var usersCursor = allUsers.cursor; /*   (  )  callback*/ var allUsers; Meteor.neo4j.query('MATCH (users:User) RETURN users', null, function(error, data){ allUsers = data.user; }); 

At this stage, we created a test application and published it on GitHub. A week later, the developer community helped us “finish” the driver and fix minor bugs. I will be glad to questions and suggestions for improvement and further development of the project. Thanks for attention.

References:

• NPM package: node-neo4j
• Neo4j Meteor Driver: neo4jdriver
• Neo4j Meteor Reactivity layer: neo4jreactivity
• An example of using Neo4j in Meteor: Neo4j based Leaderboard Meteor app

PS At the moment, the company Neo4j is actively involved in the development of the project and recognized this driver for Meteor as official.