News DerbyJS 09.2014

This article describes the latest Derby-related events. I will also share my thoughts and useful links.

Lever

Lever - the company that created Derby, passed the next (second, second, third?) Round of investment and secured a stable existence for at least the next two years. With what we congratulate them.
Also, after Derby 0.6 was released, Lever makes some efforts to promote the framework:

• Every Thursday at 9 am PST (8 pm Moscow time) video conferences are held. Announcements and links to google hangout catch in google groups mailing list (called Office Hours). We asked to spend at such a time, because there are a lot of Russians. Usually conferences are led by Ian (Lever developer), you can often see Nate (CEO Lever, creator of Derby) and Joseph (creator of ShareJS). Stupid and smart questions are asked, concepts and plans are discussed, everyone shares their thoughts and experiences. Join now. Records of past video conferences with a brief description of the main topics can be found on the Lever blog .
• updated website design. Now there is a solid FAQ (including Russian) and links to useful resources. The only thing missing is (and this is the main problem at the moment) - documentation for 0.6. Workaround: Documentation 0.5 is relevant for Racer, and the zag2art series of articles (starting with this one ) will cover a significant portion of the documentation for Derby 0.6.
• participation in conferences: Joseph was on ForwardJS (unfortunately did not find the video). Initially, Nate had to go and talk about the subtleties and problems of Agnostic Rendering (a template engine that works on both the client and the server), but for some reason could not, I hope to hear this report. Ian was at a JS meeting in San Francisco.

Github

Derby moved to a new group . This is due to the fact that much rubbish has accumulated in the old one and the name Codeparty did not reflect the essence.
The active part of the community has created its own group , where modules are being sawn slowly.
Lever also spreads his work . You should also pay attention to the developments of the company Decision Mapper .
')

Community

The community is slowly but surely arriving. Often the path looks like this: client framework (AngularJS, Ember, etc.) -> Meteor -> Derby. Meteor helps to penetrate the concepts of realtime, full-stack, database on the client, isomorphism, and so on, but does not satisfy everyone due to various reasons. In general, here Nate very interestingly told about the main frameworks.
Recently created a chat on gitter .

Ideas

There are a few thoughts on what can be added or changed in Derby:

Use FoundationDB as a repository.

In the current implementation of livedb, document change events are caught by the Redis pub-sub (for one node of the process, it is possible without Redis). And this part of the functionality is built-in. To store the log of operations (oplog) and current versions of documents (snapshots) an external driver is used, which can be written, in general, for any database, for example, for Mongo .

Problems in the current implementation:
- the use of two DB, instead of one.
- the use of sequential execution of single-threaded Lua scripts as an alternative to transactions, leads to the fact that Redis potentially becomes a bottleneck (although so far no one has encountered this). Also, theoretically, with high collaboration (many clients served by different node processes change one document), Redis will need significantly more than one attempt to apply an operation to data, since the current version of the data will vary from other node processes, which can affect performance (no one has yet had any problems here either).
- for cases when the client is subscribed to a sorted query to the collection (Query), there is no possibility of cheaply understanding how changing the next document will affect its position in the given query. At the moment, this is done in a rather clumsy way: the request is applied to the collection, all documents for this request are retrieved, the difference is calculated and sent to the client (for applications with a large number of such requests, this causes performance problems). In this case, the problem could be solved with the help of events to change the index in the database. Unfortunately, there is no DB with events to change the index (if you know, share it). Perhaps it is not so difficult to add to most of the database, but there was never a need (as an option, you can ask to add it to Mongu).

FoundationDB has a Pub-Sub and can replace both Redis and Mongo. The presence of embedded transactions will simplify the logic of the application operations.
Also on top of FoundationDB you can write your Document Model layer, for example fowl , in which you can implement events on index changes. Such an adapter will solve the problem of sorted queries.

Cons FoundationDB - closed source, not free.

Joseph has already livedb-foundation . The case for the adapter.

Client cache

The idea of ​​Offline web applications implies that the client stores data downloaded from the server, even when the application is not running. And uses this cache if you can not connect to the server. The client also stores changes to the data and, when connected to the server, synchronizes the data in the client cache with the data in the database, while conflict situations are resolved. Ideally, the user should be able to open the application without connecting to the Internet, work in it, close it and be sure that the next time he opens the application and has access to the Internet, all the results of his work will go to the server.

Derby / ShareJS seems like a good platform for this. All you need to synchronize the data in the model with IndexedDB, teach Derby to look for data in the client cache first, and then (if the server is available) request the difference from it (new versions). You also need to store operations in IndexedDB when offline, and send them to the server when a connection appears.

Access-control

The first access-control module - racer-access has vulnerabilities with access to queries and is currently not recommended for use. At the moment, it is recommended to perform access control at the ShareJS level manually or using the share-access module.

There is an idea to combine the validation module and access-control, adding to the racer-schema schema a similarity to Firebase Security .

Afterword

Anyone who has questions on Derby / Racer / ShareJS, write me on Skype vmakhaev.
Do not miss the Open Hours in less than an hour. Good luck!