Deform: simple database as a service

Imagine a database that can validate JSON data using a JSON Schema scheme, process events, and process data. And if the idea is implemented as a service using mongodb ?

We have developed just such a service. He is used by our side projects:

docast.me - any data source you can listen / watch in your favorite podcasting application
watchlater - solution to watch later youtube problem
warhealth - mobile app

Under the cut a brief description of the main features of the service with examples.

This article contains sections:

JSON - JSON documents
Validation - data validation
Files - work with files
Processing - data processing
WebHook - event processing

Json

All entities in the service are JSON documents. Thanks to that, mongodb was great for us.

You can work with documents through HTTP REST Api, CLI and Python clients.

I prefer CLI.

Here are some examples of its use:

Create a document in the venues collection

 deform document create -c venues -d '{"text":"hello world"}'

Find out how many users in the project

 $ deform documents count -c my_project_users 108

Validation

I will say right away - the choice fell on JSON Schema draft v4. This scheme provides a lot of useful.

Let's say that 2 documents come to you:

 { "name": "  1 ", "price": 1.0, "currency": "RUB" }

 { "name": "1$ stuff", "price": 1.99, "currency": "United States Dollar" }

And we need the second document to fail to go to the database - the currency has been transferred incorrectly ( currency property).

Let's create a scheme by which the first document passes, and the second one does not.

 { "description": "General goods schema", "type": "object", "properties": { "name": { "type": "string" }, "price": { "type": "number" }, "currency": { "type": "string", "enum": ["RUB", "USD", "CAD", "GBP"] } }, "required": ["name", "price", "currency"] }

Pay attention to the currency and its enum property. Only documents whose currency coincides with the lines "RUB", "USD", "CAD", "GBP" will be considered correct.

Files

Deform supports working with files. All your files will be documents in the _files collection. And in the document itself, where you attached the file, it will also be available as an object.

Let's add to the scheme with the goods a new property - the image of the goods.

 { "description": "General goods schema", "type": "object", "properties": { "name": { "type": "string" }, "price": { "type": "number" }, "currency": { "type": "string", "enum": ["RUB", "USD", "CAD", "GBP"] }, "image": { "type": "file" } }, "required": ["name", "price", "currency"] }

Now we can upload a picture to this product.

Using the CLI, create a document:

 $ deform document create -c goods -d '{ "name": "Teapot", "price": 11.99, "currency": "GBP", "image": @"/tmp/teapot.png" }'

As a result, we get

 { "_id": "576ffce308888f000599ee17", "name": "Teapot", "price": 11.99, "currency": "GBP", "image": { "_id": "5772db5308888f000599f095", "collection_id": "goods", "content_type": "image/png", "date_created": "2016-06-25T18:31:12.356Z", "document_id": "576ec022bd4db46b765ae94a", "last_access": "2016-06-25T17:32:18.347Z", "md5": "a8eda376612338e0286ff1c1a725b111", "name": "teapot.png", "size": 16214 } }

You can get the contents of the file both from the collection with the files and from the product document itself.

Link to the documentation section with files

Processing

Deform can process data. Full list of handlers .

Suppose that we need to change the size of the product image at the time of its creation.

This will help us resize processor (at the moment it works with images only)

The scheme will turn into:

 { "description": "General goods schema", "type": "object", "properties": { "name": { "type": "string" }, "price": { "type": "number" }, "currency": { "type": "string", "enum": ["RUB", "USD", "CAD", "GBP"] }, "photo": { "type": "file" }, "300x150": { "type": "file", "processors": [ { "name": "resize", "in": { "original_image": { "property": "photo" }, "size": { "value": [300,150] } } } ] } } }

Link to the documentation section with processing

Webhook

Our webhukas write history and they have events. They also have headers, methods, and their own validation of matched documents.

Suppose that we need a notification in slack that the product was created. The hook will look like this:

 { "name": "Slack notification", "url": "https://hooks.slack.com/services/....", "method": "POST", "triggers": ["created"], "collection": "slack_notifications" }

Pay attention to the triggers property - in this case only the created documents will trigger the hook.

Now suppose that we only need notifications about goods, where the price is more than 1000 and the USD currency:

 { "name": "Slack notification", "url": "https://hooks.slack.com/services/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", "method": "POST", "triggers": ["created"], "collection": "slack_notifications", "condition": { "type": "object", "additionalProperties": true, "price": { "type": "number", "minimum": 999 }, "currency": { "type": "string", "enum": ["USD"] } } }