Cheat sheet on mongodb: e-commerce, migration, frequently used operations and a little about transactions

This post is a small cheat sheet for mongodb and some long requests with a couple of recipes. Sometimes it is convenient when some little things are collected in one place, I hope everyone who is interested in mongodb will find something useful for themselves.

It would not be desirable, that the post was perceived in a key of holivar on a subject SQL vs. NOSQL SQL vs. NOSQL And so it is clear that everywhere there are pluses and minuses, in this case it is just somewhere a little bit of help, somewhere there are a few examples of what you have come across. Examples on mongo shell and python .

1. Migrating to new versions in mongodb
   
   • Up to version 2.6
   • From 2.6 to 3.0 versions
   • New in PyMongo
2. Comparison and logical queries
3. Full-text search in Mongodb, regexp, indexes, etc.
4. Atomic operators (modifying data)
5. Some words about transactions in Mongodb
   
   • Unique index
   • Biphasic commit
6. Aggregation framework and JOINs in Mongodb
7. Examples
   
   • Trees, comments in Mongodb
   • Tags, blogs
   • E-commerce, and complex filters in Mongodb
8. Small Python sandbox

Mongodb Migration

Up to version 2.6

After the release of version 2.6 , mongodb added a new system for assigning user rights to databases, separate collections. And, accordingly, when updating this should be considered.

1) It is necessary to switch from version 2.4 to version 2.6 . From 2.2 to 2.6 there will be no backward compatibility, so you need to upgrade step by step.

Actually, the update itself:

 apt-get update apt-get install mongodb-org 

2) After upgrading to 2.6 you need to go to the admin database and execute several commands that will check the compatibility of documents.

 use admin db.upgradeCheckAllDBs() 

3) Since from version 2.6 in mongodb , as already mentioned, there are distinctions on roles and setting rights for any user up to the collection to read, write, etc., respectively, these roles must be set, otherwise you will not be able to execute the auth command .

 db.auth('admin','password') 

To do this, you first need to create the user "Administrator" in the admin database

 db.createUser({user:"admin", pwd:"passwd", roles:[{role:"userAdminAnyDatabase", db:"admin"}]}) 

4) After that, go to your desired database, with which they are going to work and to which we want to connect, and create a user there.

 use newdb db.createUser({user:"admin", pwd:"passwd", roles:[{role:"dbAdmin", db:"newdb"}]}) 

A record will automatically be created in the admin database in the system.users collection system.users
You can view database users with the command:

 db.runCommand( { usersInfo: [ { user: "admin", db: "newdb" } ], showPrivileges: true } ) 

Well, do not forget to reboot after all this.

 service mongod restart 

In ubuntu c this version of the service is called not mongodb but mongod and the config in /etc is called mongod.conf . Most likely, this is due to the lack of backward compatibility so that you don’t mix up the update.

From 2.6 to 3.0 versions

A lot has already been written about the new version 3.0 and the revolutionary changes in the storage engine; I will not repeat it.

Before upgrading to 3.0 it is recommended to consistently upgrade to version 2.6 without skipping. That is, 2.2->2.4->2.6 .
The latest version is recommended at least 2.6.5 .
The installation for ubuntu pretty standard.
Add the 3rd version repository:

 sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 7F0CEB10 echo "deb http://repo.mongodb.org/apt/ubuntu "$(lsb_release -sc)"/mongodb-org/3.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.0.list 

For Ubuntu 15.04

 echo "deb http://repo.mongodb.org/apt/ubuntu trusty/mongodb-org/3.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.0.list 

Install:

 apt-get update apt-get install -y mongodb-org 

For each component, specify the version during installation.

 apt-get install -y mongodb-org=3.0.2 mongodb-org-server=3.0.2 mongodb-org-shell=3.0.2 mongodb-org-mongos=3.0.2 mongodb-org-tools=3.0.2 

Thereafter:

 service mongod stop service mongod start 

mongodb version of mongodb :

 root@user-pc:~# mongo MongoDB shell version: 3.0.2 connecting to: test > db.version() 3.0.2 > 

If version 3 then everything went fine and now you can change the storage. The default is MMAPv1 .

To change to /etc/mongo.conf we set the option:

 storageEngine = wiredTiger 

Read more about the possible options associated with the new storage here.
And we see that the /var/lib/mongodb empty, otherwise mongodb will not start, of course, before this, all databases need to be made mongodump

 service mongod restart 

Check the version of the engine for storage:

 root@user-pc:/etc# mongo MongoDB shell version: 3.0.2 connecting to: test > db.serverStatus() 

We are looking for storageEngine if wiredTiger then everything is fine.

 "storageEngine" : { "name" : "wiredTiger" } 

Now you need to import the database, including admin

 mongorestore --port 27017 -d admin 

New in PyMongo

Together with the new version of the database, a new version of the driver for Python PyMongo , and some obsolete methods have been removed there and after

 pip install -U pymongo 

Even without updating the database itself, not everything will work as before. From what was immediately noticed:

1. For universalization and unification, methods update_one, insert_many, find_one_and_delete more detail in the specification
2. Also, only one connector to the MongoClient database was left for unification; options such as 'slave_okay': True removed from it 'slave_okay': True . ReplicaSetConnection and MasterSlaveConnection now removed. MongoReplicaSetClient left for some time for compatibility.

Usage example:

 >>> # Connect to one standalone, mongos, or replica set member. >>> client = MongoClient('mongodb://server') >>> >>> # Connect to a replica set. >>> client = MongoClient('mongodb://member1,member2/?replicaSet=my_rs') >>> >>> # Load-balance among mongoses. >>> client = MongoClient('mongodb://mongos1,mongos2') 

1. Removed copy_database method
2. Removed end_request() method instead recommended to use close()
3. Part of the community expected native support for asynchronous programming and asyncio from python3 , but unfortunately, alas. For tornado there is a good motor driver. And for asyncio , unfortunately, only the experimental asyncio-mongo driver remains underdeveloped and lacks GridFS support GridFS

Comparison and logical queries

$eq comparison operator
$eq db.test.find({ field: <value> }) equivalent to db.test.find({ field: <value> })

 { _id: 1, item: { name: "ab", code: "123" }, qty: 15, tags: [ "A", "B", "C" ] } { _id: 2, item: { name: "cd", code: "123" }, qty: 20, tags: [ "B" ] } db.test.find( { qty: { $eq: 20 } } ) # db.test.find( { qty: 20 } ) db.test.find( { tags: { $eq: [ "A", "B" ] } } ) # : db.test.find( { tags: [ "A", "B" ] } ) 

$ gt more than
$gt selects those documents where the field value is greater than (>) specified value.

 db.test.find( { qty: { $gt: 10 } } ) 

$ gte is greater than or equal to
$gte selects those documents where the field value is greater than or equal to (>=) specified value.

 db.test.find( { qty: { $gte: 10 } } ) 

$ lt less than
$lt selects those documents where the field value is less than (<) specified

 db.test.find( { qty: { $lt: 10 } } ) 

$ lte is less than or equal to
$lte selects those documents where the field value is less than or equal to (<=) specified

 db.test.find( { qty: { $lte: 10 } } ) 

If the seller’s profit is less than 100 , then the bonus is canceled.

 db.test.update({ "vendor.profit": { $lte: 100 } }, { $set: { premium: 0 } }) 

$ ne is not equal
$ne selects documents where the field value is not equal (! =) specified value.

 db.test.find( { qty: { $ne: 10 } } ) 

$ in entry check

 { _id: 1, qty: 10, tags: [ "name", "lastname" ], } db.test.find({ tags: { $in: ["name", "lastname"] } } ) 

Regular expression example

 db.test.find( { tags: { $in: [ /^be/, /^st/ ] } } ) 

$ nin check for not entering
Same as $in but on the contrary, it checks that some value is missing in the array.

 db.test.find( { qty: { $nin: [ 5, 15 ] } } ) 

$ or operator or
The classical operator takes several values ​​and checks that at least one of them matches the condition.

 db.test.find( { $or: [ { quantity: { $lt: 10 } }, { price: 10 } ] } ) 

For this request it is proposed to compile two indices:

 db.test.createIndex( { quantity: 1 } ) db.test.createIndex( { price: 1 } ) 

If $or used together with the $text operator intended for full-text search, then the index must be.

$ and operator "and"
The operator checks the presence of all listed values ​​in the desired documents.

 db.test.find( { $and: [ { price:10 }, { check: true } } ] 

Example with $or :

 db.test.find( { $and : [ { $or : [ { price : 50 }, { price : 80 } ] }, { $or : [ { sale : true }, { qty : { $lt : 20 } } ] } ] } ) 

$ not negation operator
Verifies that there are no documents in the sample that match the condition.

 db.test.find( { price: { $not: { $gt: 10 } } } ) 

Example with regular expression:

 import re for no_docs in db.test.find( { "item": { "$not": re.compile("^p.*") } } ): print no_docs 

$ nor operator not or

 db.test.find( { $nor: [ { price: 10 }, { qty: { $lt: 20 } }, { sale: true } ] } ) 

This query in the test collection will find those documents in which:

• The value of the field `price` is not equal to 10
• The value of the field `qty` at least 20
• `Sale` value is not` true`

$ exists checking the field for existence
$exists retrieves those documents in which a particular key is present or missing.
If we specify false as the parameter in $exists , the query will return those documents in which the qty key is not defined.

 db.test.find( { qty: { $exists: true } } ) 

$ type BSON type check

 db.test.find( { field: { $type: -1 } } ); 

$ mod
The $mod operator is used to select fields whose values ​​are divided by the first argument and the remainder of the division is equal to the second.
For example, there are documents:

 { "_id" : 1, "item" : "aa123", "qty" : 0 } { "_id" : 2, "item" : "bb123", "qty" : 7 } { "_id" : 3, "item" : "cc123", "qty" : 15 } 

Request:

 db.test.find( { qty: { $mod: [ 5, 0 ] } } ) 

Return the following documents:

 { "_id" : 1, "item" : "aa123", "qty" : 0 } { "_id" : 3, "item" : "cc123", "qty" : 15 } 

Analogue from SQL

 select * from t where qty % 5 = 0; 

With version 2.6 forbidden to transmit only one item, an error will be returned. Also, if you pass more than three arguments, it will also give an error; in previous versions, redundant arguments were simply ignored.
$ all select matching all
Makes a sample of more than one array element.

 db.test.find( { tags: { $all: [ "python", "mongodb", "javascript" ] } } ) 

$ elemMatch
Used when you want to compare two or more attributes belonging to the same subdocument.
Checks that there is an element in the array that meets all conditions.

 { _id: 1, results: [ 82, 85, 88 ] } { _id: 2, results: [ 75, 88, 89 ] } db.test.find( { results: { $elemMatch: { $gte: 80, $lt: 85 } } } ) 

We get the result:

 { _id: 1, results: [ 82, 85, 88 ] } 

Another example:

 { _id: 1, results: [{ product: "abc", score: 10 }, { product: "xyz", score: 5}] } { _id: 2, results: [{ product: "abc", score: 8 }, { product: "xyz", score: 7}] } { _id: 3, results: [{ product: "abc", score: 7 }, { product: "xyz", score: 8}] } >db.test.find( { results: { $elemMatch: { product: "xyz", score: { $gte: 8 } } } } ) { "_id": 3, "results": [{ "product": "abc", "score": 7 }, { "product": "xyz", "score": 8 } ] } 

$ size searches by array length
The $size operator finds documents in which the number of elements in the array is equal to the value of $size . For example, retrieve all the documents in which there are two elements in the laguages ​​array:

 db.persons.find ({languages: {$size:2}}) 

Such a request would correspond, for example, to the following document:

 {"name": "Alex", "age": "32", languages: ["python", "mongodb"]} 

$ positional operator
$ can be used in different cases. When we do not know by what index the value lies in the array but we want to use it, then we use the "positional operator"
For example, there are documents:

 { "_id" : 3, "semester" : 1, "grades" : [ 85, 100, 90 ] } { "_id" : 4, "semester" : 2, "grades" : [ 79, 85, 80 ] } 

And we want to see after the search for them only one value corresponding to the request, not the entire document, but we don’t know in advance what value is there.

 >db.test.find( { semester: 1, grades: { $gte: 85 } }, { "grades.$": 1 } ) { "_id" : 3, "grades" : [ 85 ] } 

Example for update:

 db.test.update( { _id: 22 } , { $set: { "array.$.name" : "alex" } } ) 

$ slice finds range
$slice - finds the range of values ​​stored in the array.
Find the first 10 events:

 db.test.find( { }, { "events" : { $slice: 10 } } ) 

Find the last 10 events:

 db.test.find( { }, { "events" : { $slice: -10 } } ) 

Full-text search in Mongodb, regexp, indexes, etc.

On Habré there was a good publication mongodb`, but since that time new operators and new features have been added.
Text search does not work without indexes, so let's talk about them.
A simple index is created by any text field or array.

 db.test.createIndex( { title: "text", content: "text" } ) 

You can use the field name or wildcard specifier:

 db.text.createIndex( { "$**": "text" } ) 

When creating an index for full-text search, you must consider the language, if it is not English.

 db.test.createIndex( { content : "text" }, { default_language: "russian" } ) 

Starting from version 2.6 , it became possible to set a text index for many languages ​​at once.

MongoDB will use the language specified in the document when building the index. The language specified in the document overrides the default language. The language in the inline document overrides all others for the index.

 { _id: 1, language: "portuguese", original: "A sorte protege os audazes.", translation: [ { language: "english", quote: "Fortune favors the bold." }, { language: "russian", quote: "  ." } ] } 

Also, it is possible to specify a field with language using the parameter language_override .
For example, for documents:

 { _id: 2, idioma: "english", quote: "Fortune favors the bold." } { _id: 3, idioma: "russian", quote: "  ." } 

The index will look like this:

 db.text.createIndex( { quote : "text" }, { language_override: "idioma" } ) 

The index can be assigned a special name { name: "name" } , for example:

 db.text.createIndex( { content: "text", "users.title": "text" }, { name: "text_Index" } ) 

The name is convenient to use to remove indexes:

 db.text.dropIndex("text_Index") 

Also, for a text index, you can set the value, the weight of the field to search.
For example, set the weight for the following fields: content - 10 , keywords - 5 , and title - 1 .

 db.test.createIndex( { content: "text", tags: "text", title: "text" }, { weights: { content: 10, tags: 5, }, name: "TextIndex"} ) 

Through the index, you can limit the number of records in the issue:

 { _id: 1, dept: "one", content: "red" } { _id: 3, dept: "one", content: "red" } { _id: 2, dept: "two", content: "gren" } db.test.createIndex( { dept: 1, content: "text" } ) db.test.find( { dept: "one", $text: { $search: "green" } } ) 

The output will be only one document instead of two, since we have limited the index.
Sample index for Python :

 #PyMongo db.text.ensure_index( [ ('descr', "text" ), ( 'title.ru', "text" ) ], default_language="russian", name="full_text") 

Text search

Immediately after the appearance of the text search in mongodb, it was carried out using runCommand for example:

 db.collection.runCommand( "text", { search: "" } ) 

but, starting with version 2.6 , a new $ text operator has appeared
Search for one word:

 db.articles.find( { $text: { $search: "coffee" } } ) 

Search for multiple words:

 db.articles.find( { $text: { $search: "bake coffee cake" } } ) 

Search by phrase:

 db.articles.find( { $text: { $search: "\"coffee cake\"" } } ) 

Excluding a field from a search in -

 db.articles.find( { $text: { $search: "bake coffee -cake" } } ) 

Also with mongodb 2.6 , another $ meta operator appeared, showing the accuracy of the result match with the query.

 db.text.insert([ { "_id": 4, "descr" : "  " }, { "_id": 3, "descr" : "    -" }, { "_id":6, "descr" : "  " }, { "_id":7, "descr" : "   " }, { "_id":8, "descr" : "  " }, { "_id":9, "descr" : ",    " }, { "_id":10, "descr" : ",    " }, { "_id":11, "descr" : "   " } { "_id":12, "descr" : "  " }, { "_id":13, "descr" : " " }, ]) db.text.createIndex( { descr : "text" }, { default_language: "russian" } ) db.text.find( { $text: { $search: "" } }, { score: { $meta: "textScore" } }).sort( { score: { $meta: "textScore" } } ) { "_id" : 13, "descr" : " ", "score" : 0.75 } { "_id" : 4, "descr" : "  ", "score" : 0.6666666666666666 } { "_id" : 6, "descr" : "  ", "score" : 0.6666666666666666 } { "_id" : 12, "descr" : "  ", "score" : 0.6666666666666666 } 

Here { score: { $meta: "textScore" } } we create a new field, its value contains the result and then it already participates in sorting.
$ Regex search
MongoDB uses Perl compatible regular expressions.

 db.test.insert([ { "_id" : 1, "descr" : "abc123" }, { "_id" : 2, "descr" : "abc123" }, { "_id" : 3, "descr" : "eee789" } ]) db.test.find( { sku: { $regex: /^ABC/i } } ) { "_id" : 1, "sku" : "abc123", "description" : "Single line description." } { "_id" : 2, "sku" : "abc123", "description" : "Single line description." } 

i - Case insensitivity.
PostgreSQL

 select title from article where title ~ '^a' 'abc' 

Atomic operators (modifying data)

As a rule, all these modifiers are used for update operations in db.test.update() and db.test.findAndModify()

$ inc increment
Increases or decreases the field by the specified value.

 db.test.update( { _id: 1 }, { $inc: { qty: -2, "orders": 1 } } ) 

$ mul multiplicative increment
Multiplies the field value by the specified amount.

 { _id: 5, item: "mac", price: 10 } db.test.update({ _id: 1 }, { $mul: { price: 2 } } ) { _id: 5, item: "mac", price : 20 } 

$ rename rename field

 { "_id": 1, "name": "alex" } db.test.update( { _id: 1 }, { $rename: { 'name': 'alias'} } ) { "_id": 1, "alias": "alex" } 

$ set changes field values
This is probably the main modifying operator used with update . Often he is remembered as unpretentious transactions in the context of mongodb.

 db.test.save({ "_id":8, "qty":"", tags:"" }) db.test.update( { _id: 8 }, { $set: { qty: 100, tags: [ "linux", "ubuntu"] } }) { "_id" : 8, "qty" : 100, "tags" : [ "linux", "ubuntu" ] } 

$ setOnInsert adds fields to a new document.
In the update third argument is the { upsert: true } option, which means that if the document for the change is not found, then we create a new one. And the $setOnInsert option tells us which fields to insert there.

 >db.test.update( { _id: 7 }, { $set: { item: "windows" }, $setOnInsert: { os: 'bad' } }, { upsert: true } ) { "_id" : 7, "item" : "windows", "os" : "bad" } 

The field for which we execute $set will also appear in the newly created document.
$ unset deletes the key

 { "_id" : 8, "qty" : 100, "tags" : [ "linux", "ubuntu" ] } db.test.update( { _id: 8 }, { $unset: { qty: "", tags: "" } } ) { "_id" : 8 } 

$ min updates if less
$min updates a field; if the specified value is less than the current field value, $min can compare values ​​of different types.

 > db.test.save({ _id: 9, high: 800, low: 200 }) > db.test.update( { _id:9 }, { $min: { low: 150 } } ) >db.test.findOne({_id:9}) { "_id" : 9, "high" : 800, "low" : 150 } 

$ max updates if more
$max updates the field if the specified value is greater than the current field value.

 > db.test.save({ _id: 9, high: 800, low: 200 }) > db.test.update( { _id:9 }, { $max: { high: 900 } } ) > db.test.findOne({_id:9}) { "_id" : 9, "high" : 900, "low" : 200 } 

$ currentDate sets the current date
Sets the field value to the current date.

 > db.test.save({ _id:11, status: "init", date: ISODate("2015-05-05T01:11:11.111Z") }) > db.test.update( { _id:12 }, { $currentDate: { date: true } } ) > db.test.findOne({_id:12}) { "_id" : 12, "status" : "a", "date" : ISODate("2015-05-10T21:07:31.138Z") } 

Array changes

$ addToSet adds value if none
Adds a value to an array, if it is not there yet, and if it is, it does nothing.

 db.test.save({ _id:1, array: ["a", "b"] }) db.test.update( { _id: 1 }, { $addToSet: {array: [ "c", "d" ] } } ) { "_id" : 1, "array" : [ "a", "b", [ "c", "d" ] ] } db.test.update( { _id: 1 }, { $addToSet: {array: "e" } } ) { "_id" : 1, "array" : [ "a", "b", [ "c", "d" ], "e" ] } 

$ pop deletes 1st or last
Removes the first or last element of an array. If -1 is specified, it will delete the first element, if it is 1, then the last one.

 > db.test.save({ _id: 1, scores: [ 6, 7, 8, 9, 10 ] }) { "_id" : 1, "scores" : [ 6, 7, 8, 9, 10 ] } > db.test.update( { _id: 1 }, { $pop: { scores: -1 } } ) > db.test.findOne({_id:1}) { "_id" : 1, "scores" : [ 7, 8, 9, 10 ] } > db.test.update( { _id: 1 }, { $pop: { scores: 1 } } ) > db.test.findOne({_id:1}) { "_id" : 1, "scores" : [ 7, 8, 9 ] } 

$ pullAll removes all specified
Removes all the specified elements from the array.

 { _id: 1, scores: [ 0, 2, 5, 5, 1, 0 ] } db.test.update( { _id: 1 }, { $pullAll: { scores: [ 0, 5 ] } } ) { "_id" : 1, "scores" : [ 2, 1 ] } 

$ pull removes as requested

 { _id: 1, votes: [ 3, 5, 6, 7, 7, 8 ] } > db.test.update( { _id: 1 }, { $pull: { votes: { $gte: 6 } } } ) { _id: 1, votes: [ 3, 5 ] } 

$ push adds values
Adds values ​​to an array.

 db.test.update( { _id: 1 }, { $push: { scores: 100} } ) 

$pushAll - considered obsolete

Modifiers for $ push

$ each at once a lot
Adds each of the listed elements to the array.
For example, if we do this: { $push: { scores: [ 2, 10 ] } }
Then the output will be the following array: "scores" : [7, 8, 9, 90, 92, 85, [ 2, 10 ] ]
that is, one more element was added which is an array.
And if in $each , then each element of the list is added as an element of the array:

 > db.test.update( { _id: 1 }, { $push: {scores: { $each: [ 90, 92, 85 ] } } } ) {"_id" : 1, "scores" : [7, 8, 9, 90, 92, 85, 2, 10 ] } 

$ slice limits the number of elements when using $ push
Limits the number of elements in an array when inserted using $push . Be sure to use $each if you try to use without it, it will return an error.

 { "_id" : 1, "scores" : [ 10, 20, 30 ] } > db.test.update( { _id: 1 }, { $push: { scores: { $each: [50, 60, 70], $slice: -5 } } } ) { "_id" : 1, "scores" : [ 20, 30, 50, 60, 70 ] } 

$slice cut the first element 20 . if we indicated not -5 but 5 he would have thrown off the last element 70 .

$ sort sorting array elements
Sorts the elements of the array according to the specified field. Also be sure to use with the $each operator. If you just need to sort without an insert, you can leave $each empty.

 { "_id" : 2, "tests" : [ 80, 70, 80, 50 ] } > db.test.update( { _id: 2 }, { $push: { tests: { $each: [40, 60], $sort: 1 } } }) { "_id" : 2, "tests" : [ 40, 50, 60, 70, 80, 80 ] } 

Another example:

 db.test.update( { _id: 1 }, { $push: { field: { $each: [ ], $sort: { score: 1 } } } }) { "_id" : 1, "field" : [ { "id" : 3, "score" : 5 }, { "id" : 2, "score" : 6 }, { "id" : 1, "score" : 7 }, ] } 

$ position indicates insertion position
Specifies from which element of the array to insert values.

 { "_id" : 1, "scores" : [ 100 ] } db.test.update({ _id: 1 }, {$push: { scores: { $each: [50, 60, 70], $position: 0 } } }) { "_id" : 1, "scores" : [ 50, 60, 70, 100 ] } 

$ bit updates bit by bit
Performs a bitwise field update. The operator supports bitwise and , or and xor .

 { "_id" : 1, "expdata" : 13 } > db.bit.update({_id:1}, {$bit:{expdata:{and:NumberInt(10)} } } ) { "_id" : 1, "expdata" : 8 } 

$ isolated - atomization
Blocks a document for reading and writing while it is being processed, for example, an update operation.
Using $isolated when deleting:

 db.test.remove( { temp: { $lt: 10 }, $isolated: 1 } ) 

Using $isolated when updating:

 db.test.update( {status: "init" , $isolated: 1 }, { $inc: { count : 1 }}, {multi: true } ) 

$ isolated does not work with shard Clusters
Since version 2.2 : $isolated operator replaced $atomic

About transactions in mongodb, a unique index, a two-phase commit

Naturally, such transactions as in classic SQL solutions like PostgreeSQL in MongoDB are not and probably cannot be. And if it does, it will be, rather, a relational database with full normalization and integrity control.
Therefore, when speaking of transactions in mongoDB , mongoDB , as a rule, mean atomic operations like $set , used in update() and findAndModify() in combination with a unique index. As well as a two-phase commit that is common among relational databases, if you need to secure transactions within several databases.

Unique index

The unique index in mongodb is the reason for rejecting all documents that contain duplicate values ​​for indexed fields.

 db.test.createIndex( { "user_id": 1 }, { unique: true } ) 

There is a collection, let's call it test , in this collection there are no documents whose name field would have the value Nik . Suppose that several clients simultaneously try to update this document with the { upsert: true } parameter (means that if by condition there is no such document to be updated, then it needs to be created).
Example:

 db.test.update( { name: "Nik" }, { name: "Nik", vote: 1 }, { upsert: true } ) 

update() , , , .

, . . , .

unique false MongoDB

, A B .
:

• `accounts` .
• `transactions` , .

accounts transactions
accounts

 db.accounts.insert( [ { _id: "A", balance: 1000, pendingTransactions: [] }, { _id: "B", balance: 1000, pendingTransactions: [] } ] ) 

transactions .

 db.transactions.insert({ _id: 1, source: "A", destination: "B", value: 100, state: "initial", lastModified: new Date()}) 

:

• source destination .
• value , .
• state . initial , pending , applied , done , canceling , canceled .
• lastModified .

1)
, initial . t

 > var t = db.transactions.findOne( { state: "initial" } ) > t { "_id" : 1, "source" : "A", "destination" : "B", "value" : 100, "state" : "initial", "lastModified" : ISODate("2015-05-26T16:35:54.637Z") } 

2) pending
initial pending .

 > db.transactions.update( { _id: t._id, state: "initial" }, { $set: { state: "pending" }, $currentDate: { lastModified: true } } ) > db.transactions.find() { "_id" : 1, "source" : "A", "destination" : "B", "value" : 100, "state" : "pending", "lastModified" : ISODate("2015-05-26T17:02:19.002Z") } > 

3)
, , value , pendingTransactions _id ( ).

 > db.accounts.update( { _id: t.source, pendingTransactions: { $ne: t._id } }, { $inc: { balance: -t.value }, $push: { pendingTransactions: t._id } } ) > db.accounts.update( { _id: t.destination, pendingTransactions: { $ne: t._id } }, { $inc: { balance: t.value }, $push: { pendingTransactions: t._id } } ) > db.accounts.find() { "_id" : "A", "balance" : 900, "pendingTransactions" : [ 1 ] } { "_id" : "B", "balance" : 1100, "pendingTransactions" : [ 1 ] } 

4) applied
.

 > db.transactions.update( { _id: t._id, state: "pending" }, { $set: { state: "applied" }, $currentDate: { lastModified: true } } ) > db.transactions.find() { "_id" : 1, "source" : "A", "destination" : "B", "value" : 100, "state" : "applied", "lastModified" : ISODate("2015-05-26T17:13:15.517Z") } 

5) _id
pendingTransactions : _id pendingTransactions .

 > db.accounts.update( { _id: t.source, pendingTransactions: t._id }, { $pull: { pendingTransactions: t._id } } ) > db.accounts.update( { _id: t.destination, pendingTransactions: t._id }, { $pull: { pendingTransactions: t._id } } ) > db.accounts.find() { "_id" : "A", "balance" : 900, "pendingTransactions" : [ ] } { "_id" : "B", "balance" : 1100, "pendingTransactions" : [ ] } 

6) done
.

 > db.transactions.update( { _id: t._id, state: "applied" }, { $set: { state: "done" }, $currentDate: { lastModified: true } } ) > db.transactions.find() { "_id" : 1, "source" : "A", "destination" : "B", "value" : 100, "state" : "done", "lastModified" : ISODate("2015-05-26T17:22:22.194Z") } 

. .
1) canceling
, , canceling .

 db.transactions.update( { _id: t._id, state: "pending" }, {$set: { state: "canceling" }, $currentDate: { lastModified: true }} ) > db.transactions.find() { "_id" : 1, "source" : "A", "destination" : "B", "value" : 100, "state" : "canceling", "lastModified" : ISODate("2015-05-26T18:29:28.018Z") } 

2)
.

 > db.accounts.update( { _id: t.destination, pendingTransactions: t._id }, { $inc: { balance: -t.value }, $pull: { pendingTransactions: t._id } } ) > db.accounts.update( { _id: t.source, pendingTransactions: t._id }, { $inc: { balance: t.value}, $pull: { pendingTransactions: t._id } } ) > db.accounts.find() { "_id" : "A", "balance" : 1000, "pendingTransactions" : [ 1 ] } { "_id" : "B", "balance" : 1000, "pendingTransactions" : [ 1 ] } > 

3) cancelled
.

 db.transactions.update( { _id: t._id, state: "canceling" }, { $set: { state: "cancelled" }, $currentDate: { lastModified: true } } ) > db.transactions.find() { "_id" : 1, "source" : "A", "destination" : "B", "value" : 100, "state" : "cancelled", "lastModified" : ISODate("2015-05-26T19:14:11.830Z") } 

, . .
findAndModify() , :

 t = db.transactions.findAndModify({ query: { state: "initial", application: { $exists: false } }, update: {$set: { state: "pending", application: "App1"}, $currentDate:{ lastModified: true }}, new: true }) 

6. JOIN-

JOIN - mongo , - . stackoverflow , , .

, , , , .

- , , { type: 'news' } .

, , .

, , aggregation framework . pipeline . , , , , , ..

, , .

 db.test.insert([ { "_id":"gomer", "type":"user", "group":["user", "author"] }, { "_id":"vasya", "type":"user", "group":["user"] } ]) 

.

 db.test.insert([ { "_id": 1, "type": "blogs", "user": "gomer", "article": "aaa" }, { "_id": 2, "type": "blogs", "user": "vasya", "article": "bbb" }, { "_id": 3, "type": "blogs", "user": "gomer", "article": "ccc" } ]) 

, . , .

, , , "".

 users = [doc._id for doc in db.test.find({"type":'user', 'group': {'$all': ['author']}})] articles = db.test.find({"type": "blogs", "user": {'$in': users}) 

c SQL :

 SELECT blogs.* FROM blogs, user, usergroup, group WHERE blogs.user = user.id AND usergroup.user = user.id AND usergroup.group = group.id AND group.name = 'author'; 

user, d jsonb. — :

 SELECT blogs.* FROM blogs, user WHERE blogs.user = user.id AND user.group ? 'author'; SELECT blogs.* FROM (SELECT * FROM test WHERE d->type = 'blogs') blogs, (SELECT * FROM test WHERE d->type = 'user') user WHERE blogs.d->user = user.id AND user.d->group ? 'author'; 

pipe.

 db.test.aggregate([ { $match: { $or: [ {type: "blogs"}, {type: "user"} ] } }, { $project: { a: 1, blogs: { $cond: { if: { type: '$blogs'}, then: {_id:"$_id", user:"$user", article:"$article"}, else: null } }, user: { $cond: { if: { type: '$user' }, then: { _id:"$_id", group:"$group"}, else: null } } } }, { $group : { _id : { a: "$a" }, user: { $push: "$user" }, blog: { $push: "$blogs" }, } }, { $unwind : "$blog" }, { $unwind : "$user" }, { $project:{ user: "$user", article: "$blog", matches: { $eq:[ "$user._id", "$blog.user" ] } } }, { $match: { matches: true } } ]) 

. 7 .
, mongodb pipeline . , , , , , .

.

 > db.ag.aggregate([ { $match: {$or:[{type:"blogs"},{type:"user"}]} } ]) { "_id" : "gomer", "type" : "user", "group" : [ "user", "author" ] } { "_id" : "vasya", "type" : "user", "group" : [ "user" ] } { "_id" : 1, "type" : "blogs", "user" : "gomer", "article" : "aaa" } { "_id" : 2, "type" : "blogs", "user" : "vasya", "article" : "bbb" } { "_id" : 3, "type" : "blogs", "user" : "gomer", "article" : "ccc" } 

$match find() , , .
c $project , blogs users . 2.6 $cond . blogs users , .

 db.test.aggregate([ { $match: {$or:[ { type:"blogs"}, { type: "user"} ] } }, { $project: { a: 1, blogs: { $cond: { if: {type: '$blogs'}, then: {_id:"$_id", user:"$user", article:"$article"}, else: null } }, user: { $cond: { if: { type: '$user'}, then: {_id:"$_id", group:"$group"}, else: null } } } } ]) { "_id": "gomer", "blogs": { "_id" : "gomer" }, "user": { "_id": "gomer", "group": [ "user", "author" ] } } { "_id": "vasya", "blogs": { "_id" : "vasya" }, "user" : { "_id" : "vasya", "group": [ "user" ] } } { "_id": 1, "user": { "_id": 1 }, "blogs" : { "_id": 1, "user": "gomer", "article": "aaa" } } { "_id": 2, "user": { "_id": 2 }, "blogs" : { "_id": 2, "user": "vasya", "article": "bbb" } } { "_id": 3, "user": { "_id": 3 }, "blogs" : { "_id": 3, "user": "gomer", "article": "ccc" } } 

:

 ...{ $group : { _id : { a: "$a" }, user: { $push: "$user" }, blog: { $push: "$blogs" }, } }... { "_id" : { "a" : null }, "user": [ { "_id": "gomer", "group": [ "user", "author" ] }, { "_id": "vasya", "group": [ "user" ] }, { "_id": 1 }, { "_id": 2 }, { "_id": 3 } ], "blog": [ { "_id": "gomer" }, { "_id": "vasya" }, { "_id": 1, "user": "gomer", "article": "aaa" }, { "_id": 2, "user": "vasya", "article": "bbb" }, { "_id": 3, "user": "gomer", "article": "ccc"} ] } 

 ....{ $unwind : "$blog" }, { $unwind : "$user" } .... { "_id": { "a":null }, "user": { "_id": "gomer", "group": [ "user", "author" ] }, "blog": { "_id": "gomer" } } { "_id": { "a":null }, "user": { "_id": "vasya", "group": [ "user" ] }, "blog" : { "_id": "gomer" } } { "_id": { "a":null }, "user": { "_id": 1 }, "blog": { "_id" : "gomer" } } { "_id": { "a" : null }, "user" : { "_id" : 2 }, "blog" : { "_id" : "gomer" } } { "_id": { "a" : null }, "user" : { "_id" : 3 }, "blog" : { "_id" : "gomer" } } { "_id": { "a": null }, "user": { "_id": "gomer", "group" : [ "user", "author" ] }, "blog": { "_id": "vasya"}} { "_id": { "a" : null }, "user" : { "_id" : "vasya", "group" : [ "user" ] }, "blog" : { "_id" : "vasya" } } { "_id": { "a" : null }, "user" : { "_id" : 1 }, "blog" : { "_id" : "vasya" } } { "_id": { "a" : null }, "user" : { "_id" : 2 }, "blog" : { "_id" : "vasya" } } { "_id": { "a" : null }, "user" : { "_id" : 3 }, "blog" : { "_id" : "vasya" } } { "_id": { "a" : null }, "user" : { "_id" : "gomer", "group" : [ "user", "author" ] }, "blog" : { "_id" : 1, "user": "gomer", "article" : "aaa" } } { "_id": { "a" : null }, "user": { "_id" "vasya", "group": [ "user" ] }, "blog": { "_id": 1, "user": "gomer", "article": "aaa" } } { "_id": { "a" : null }, "user" : { "_id" : 1 }, "blog" : { "_id" : 1, "user" : "gomer", "article" : "aaa" } } { "_id" : { "a" : null }, "user" : { "_id" : 2 }, "blog" : { "_id" : 1, "user" : "gomer", "article" : "aaa" } } { "_id": { "a" : null }, "user" : { "_id" : 3 }, "blog" : { "_id" : 1, "user" : "gomer", "article" : "aaa" } } { "_id": { "a" : null }, "user" : { "_id" : "gomer", "group" : [ "user", "author" ] }, "blog" : { "_id" : 2, "user": "vasya", "article" : "bbb" } } { "_id": { "a" : null }, "user" : { "_id" : "vasya", "group" : [ "user" ] }, "blog" : { "_id" : 2, "user" : "vasya", "article" : "bbb" } } { "_id": { "a" : null }, "user" : { "_id" : 1 }, "blog" : { "_id" : 2, "user" : "vasya", "article" : "bbb" } } { "_id": { "a" : null }, "user" : { "_id" : 2 }, "blog" : { "_id" : 2, "user" : "vasya", "article" : "bbb" } } { "_id": { "a" : null }, "user" : { "_id" : 3 }, "blog" : { "_id" : 2, "user" : "vasya", "article" : "bbb" } } 

, $eq:[ "$user._id", "$blog.user" ] "user" : { "_id" : 2 } "blog" : { "user" : "vasya" } .

 ...{ $project:{ user:"$user", article:"$blog", matches:{ $eq:[ "$user._id", "$blog.user" ] } } } ..... 

 { "_id" : { "a" : null }, "user" : { "_id" : 1 }, "article" : { "_id" : 1, "user" : "gomer", "article" : "aaa" }, "matches" : false } { "_id" : { "a" : null }, "user" : { "_id" : 2 }, "article" : { "_id" : 1, "user" : "gomer", "article" : "aaa" }, "matches" : false } { "_id" : { "a" : null }, "user" : { "_id" : 3 }, "article" : { "_id" : 1, "user" : "gomer", "article" : "aaa" }, "matches" : false } { "_id" : { "a" : null }, "user" : { "_id" : "gomer", "group" : [ "user", "author" ] }, "article" : { "_id" : 2, "user" : "vasya", "article" : "bbb" }, "matches" : false } { "_id" : { "a" : null }, "user" : { "_id" : "vasya", "group" : [ "user" ] }, "article" : { "_id" : 2, "user" : "vasya", "article" : "bbb" }, "matches" : true } { "_id" : { "a" : null }, "user" : { "_id" : 1 }, "article" : { "_id" : 2, "user" : "vasya", "article" : "bbb" }, "matches" : false } { "_id" : { "a" : null }, "user" : { "_id" : 2 }, "article" : { "_id" : 2, "user" : "vasya", "article" : "bbb" }, "matches" : false } { "_id" : { "a" : null }, "user" : { "_id" : 3 }, "article" : { "_id" : 2, "user" : "vasya", "article" : "bbb" }, "matches" : false } Type "it" for more 

:

 { $match: { matches:true } } 

, .
, .

, pipeline .

7.

, , . :

• , .
• , .

, Mongodb

, — . , , , .

.
:

 { _id:1, type:"blog", title:{ru:"O MongoDB", en:""}, comments: [ { _id: 1, title: "one", "user": "Alex", parent: "root", child: [2, 3]}, { _id: 2, title: "two", "user": "Serg", parent: 1 }, { _id: 3, title: "two", "user": "Andrey", parent: 1 } ] } 

:

 { _id: 1, type: "comment", title: "one", "user": "Alex", parent: "root", child: [ 2, 3 ] }, { _id: 2, type: "comment", title: "two", "user": "Serg", parent: 1 }, { _id: 3, type: "comment", title: "two", "user": "Andrey", parent: 1 } 

.
, .
, , "" .

.
:

 db.test.update( { _id: 1 }, { $pull: { child: 2 } } ) db.test.update( { _id: 1 }, { $pullAll: { child: [ 2, 3 ] } } ) 

:

 db.test.update( { _id: 1 }, { $push: { child: 2 } } } ) db.test.update( { _id: 1 }, { $push: { child: { $each: [ 2, 3 ] } } } ) 

:

 def getTree(docs): tree = { doc["_id"]: doc for doc in docs } for doc in docs: doc['child'] = [] for doc in docs: parent = doc["parent"] if parent != "root": tree[parent]["child"].append(doc) docs={"_id": "root", "child": [doc for doc in docs if doc['parent'] == "root" ]} return docs { _id: 1, type: "comment", title: "one", "user": "Alex", parent: "root", child: [ { _id: 2, type: "comment", title: "two", "user": "Serg", parent: 1 }, { _id: 3, type: "comment", title: "two", "user": "Andrey", parent: 1 } ] } 

, , , _id :

 def count(cls): db = connect() ctr = db.test.find({'type':'goods', 'class':cls}).count() childs = db.test.find_one({'_id':cls}) for res in childs['child']: ctr += count(res) return ctr 

, _id , :

 def path( id ): p = [] parent = db.test.find_one( {"_id": id }, { "parent": 1, "alias":1, "title":1}) else: path.append( ( parent['alias'], parent['title'] ) ) p += path( parent['parent'] ) return p print ( path("123") ) >>>[ ("one", " "), ("two", " ") ] 

,

, { tags : { ru: ", " } } , . { tags : [ "", "" ] } .
, .
, , , :

 { _id: 1, title: " ", tags: [ "php", "python" ] } db.test.find({ tags: { $in: ["php", "python" ] } } ) 

:

• , , `python, javascript`.
• , , , , «».
• , , , .
• .
• , .
• , , , , 5 .
• , `+2` .
• .
• `10` .
• , , ( ).

 dt = ( datetime.today() + timedelta( days = -5 ) ) db.test.aggregate([ { $match: { //       type: "news", date: { $gt: dt }, //   . vate: { $gte: 2 }, //   . user: { $in: [ "alex", "pavel" ] } $and: [ //      . { pub: true }, { accept: true }, //       . { tags: { $in: ["php", "python" ] } } , //        . { tags: { $nin: [""] } } ] }, //      . { $sort: {'primary': -1, view: -1}}, //   ,    5 { $limit:3}, //    1    ,   . { $group: { '_id':'$user', 'id': {'$first':'$_id'}, 'type':{'$first':'$type'}, 'title': {'$first':'$title'}, 'content':{'$first':'$content'}, 'count':{'$first':'$count_comm'}, 'last_comm':{'$first':'$last_comm'}, 'vote':{'$first':'$vote'}, 'tags':{'$first':'$tags'} } }, //       . { $project :{ '_id':'$id', 'title':1, 'content':1, 'type':1, 'count':1, 'last_comm':1, 'tags':1, 'vote':1 } } ]) 

 db.test.aggregate([ { $match: { type: "news", date: { $gt: dt }, vate: { $gte: 2 }, user: { $in: [ "alex", "pavel" ] } $and: [ { pub: true }, { accept: true }, { tags: { $in: ["php", "python" ] } } , { tags: { $nin: [""] } } ] }, { $sort: {'primary': -1, view: -1}}, { $limit:3}, { $group: {'_id':'$user', 'id': {'$first':'$_id'}, 'type':{'$first':'$type'}, 'title': {'$first':'$title'}, 'content':{'$first':'$content'}, 'count':{'$first':'$count_comm'}, 'last_comm':{'$first':'$last_comm'}, 'vote':{'$first':'$vote'}, 'tags':{'$first':'$tags'} } }, { $project :{ '_id':'$id', 'title':1, 'content':1, 'type':1, 'count':1, 'last_comm':1, 'tags':1, 'vote':1 } } ]) 

, — .

-commerce Mongodb

, - — - . , , , , , , .

, - , . .

, , , -.

1. ( )
2. .

, , , — . , , .

().

:

1. , .
2. .

, , . :

 cursor = db.test.find({ "type": "filters", "category": "id_category" }) 

, , , , .

=>
=> 15.6 , 17 .

, . . ., .
— , , — .

. , .

, , . .

, . _id .
$addToSet .

 db.test.aggregate([ //         id  { '$match': { type : "goods_attr", category: id_category } }, //       ,    { '$project': { "title" : "$title.ru", 'value': "$attr.ru", 'category': "$category", '_id': 0 } }, { '$group' : { '_id': { 'category' :"$category", 'title': "$title"} , 'filters': { '$addToSet': "$value" } } }, { '$group' : { '_id' :"$_id.category", 'title':{ '$addToSet': { 'title': "$_id.title", 'filters': "$filters" } } } } ]) 

group :

 ...{ '$group' : { '_id': { 'category' :"$category", 'title': "$title"} , 'filters': { '$addToSet': "$value" } } }.... { "_id": { "category": "id", "title": "   " }, "filters": [ "" ] } { "_id" : { "category" : "id", "title" : "" }, "filters" : [ " " ] } { "_id" : { "category" : "id", "title" : " " }, "filters" : [ "" ] } 

group .

 ...{ '$group' : { '_id':"$_id.category", 'title':{'$addToSet': {'title': "$_id.title", 'filters': "$filters" }} } ...} { "_id" : "id_category", "title" : [ { "title" : "   ", "filters" : [ "  ", "  ", " ", "  " ] }, { "title" : " ", "filters" : [ "" ] }, { "title" : "", "filters" : [ " " ] }, { "title" : "   ", "filters" : [ "" ] } ] } 

, , . , , owner_id _id .

 db.test.aggregate([ { '$match' : { 'type' : "goods_attr", "category'':"id", '$or': [ {'title': '', 'attr': ' '}, {'title': '   ', 'attr_val': '  '} ] } }, { '$group': {'_id': "$owner_id", "attr": { '$push': "$title" }}}, { '$match': {"attr": {'$all': [ '', '   ' ] }}}, { '$project': {"_id":1 } } ]) 

_id .

Python

.

, , , ( — ) . , , .

ERP . , , - . , , .

python , , .

exec

 src = ''' result = 0 for i in xrange(100): result += i ''' assert '__' not in src, 'Prohibited to use symbols "__"' pr = compile(src, '<string>', mode='exec') glob = { '__builtins__':{ 'xrange':xrange } } exec(pr, glob) print glob['result'] 

eval ( )

 src = 'max(5,7,3)' glob = { '__builtins__':{ 'max':max } } assert '__' not in src, 'Prohibited to use symbols "__"' print ( eval(src, glob) ) 

— :
assert '__' not in src, 'Prohibited to use symbols "__"'

• , : __class__, __base__
  python
  

   glob = { '__builtins__':{ 'xrange':xrange } } 

  
  

  (import, type...) / .