RESTful API on Node.js + MongoDB

As a mobile application developer, I often need backend services for storing user data, authorization, and more. Of course, you can use BaaS (Parse, Backendless, etc ...) for such tasks. But its solution is always more convenient and practical.

And I nevertheless decided to study technologies completely unknown to me, which are now quite popular and positioned as easily mastered by beginners and not requiring in-depth knowledge and experience for implementing large-scale projects. So let's check together whether a non-specialist can write his effective and correct backend.

This article will discuss building a REST API for a mobile application on Node.js using the Express.js framework and the Mongoose.js module for working with MongoDB. To control access, let's use OAuth 2.0 technology using OAuth2orize and Passport.js modules.
')
I am writing from the perspective of an absolute beginner. Glad to any feedback and amendments on the code and logic!

Content

1. Node.js + Express.js, a simple web server
2. Error handling
3. RESTful API endpoints CRUD
4. MongoDB & Mongoose.js
5. Access control - OAuth 2.0, Passport.js

I work in OSX, IDE - JetBrains WebStorm .

Basics Node.js gathered in the screencasts of Ilya Kantor , highly recommend! ( And here is a post about them in Habré )

The finished project at the last stage can be taken on GitHub . To install all modules, run the npm install command in the project folder.

1. Node.js + Express.js, a simple web server

Node.js has non-blocking I / O, which is cool for an API that many clients will call on. Express.js is a developed, lightweight framework that allows you to quickly describe all the paths (API endpoints) that we will process. You can also find many useful modules for it.

Create a new project with a single server.js file. Since the application will rely entirely on Express.js , install it. Installation of third-party modules occurs through the Node Package Manager by running the npm install modulename in the project folder.

 cd NodeAPI npm i express 

Express will be installed in the node_modules folder. Connect it to the application:

 var express = require('express'); var app = express(); app.listen(1337, function(){ console.log('Express server listening on port 1337'); }); 

Run the application through the IDE or console ( node server.js ). This code will create a web server on localhost: 1337. If you try to open it - it will display the message Cannot GET / . This is because we have not yet specified a single route (route). Next, create several paths and make the basic Express settings.

 var express = require('express'); var path = require('path'); //     var app = express(); app.use(express.favicon()); //   ,      app.use(express.logger('dev')); //        app.use(express.bodyParser()); //  ,   JSON   app.use(express.methodOverride()); //  put  delete app.use(app.router); //       app.use(express.static(path.join(__dirname, "public"))); //    ,     public/ (    index.html) app.get('/api', function (req, res) { res.send('API is running'); }); app.listen(1337, function(){ console.log('Express server listening on port 1337'); }); 

Now localhost: 1337 / api will return our message. localhost: 1337 will display index.html.

Here we turn to error handling.

2. Error handling

At first we will connect the convenient module for logging Winston . We will use it through our wrapper. Install npm i winston in the project root, then create the libs / folder and log.js file there.

 var winston = require('winston'); function getLogger(module) { var path = module.filename.split('/').slice(-2).join('/'); //    ,    return new winston.Logger({ transports : [ new winston.transports.Console({ colorize: true, level: 'debug', label: path }) ] }); } module.exports = getLogger; 

We created 1 transport for logs - to the console. We can also separately sort and save messages, for example, in a database or file. Let's connect a logger to our server.js.

 var express = require('express'); var path = require('path'); //     var log = require('./libs/log')(module); var app = express(); app.use(express.favicon()); //   ,      app.use(express.logger('dev')); //        app.use(express.bodyParser()); //  ,   JSON   app.use(express.methodOverride()); //  put  delete app.use(app.router); //       app.use(express.static(path.join(__dirname, "public"))); //    ,     public/ (    index.html) app.get('/api', function (req, res) { res.send('API is running'); }); app.listen(1337, function(){ log.info('Express server listening on port 1337'); }); 

Our informational message is now beautifully separately displayed in the console. Add error handling 404 and 500.

 app.use(function(req, res, next){ res.status(404); log.debug('Not found URL: %s',req.url); res.send({ error: 'Not found' }); return; }); app.use(function(err, req, res, next){ res.status(err.status || 500); log.error('Internal error(%d): %s',res.statusCode,err.message); res.send({ error: err.message }); return; }); app.get('/ErrorExample', function(req, res, next){ next(new Error('Random error!')); }); 

Now, if there are no paths available, Express will return our message. If an internal application error occurs, our handler will also work; you can check this by contacting localhost: 1337 / ErrorExample.

3. RESTful API endpoints, CRUD

Add paths for processing certain "articles" (articles). On Habré there is an excellent article explaining how to make a convenient API correctly. We will not fill them with logic yet, we will do it in the next step, with the connection of the database.

 app.get('/api/articles', function(req, res) { res.send('This is not implemented now'); }); app.post('/api/articles', function(req, res) { res.send('This is not implemented now'); }); app.get('/api/articles/:id', function(req, res) { res.send('This is not implemented now'); }); app.put('/api/articles/:id', function (req, res){ res.send('This is not implemented now'); }); app.delete('/api/articles/:id', function (req, res){ res.send('This is not implemented now'); }); 

For testing post / put / delete I would recommend a great wrapper over cURL - httpie . Further I will give examples of requests using this tool.

4. MongoDB & Mongoose.js

Choosing a DBMS, I was again guided by the desire to learn something new. MongoDB is the most popular NoSQL document-oriented DBMS. Mongoose.js is a wrapper that allows you to create convenient and functional document schemes.

Download and install MongoDB . Install mongoose: npm i mongoose . I allocated work with DB to the file libs / mongoose.js.

 var mongoose = require('mongoose'); var log = require('./log')(module); mongoose.connect('mongodb://localhost/test1'); var db = mongoose.connection; db.on('error', function (err) { log.error('connection error:', err.message); }); db.once('open', function callback () { log.info("Connected to DB!"); }); var Schema = mongoose.Schema; // Schemas var Images = new Schema({ kind: { type: String, enum: ['thumbnail', 'detail'], required: true }, url: { type: String, required: true } }); var Article = new Schema({ title: { type: String, required: true }, author: { type: String, required: true }, description: { type: String, required: true }, images: [Images], modified: { type: Date, default: Date.now } }); // validation Article.path('title').validate(function (v) { return v.length > 5 && v.length < 70; }); var ArticleModel = mongoose.model('Article', Article); module.exports.ArticleModel = ArticleModel; 

In this file, a connection is made to the database, as well as object schemas are declared. Articles will contain image objects. A variety of complex validations can also be described here.

At this stage, I suggest connecting the nconf module to store the path to the database in it. Also in the config save the port on which the server is created. The module is installed using the npm i nconf . The wrapper is libs / config.js

 var nconf = require('nconf'); nconf.argv() .env() .file({ file: './config.json' }); module.exports = nconf; 

It follows that we need to create config.json in the root of the project.

 { "port" : 1337, "mongoose": { "uri": "mongodb://localhost/test1" } } 

Changes mongoose.js (only in the header):

 var config = require('./config'); mongoose.connect(config.get('mongoose:uri')); 

Server.js changes:

 var config = require('./libs/config'); app.listen(config.get('port'), function(){ log.info('Express server listening on port ' + config.get('port')); }); 

Now add CRUD actions to our existing paths.

 var log = require('./libs/log')(module); var ArticleModel = require('./libs/mongoose').ArticleModel; app.get('/api/articles', function(req, res) { return ArticleModel.find(function (err, articles) { if (!err) { return res.send(articles); } else { res.statusCode = 500; log.error('Internal error(%d): %s',res.statusCode,err.message); return res.send({ error: 'Server error' }); } }); }); app.post('/api/articles', function(req, res) { var article = new ArticleModel({ title: req.body.title, author: req.body.author, description: req.body.description, images: req.body.images }); article.save(function (err) { if (!err) { log.info("article created"); return res.send({ status: 'OK', article:article }); } else { console.log(err); if(err.name == 'ValidationError') { res.statusCode = 400; res.send({ error: 'Validation error' }); } else { res.statusCode = 500; res.send({ error: 'Server error' }); } log.error('Internal error(%d): %s',res.statusCode,err.message); } }); }); app.get('/api/articles/:id', function(req, res) { return ArticleModel.findById(req.params.id, function (err, article) { if(!article) { res.statusCode = 404; return res.send({ error: 'Not found' }); } if (!err) { return res.send({ status: 'OK', article:article }); } else { res.statusCode = 500; log.error('Internal error(%d): %s',res.statusCode,err.message); return res.send({ error: 'Server error' }); } }); }); app.put('/api/articles/:id', function (req, res){ return ArticleModel.findById(req.params.id, function (err, article) { if(!article) { res.statusCode = 404; return res.send({ error: 'Not found' }); } article.title = req.body.title; article.description = req.body.description; article.author = req.body.author; article.images = req.body.images; return article.save(function (err) { if (!err) { log.info("article updated"); return res.send({ status: 'OK', article:article }); } else { if(err.name == 'ValidationError') { res.statusCode = 400; res.send({ error: 'Validation error' }); } else { res.statusCode = 500; res.send({ error: 'Server error' }); } log.error('Internal error(%d): %s',res.statusCode,err.message); } }); }); }); app.delete('/api/articles/:id', function (req, res){ return ArticleModel.findById(req.params.id, function (err, article) { if(!article) { res.statusCode = 404; return res.send({ error: 'Not found' }); } return article.remove(function (err) { if (!err) { log.info("article removed"); return res.send({ status: 'OK' }); } else { res.statusCode = 500; log.error('Internal error(%d): %s',res.statusCode,err.message); return res.send({ error: 'Server error' }); } }); }); }); 

Thanks to the Mongoose and the described schemes - all operations are very clear. Now, besides node.js, you should run mongoDB with the mongod command. mongo - utility for working with the database, the service itself - mongod . Create previously in the database do not need anything.

Sample requests using httpie:

 http POST http://localhost:1337/api/articles title=TestArticle author='John Doe' description='lorem ipsum dolar sit amet' images:='[{"kind":"thumbnail", "url":"http://habrahabr.ru/images/write-topic.png"}, {"kind":"detail", "url":"http://habrahabr.ru/images/write-topic.png"}]' http http://localhost:1337/api/articles http http://localhost:1337/api/articles/52306b6a0df1064e9d000003 http PUT http://localhost:1337/api/articles/52306b6a0df1064e9d000003 title=TestArticle2 author='John Doe' description='lorem ipsum dolar sit amet' images:='[{"kind":"thumbnail", "url":"http://habrahabr.ru/images/write-topic.png"}, {"kind":"detail", "url":"http://habrahabr.ru/images/write-topic.png"}]' http DELETE http://localhost:1337/api/articles/52306b6a0df1064e9d000003 

The project at this stage can take a look at GitHub .

5. Access control - OAuth 2.0, Passport.js

For access control, I will resort to OAuth 2. It may be redundant, but later this approach facilitates integration with other OAuth providers. In addition, I did not find any working examples of user-password OAuth2 flow for Node.js.
Passport.js will monitor access control directly. For the OAuth2 server, a solution from the same author, oauth2orize, is useful . Users, tokens will be stored in MongoDB.
First you need to install all the modules that we need:

• Faker
• oauth2orize
• passport
• passport-http
• passport-http-bearer
• passport-oauth2-client-password

Then, in mongoose.js, you need to add schemas for users and tokens:

 var crypto = require('crypto'); // User var User = new Schema({ username: { type: String, unique: true, required: true }, hashedPassword: { type: String, required: true }, salt: { type: String, required: true }, created: { type: Date, default: Date.now } }); User.methods.encryptPassword = function(password) { return crypto.createHmac('sha1', this.salt).update(password).digest('hex'); //more secure - return crypto.pbkdf2Sync(password, this.salt, 10000, 512); }; User.virtual('userId') .get(function () { return this.id; }); User.virtual('password') .set(function(password) { this._plainPassword = password; this.salt = crypto.randomBytes(32).toString('base64'); //more secure - this.salt = crypto.randomBytes(128).toString('base64'); this.hashedPassword = this.encryptPassword(password); }) .get(function() { return this._plainPassword; }); User.methods.checkPassword = function(password) { return this.encryptPassword(password) === this.hashedPassword; }; var UserModel = mongoose.model('User', User); // Client var Client = new Schema({ name: { type: String, unique: true, required: true }, clientId: { type: String, unique: true, required: true }, clientSecret: { type: String, required: true } }); var ClientModel = mongoose.model('Client', Client); // AccessToken var AccessToken = new Schema({ userId: { type: String, required: true }, clientId: { type: String, required: true }, token: { type: String, unique: true, required: true }, created: { type: Date, default: Date.now } }); var AccessTokenModel = mongoose.model('AccessToken', AccessToken); // RefreshToken var RefreshToken = new Schema({ userId: { type: String, required: true }, clientId: { type: String, required: true }, token: { type: String, unique: true, required: true }, created: { type: Date, default: Date.now } }); var RefreshTokenModel = mongoose.model('RefreshToken', RefreshToken); module.exports.UserModel = UserModel; module.exports.ClientModel = ClientModel; module.exports.AccessTokenModel = AccessTokenModel; module.exports.RefreshTokenModel = RefreshTokenModel; 

The password virtual property is an example of how mongoose can incorporate convenient logic into models. About hashes, algorithms and salt - not this article, we will not go into the details of implementation.

So, the objects in the database:

1. User - a user who has the name, password hash and salt of his password.
2. Client is a client application that is granted access on behalf of the user. Have a name and secret code.
3. AccessToken - a token (type bearer), issued to client applications, is limited in time.
4. RefreshToken is another type of token that allows you to request a new bearer-token without re-requesting a password from the user.

In the config.json add the token lifetime:

 { "port" : 1337, "security": { "tokenLife" : 3600 }, "mongoose": { "uri": "mongodb://localhost/testAPI" } } 

Let's separate into separate modules the OAuth2 server and authorization logic. In oauth.js, the “strategies” of passport.js are described, we include 3 of them - 2 on OAuth2 username-password flow, 1 for token checking.

 var config = require('./config'); var passport = require('passport'); var BasicStrategy = require('passport-http').BasicStrategy; var ClientPasswordStrategy = require('passport-oauth2-client-password').Strategy; var BearerStrategy = require('passport-http-bearer').Strategy; var UserModel = require('./mongoose').UserModel; var ClientModel = require('./mongoose').ClientModel; var AccessTokenModel = require('./mongoose').AccessTokenModel; var RefreshTokenModel = require('./mongoose').RefreshTokenModel; passport.use(new BasicStrategy( function(username, password, done) { ClientModel.findOne({ clientId: username }, function(err, client) { if (err) { return done(err); } if (!client) { return done(null, false); } if (client.clientSecret != password) { return done(null, false); } return done(null, client); }); } )); passport.use(new ClientPasswordStrategy( function(clientId, clientSecret, done) { ClientModel.findOne({ clientId: clientId }, function(err, client) { if (err) { return done(err); } if (!client) { return done(null, false); } if (client.clientSecret != clientSecret) { return done(null, false); } return done(null, client); }); } )); passport.use(new BearerStrategy( function(accessToken, done) { AccessTokenModel.findOne({ token: accessToken }, function(err, token) { if (err) { return done(err); } if (!token) { return done(null, false); } if( Math.round((Date.now()-token.created)/1000) > config.get('security:tokenLife') ) { AccessTokenModel.remove({ token: accessToken }, function (err) { if (err) return done(err); }); return done(null, false, { message: 'Token expired' }); } UserModel.findById(token.userId, function(err, user) { if (err) { return done(err); } if (!user) { return done(null, false, { message: 'Unknown user' }); } var info = { scope: '*' } done(null, user, info); }); }); } )); 

For the issuance and update of the token is responsible oauth2.js. One exchange-strategy is to receive a token via username-password flow, another one is to exchange a refresh_token.

 var oauth2orize = require('oauth2orize'); var passport = require('passport'); var crypto = require('crypto'); var config = require('./config'); var UserModel = require('./mongoose').UserModel; var ClientModel = require('./mongoose').ClientModel; var AccessTokenModel = require('./mongoose').AccessTokenModel; var RefreshTokenModel = require('./mongoose').RefreshTokenModel; // create OAuth 2.0 server var server = oauth2orize.createServer(); // Exchange username & password for access token. server.exchange(oauth2orize.exchange.password(function(client, username, password, scope, done) { UserModel.findOne({ username: username }, function(err, user) { if (err) { return done(err); } if (!user) { return done(null, false); } if (!user.checkPassword(password)) { return done(null, false); } RefreshTokenModel.remove({ userId: user.userId, clientId: client.clientId }, function (err) { if (err) return done(err); }); AccessTokenModel.remove({ userId: user.userId, clientId: client.clientId }, function (err) { if (err) return done(err); }); var tokenValue = crypto.randomBytes(32).toString('base64'); var refreshTokenValue = crypto.randomBytes(32).toString('base64'); var token = new AccessTokenModel({ token: tokenValue, clientId: client.clientId, userId: user.userId }); var refreshToken = new RefreshTokenModel({ token: refreshTokenValue, clientId: client.clientId, userId: user.userId }); refreshToken.save(function (err) { if (err) { return done(err); } }); var info = { scope: '*' } token.save(function (err, token) { if (err) { return done(err); } done(null, tokenValue, refreshTokenValue, { 'expires_in': config.get('security:tokenLife') }); }); }); })); // Exchange refreshToken for access token. server.exchange(oauth2orize.exchange.refreshToken(function(client, refreshToken, scope, done) { RefreshTokenModel.findOne({ token: refreshToken }, function(err, token) { if (err) { return done(err); } if (!token) { return done(null, false); } if (!token) { return done(null, false); } UserModel.findById(token.userId, function(err, user) { if (err) { return done(err); } if (!user) { return done(null, false); } RefreshTokenModel.remove({ userId: user.userId, clientId: client.clientId }, function (err) { if (err) return done(err); }); AccessTokenModel.remove({ userId: user.userId, clientId: client.clientId }, function (err) { if (err) return done(err); }); var tokenValue = crypto.randomBytes(32).toString('base64'); var refreshTokenValue = crypto.randomBytes(32).toString('base64'); var token = new AccessTokenModel({ token: tokenValue, clientId: client.clientId, userId: user.userId }); var refreshToken = new RefreshTokenModel({ token: refreshTokenValue, clientId: client.clientId, userId: user.userId }); refreshToken.save(function (err) { if (err) { return done(err); } }); var info = { scope: '*' } token.save(function (err, token) { if (err) { return done(err); } done(null, tokenValue, refreshTokenValue, { 'expires_in': config.get('security:tokenLife') }); }); }); }); })); // token endpoint exports.token = [ passport.authenticate(['basic', 'oauth2-client-password'], { session: false }), server.token(), server.errorHandler() ] 

To connect these modules, add to server.js:

 var oauth2 = require('./libs/oauth2'); app.use(passport.initialize()); require('./libs/auth'); app.post('/oauth/token', oauth2.token); app.get('/api/userInfo', passport.authenticate('bearer', { session: false }), function(req, res) { // req.authInfo is set using the `info` argument supplied by // `BearerStrategy`. It is typically used to indicate scope of the token, // and used in access control checks. For illustrative purposes, this // example simply returns the scope in the response. res.json({ user_id: req.user.userId, name: req.user.username, scope: req.authInfo.scope }) } ); 

For example, the protection is on the address localhost: 1337 / api / userInfo.

To check the operation of the authorization mechanism, you must create a user and a client in the database. I will give the application on Node.js, which will create the necessary objects and remove unnecessary from the collections. It helps to quickly clear the database of tokens and users when testing, I think one launch will be enough :)

 var log = require('./libs/log')(module); var mongoose = require('./libs/mongoose').mongoose; var UserModel = require('./libs/mongoose').UserModel; var ClientModel = require('./libs/mongoose').ClientModel; var AccessTokenModel = require('./libs/mongoose').AccessTokenModel; var RefreshTokenModel = require('./libs/mongoose').RefreshTokenModel; var faker = require('Faker'); UserModel.remove({}, function(err) { var user = new UserModel({ username: "andrey", password: "simplepassword" }); user.save(function(err, user) { if(err) return log.error(err); else log.info("New user - %s:%s",user.username,user.password); }); for(i=0; i<4; i++) { var user = new UserModel({ username: faker.random.first_name().toLowerCase(), password: faker.Lorem.words(1)[0] }); user.save(function(err, user) { if(err) return log.error(err); else log.info("New user - %s:%s",user.username,user.password); }); } }); ClientModel.remove({}, function(err) { var client = new ClientModel({ name: "OurService iOS client v1", clientId: "mobileV1", clientSecret:"abc123456" }); client.save(function(err, client) { if(err) return log.error(err); else log.info("New client - %s:%s",client.clientId,client.clientSecret); }); }); AccessTokenModel.remove({}, function (err) { if (err) return log.error(err); }); RefreshTokenModel.remove({}, function (err) { if (err) return log.error(err); }); setTimeout(function() { mongoose.disconnect(); }, 3000); 

If you created the data with a script, the following commands for authorization will also work for you. Let me remind you that I use httpie .

 http POST http://localhost:1337/oauth/token grant_type=password client_id=mobileV1 client_secret=abc123456 username=andrey password=simplepassword http POST http://localhost:1337/oauth/token grant_type=refresh_token client_id=mobileV1 client_secret=abc123456 refresh_token=TOKEN http http://localhost:1337/api/userinfo Authorization:'Bearer TOKEN' 

Attention! On a production server, be sure to use HTTPS, which is implied by the OAuth 2 specification. And do not forget about the correct password hashing. Implementing https in this example is easy, there are many examples on the net.
Let me remind you that all the code is contained in the repository on GitHub .
To work, you need to run npm install in the directory, run mongod , node dataGen.js (wait for execution), and then node server.js .

If some part of the article is worth describing in more detail, please indicate this in the comments. The material will be processed and supplemented as feedback is received.

To summarize, I want to say that node.js is a cool, convenient server solution. Document-based MongoDB is a very unusual, but undoubtedly useful tool, most of which I haven't used yet. Around Node.js is a very large community where there are a lot of open-source developments. For example, the creator of oauth2orize and passport.js - Jared Hanson has made great projects that make it as easy as possible to implement properly protected systems.