The story of one small experiment with Node.js
That's what can happen if you make an application on unfamiliar technologies with a limited time.
This article will be interesting most likely to beginners well, or just for fun. Be careful the article is full of subjective assessments and the stupid reasoning of the author.
It all started with this article and a longtime desire to get acquainted with Node.js. It was impossible to miss such a coincidence of circumstances). What came out of this you will find under the cut.
')
So, let's begin. Task
It was decided to make a small and fluffy web application for hosting formulas. Well, that is, you come, enter the formula, press the button, get a link to the render of the formula and rejoice. Formulas will be introduced in MathML. Well, it would be cool to be able to find the formula of interest to us and get its render or MathML. It is important to reduce resource consumption to a minimum.
Task defined, further architecture
As a result, the following flowed from the above:
Further, a little more about everything
Client / UI / Browser
The client is already pretty standard AngularJS. In my opinion, while there is nothing better for resolving the binding and building the interface, in my opinion this is the main problem when writing web applications and an anguar with it copes with 5 plus. In addition to the angular client, I used Typescript, Jade, and Sass.
Icons we take Font Awesome, a palette from google material here and while we do without jQuery.
Server
Everything is much worse than it might seem at first glance. On the node, I never wrote anything and this is my first pancake, which turned out to be not quite a lump.
The first thing that surprised me was the number of different libraries. Of course this is not java, but for the task I have found, all I could need
intel is a fairly convenient and functional library for logging in a node
config - a simple way to use configuration files
mongoose is a kind of hibernate between node and monga
express is a web framework for the node. Simplifies the already not complicated process of creating a web server on a node
body-parser - POST request handling
Agenda - job scheduling
Libxmljs - working with XML, which is important without JVM in dependencies
The quality of these modules is sometimes not as desired.) For example, libxmljs ... In Ubunt, it is not set up without a tambourine like that
On the server side, I also used Typescript for exactly the same reasons as on the client. But on this side some problems arose ... If the client managed to find an approach in which the code did not become complicated, then in the node the use of Typescript complicated the code a little, but not so much to give up on it.
mongoose
This is an interesting library that allows you to create data schemas, automatically validate, save and read data from a monitor in accordance with the scheme. This is a very useful and convenient thing. Mongoose makes it easy to work with Monga from the node to the obscene ...
For example, this is how you can save a new formula to the database
Or find a formula
Agenda
The queue is an integral part of algorithms similar to the developed formula conversion algorithm due to performance problems. A large stream of conversion requests can kill our application very quickly. The most logical and easiest way is to turn.
It is worth noting that for queuing there are several libraries for Node.js. I chose Agenda for the following reasons:
Add a job to the queue
Register a job handler
Libxmljs
It is important and even critical to verify the solution data in the request. To validate the data, I managed to google sufficiently interesting express-validator and validator.js libraries that allow performing the same validation quite effectively. But I needed to check MathML and for this I met two libraries xsd-schema-validator and Libxmljs . The first one is using Java / SAX, which means you need to have a JVM and something hasn't worked out with it since the first minutes, so as a result I used Libxmljs. No java is needed for her work, but just as she brought bugs to which I spent quite a lot of time.
For the MathML validation, I used the schemas of the second version, which I got here .
Something like this validation
Rendering
For rendering formulas we use MathJax in conjunction with PhantomJS. Not the fastest and stable option, but it turned out to be implemented quickly enough that under conditions of limited time is an undeniable advantage;).
MongoDB
To store the formulas used MongoDB. Hosting formulas require quick access to them. For this reason, Monge stores not only MathML but also the rendering of formula formulas (not due to quick access, but because they need to be stored somewhere). Pictures take not much space and just fit into the recommendations to store them as a field in the database without using additional libraries and frameworks such as GridFS. On the other hand, storing rendered images in Mong gives all the advantages of storing data in a database and not stupid on the file system.
Data schema
Build project
To build the project, I used Gulp and it was the first and 'strange' experience ... I was pleased with the quantity and quality of plug-ins for it. The first time I wrote a script for Gulp after Grunt, I got a script that lived my life, it did not do what I expected. The reason for all of my brain is not parallel oriented after Grunt. After realizing the errors, I started reworking the build script, but it was a bit late and the work is not yet complete. Now the build is performed by a bash script that calls a couple of Gulp tasks. It is not beautiful at all but it works and the alteration is almost complete. Now the problem is mainly with the compilation of the typing script, with the chosen approach (compilation into just one file) it is not possible to catch the end of the compilation process. Here I will be glad to help.
Application Hosting
I placed the app on two microinstans in a Google cloud using a generous $ 300 gift for experiments. How come we can. Well, again, the double sensation of using the cloud. Firstly, it is very, very cool and I was very pleased with the use of cloud technologies (at least at this stage). And, of course, in a barrel of honey in any way without a fly in the ointment. The problem is that I did not manage to write an adequate deployment script that would install the application to the instance in the cloud along with the installation of the node modules. The procedure works fine when using Debian on a local virtual machine, but in Debian, when accessing files in the cloud, an error suggested when creating an instance in the cloud, new files are created during this procedure from the user and not from the root, even if the command is executed on his behalf. Perhaps the problem is in crooked hands, but now the installation is done in two stages. On the first we create all the modules on behalf of the user and on the second we perform the installation from root. In general, the cloud is good, especially if it is a Google cloud;)
Well, so that life does not seem like honey, choose a domain in the RF zone. There was no particular problem. The only thing is that in the Nginx settings you need to specify the punycode name and not put Russian letters there.
For my nginx it looks like this
Application and JSON
The result is a very interesting system in which the data is represented by JSON objects at all stages from the interface in the browser, continuing with the node and ending with the database. This is my first experience with such a system, and to be honest, I got some kind of pleasure from it. This is a trifle when there are no data transformations in your application, but for me personally this trifle gives some transparency, which makes me personally good :).
Well, what happened in the end
And it turned out a simple web application that allows you to host formulas. With relatively low iron resources (two microinsts of the Google cloud), we managed to make a rather simple and load-resistant application.
In general, I was pleased with the technologies described above. I was pleasantly surprised by the quantity and quality of libraries for the node and the ease of writing code. Unpleasant sediment left a couple of bugs in the libraries of the node. Sometimes it even seemed all terribly damp and unfinished. But the mistakes were solved and did not take away too much time. By the way, I spent less time analyzing and correcting errors (including one revision in the source code) than when setting up and understanding some of the libraries in Java. Do not throw poop I perfectly understand the difference in the level of the node and java, I'm only talking about feelings.
Result
Code
I will be glad to constructive criticism and just reviews.
This article will be interesting most likely to beginners well, or just for fun. Be careful the article is full of subjective assessments and the stupid reasoning of the author.
It all started with this article and a longtime desire to get acquainted with Node.js. It was impossible to miss such a coincidence of circumstances). What came out of this you will find under the cut.
')
So, let's begin. Task
It was decided to make a small and fluffy web application for hosting formulas. Well, that is, you come, enter the formula, press the button, get a link to the render of the formula and rejoice. Formulas will be introduced in MathML. Well, it would be cool to be able to find the formula of interest to us and get its render or MathML. It is important to reduce resource consumption to a minimum.
Task defined, further architecture
As a result, the following flowed from the above:
- The application is strictly divided into two parts: the client and the server.
- Client communication with the server via “almost” :) RESTfull
- On the Node.js server
- As a MongoDB database
Further, a little more about everything
Client / UI / Browser
The client is already pretty standard AngularJS. In my opinion, while there is nothing better for resolving the binding and building the interface, in my opinion this is the main problem when writing web applications and an anguar with it copes with 5 plus. In addition to the angular client, I used Typescript, Jade, and Sass.
- Typescript - the possibilities are relatively easier to break the code into modules and, of course, to write typed code and use human classes will not replace perverts like all kinds of JS.
- Jade - beautiful, modules, features, minimalism.
- Sass - due to the possibility of breaking the code for modules, variables, etc. little things that make my life a little more pleasant.
Icons we take Font Awesome, a palette from google material here and while we do without jQuery.
Server
Everything is much worse than it might seem at first glance. On the node, I never wrote anything and this is my first pancake, which turned out to be not quite a lump.
The first thing that surprised me was the number of different libraries. Of course this is not java, but for the task I have found, all I could need
intel is a fairly convenient and functional library for logging in a node
config - a simple way to use configuration files
mongoose is a kind of hibernate between node and monga
express is a web framework for the node. Simplifies the already not complicated process of creating a web server on a node
body-parser - POST request handling
Agenda - job scheduling
Libxmljs - working with XML, which is important without JVM in dependencies
The quality of these modules is sometimes not as desired.) For example, libxmljs ... In Ubunt, it is not set up without a tambourine like that
cp build/Release/lib.target/xmljs.node build/xmljs.node On the server side, I also used Typescript for exactly the same reasons as on the client. But on this side some problems arose ... If the client managed to find an approach in which the code did not become complicated, then in the node the use of Typescript complicated the code a little, but not so much to give up on it.
mongoose
This is an interesting library that allows you to create data schemas, automatically validate, save and read data from a monitor in accordance with the scheme. This is a very useful and convenient thing. Mongoose makes it easy to work with Monga from the node to the obscene ...
For example, this is how you can save a new formula to the database
var formula: any = new entity.Formula(); formula.name = req.body.name; formula.description = req.body.descr; formula.mathml = req.body.data; formula.save(); Or find a formula
entity.Formula.find( {_id: req.params.id} ) .select(bdKey) .exec( 'find', (err, formula): void => { if ( err || (formula.length !== 1) ) { res.send( 404 ); } else { res.header("Content-Type", "image/png"); res.send(formula[0][bdKey], {}, function (err) {}); } }); Agenda
The queue is an integral part of algorithms similar to the developed formula conversion algorithm due to performance problems. A large stream of conversion requests can kill our application very quickly. The most logical and easiest way is to turn.
It is worth noting that for queuing there are several libraries for Node.js. I chose Agenda for the following reasons:
- It is possible to save tasks in MongoDB
- The library allows you to organize not only the queue in its usual sense, but also the execution of why on a schedule, which is a very nice addition.
Add a job to the queue
service.agenda.now( 'process new formula', {fid: fid} ); Register a job handler
service.agenda.define('process new formula', (job, done): void => { var data = job.attrs.data; log.debug( "process new formula: " + data.fid ); AgendaService.processRendering( data.fid, done ); }); Libxmljs
It is important and even critical to verify the solution data in the request. To validate the data, I managed to google sufficiently interesting express-validator and validator.js libraries that allow performing the same validation quite effectively. But I needed to check MathML and for this I met two libraries xsd-schema-validator and Libxmljs . The first one is using Java / SAX, which means you need to have a JVM and something hasn't worked out with it since the first minutes, so as a result I used Libxmljs. No java is needed for her work, but just as she brought bugs to which I spent quite a lot of time.
For the MathML validation, I used the schemas of the second version, which I got here .
Something like this validation
fs.readFile('mathml2.xsd', {encoding:'utf-8'}, (err, data): void => { try { if (err) { process.chdir(cwd); fail(); return } var xsdDoc = libxmljs.parseXmlString(data); var xmlDoc = libxmljs.parseXmlString(xsd); var valid = xmlDoc.validate(xsdDoc); log.debug( "xsd validation" + valid ); process.chdir(cwd); if ( valid ) ok(); else fail(); } catch(e) { process.chdir(cwd); fail(); } }); Rendering
For rendering formulas we use MathJax in conjunction with PhantomJS. Not the fastest and stable option, but it turned out to be implemented quickly enough that under conditions of limited time is an undeniable advantage;).
MongoDB
To store the formulas used MongoDB. Hosting formulas require quick access to them. For this reason, Monge stores not only MathML but also the rendering of formula formulas (not due to quick access, but because they need to be stored somewhere). Pictures take not much space and just fit into the recommendations to store them as a field in the database without using additional libraries and frameworks such as GridFS. On the other hand, storing rendered images in Mong gives all the advantages of storing data in a database and not stupid on the file system.
Data schema
module entity.schema { export var IFormula: any = new mongoose.Schema({ name: {type: String, default: ''}, description: {type: String, default: ''}, created: {type: Date, default: Date.now}, modified: {type: Date, default: Date.now}, mathml: {type: String, default: ''}, png200: {type: Buffer}, png100: {type: Buffer}, png50: {type: Buffer}, png200t: {type: Buffer}, png100t: {type: Buffer}, png50t: {type: Buffer}, ready: {type: Boolean, default: false}, error: {type: Boolean, default: false} }); } Build project
To build the project, I used Gulp and it was the first and 'strange' experience ... I was pleased with the quantity and quality of plug-ins for it. The first time I wrote a script for Gulp after Grunt, I got a script that lived my life, it did not do what I expected. The reason for all of my brain is not parallel oriented after Grunt. After realizing the errors, I started reworking the build script, but it was a bit late and the work is not yet complete. Now the build is performed by a bash script that calls a couple of Gulp tasks. It is not beautiful at all but it works and the alteration is almost complete. Now the problem is mainly with the compilation of the typing script, with the chosen approach (compilation into just one file) it is not possible to catch the end of the compilation process. Here I will be glad to help.
Application Hosting
I placed the app on two microinstans in a Google cloud using a generous $ 300 gift for experiments. How come we can. Well, again, the double sensation of using the cloud. Firstly, it is very, very cool and I was very pleased with the use of cloud technologies (at least at this stage). And, of course, in a barrel of honey in any way without a fly in the ointment. The problem is that I did not manage to write an adequate deployment script that would install the application to the instance in the cloud along with the installation of the node modules. The procedure works fine when using Debian on a local virtual machine, but in Debian, when accessing files in the cloud, an error suggested when creating an instance in the cloud, new files are created during this procedure from the user and not from the root, even if the command is executed on his behalf. Perhaps the problem is in crooked hands, but now the installation is done in two stages. On the first we create all the modules on behalf of the user and on the second we perform the installation from root. In general, the cloud is good, especially if it is a Google cloud;)
Well, so that life does not seem like honey, choose a domain in the RF zone. There was no particular problem. The only thing is that in the Nginx settings you need to specify the punycode name and not put Russian letters there.
For my nginx it looks like this
listen 80; server_name xn--c1ajpkj4do.xn--p1ai; Application and JSON
The result is a very interesting system in which the data is represented by JSON objects at all stages from the interface in the browser, continuing with the node and ending with the database. This is my first experience with such a system, and to be honest, I got some kind of pleasure from it. This is a trifle when there are no data transformations in your application, but for me personally this trifle gives some transparency, which makes me personally good :).
Well, what happened in the end
And it turned out a simple web application that allows you to host formulas. With relatively low iron resources (two microinsts of the Google cloud), we managed to make a rather simple and load-resistant application.
In general, I was pleased with the technologies described above. I was pleasantly surprised by the quantity and quality of libraries for the node and the ease of writing code. Unpleasant sediment left a couple of bugs in the libraries of the node. Sometimes it even seemed all terribly damp and unfinished. But the mistakes were solved and did not take away too much time. By the way, I spent less time analyzing and correcting errors (including one revision in the source code) than when setting up and understanding some of the libraries in Java. Do not throw poop I perfectly understand the difference in the level of the node and java, I'm only talking about feelings.
Result
Code
I will be glad to constructive criticism and just reviews.
Source: https://habr.com/ru/post/246621/
All Articles