Geekly Articles each Day
📜 ⬆️ ⬇️

Node.js and JavaScript instead of the old web

Introduction


This article is about an experimental general technology stack. It does not just duplicate my report at the OdessaSJS 2016 conference, but contains everything that the report did not fit due to lack of time and the exceptional scale of the topic. I even re-recorded the report in the voice of the text of the article and you can listen to it, not read it. I already spoke on this topic at Wuhan University (China), and at the Kiev Polytechnic Institute I conducted a series of seminars in 2015-2016. The basic idea is that the problems of technology fragmentation can be solved by designing the entire technological stack, focusing on the data structures, syntax and protocol of the interaction of components. Most issues of incompatibility will disappear by itself. Even if this approach will be alternative and experimental, its task will be accomplished if it outlines the way and demonstrates the fundamental possibility of creating a simple and elegant general-purpose solution. This idea is a natural continuation of the Node.js approach when we reduce the number of languages ​​and technologies in the system by developing both the client and the server in JavaScript. Despite the experimentation, the JSTP protocol is already used in commercial products, for example, for interactive television by SinceTV, where it allows you to connect tens of millions of users simultaneously. This decision received a prize for innovation in the field of television at the international competition Golden Panda Awards 2015 in Chendu (China). There are implementations in the field of server cluster management, solutions are being prepared for medicine, interactive games, e-commerce and services.



Slides / Audio Version



Performance


Good afternoon, my name is Timur Shemsedinov, I am the architect of SinceTV, the author of the Impress Application Server for Node.js, a researcher at the Institute of System Technologies and a teacher at the KPI. Today's topic is not so much about the node, but rather about the development of the idea that formed the basis of the node.


Anyone who has developed applications not only for the web can easily compare the complexity of the technological stack, the volume of problems encountered and, ultimately, the speed of development, with projects of similar complexity for mobile devices or window applications for desktops, supercomputers or embedded hardware, programming for servers and industrial automation, data processing or other industries programming applications.


Every step on the web is a struggle, every minute is a pain, every line is a compromise or a patch. There are many articles on the Internet that chaos has swept through the old web and good experts spend weeks on the simplest tasks, such as moving the button 3 pixels up or catching a wandering bug in the zoo of browsers. For example, an article from the Medium “I'm a web developer and I've been stuck with the app for the last 10 days” translated in Habré “I’m a web developer and for 10 days I can’t write a simple application . ”


What is the problem? How can we identify and recognize it? Are there any other ways? And can we move in an evolutionary rather than revolutionary way, solving problems in parts? This will be discussed further, and in the whole series of reports and articles that I and our entire team are preparing, and to which I invite you to discuss.


The old web has really plunged into a state in which all decisions are temporary, there is nothing stable and living without corrections for more than a year. If you just leave the system unchanged, it will quickly become unusable. This is due to the many dependencies that have been developed at different times and by different people who have never imagined that these software modules will be used together. Everything in the old web was created by chance, without a single plan, and even the authors are not going to follow the standards. This is a global fragmentation of technology, otherwise it could not be, but now we are already seeing design errors, examples of success and analogues in the outside web, and I want to offer a fairly simple solution. This way will not break your whole life, but still radical enough to be perceived without shocks.


Imagine that we give up HTTP, HTML, CSS, DOM, URL, XMLHttpRequest, AJAX, JSON, JSONP, XML, REST, Server-Sent Events, CORS, Cookie, MIME, WebSocket, localStorage, indexedDB, WebSQL, and everything else except for one thing, we’ll leave JavaScript and make absolutely everything that is necessary for developing applications out of it. At first glance, this may seem strange, but then I will show how we do it. It will be a holistic technology stack on JavaScript alone and you will like it. Of course, we will involve other technologies to create the application execution environment, but all these insides will be hidden from the application developer.


So, we will begin to design an experimental technological stack by defining a wide range of its applications, which will determine the set of technologies and requirements for them. We will consider the generally accepted ways of implementation and identify their critical shortcomings, after which we will think about possible replacement options.


We want to develop application software for information systems in the following areas: business, economics, education, medicine, corporate information systems and organizational and management systems, electronic commerce, personal and group communication, social networks, gaming applications, television and other media .


These systems should be:



With the help of what technologies we can develop such applications today, we will list:



In other words, the same application has to be implemented several times on a different technological stack, for different operating systems and mobile platforms. The same interface, the same functions need not only to be implemented several times, but also to make changes synchronously, keeping all client applications up to date. For the web, on the contrary, one application, but it pulls a pocket browser with all its technology stack, protocols, renderer and problems.


The web could be a cross-platform solution, but what kind of technology do we have in the old web:


Let's start with HTTP. I think it is no secret to anyone that he is blocking, i.e. through one connection that the browser has established, only one request to the server can be sent at a time and the connection waits for the server to respond. Because in HTTP there is no numbering of requests, just the usual, primitive, the simplest numbering is not, and an open connection is waiting. After all, if you send a second request, then having received the answer, it will not be clear whether this is the answer to the first request or to the second. Not to mention the transfer of the answer parts that would be mixed. But when loading the application, you need to make a lot of requests and the browser establishes the second connection for the second request, then the third ... In most browsers this number is limited to 6 simultaneous connections, i.e. it is impossible to make more parallel queries, neither for static loading, nor for AJAX. Unless, of course, you build a crutch, create multiple subdomains, and scatter requests for subdomains, because the restriction on 6 connections from the browser is tied to the host. But these tens and hundreds of connections use the network inefficiently.


In general, TCP assumes reliable delivery and long-term interaction, but HTTP breaks the connection very often, it simply does not know how to effectively re-use already established connections and constantly breaks them and creates new ones. There is, of course, Keep Alive, but not only is it specified, it must be supported by the web server, browser, and developers must know about it, properly form the headers and process them. Usually something is missing, then the administrator without a hand, the developers have not heard about this possibility, and browsers have features, as it turned out, and Keep Alive does not work very efficiently, because of the departure from the specification, the server often cannot agree with the browser and the connection does not go into keep alive mode. I already gave a talk about it and wrote an article a year ago, “ How to fix a mistake in Node.js and unintentionally raise productivity by 2 times. ” The behavior of HTTP can be traced in the developer’s tools that are in the browser. Optimization in HTTP is a separate task, out of the box it is not and it does not matter to everyone.


Establishing a TCP connection is a difficult task; it takes longer than transferring information over an already open socket. And in addition to TCP, when new connections are made, HTTP re-sends headers, including cookies. And we know that HTTP headers do not cling to gzip, along with cookies in large projects there are tens and hundreds of kilobytes of headers. Every time, and at carefree people also the statics all get out with sending a cookie in each request. All the same does not matter.


HTTP has a lot of problems, they were tried to be solved using Websocket, SSE, SPDY, HTTP / 2, QUIC, but they all depend on countless things inherited from the old web and none of these technologies achieved acceptable characteristics nor broad support. You can condense the existing traffic, make multiplexing inside other transport, but the traffic structure will not change, it will be all the same HTML, AJAX, JSON, because the task was to repair the transport so that nothing broke and all the same applications continued to go through it . But the goal has not been achieved, everything is difficult, it is not being introduced massively and the increase in speed is not very noticeable during implementation. Radically and comprehensively no one tried to solve the problem.


For games, instant messengers and other interactive applications, now there is only one live option - this is Websocket, but this layer cake from the protocols has reached the limit. IP packet, based on TCP streaming with the establishment of long connections, on top of HTTP - it is again a packet with a permanent disconnect and on top of Websocket - and it is streaming, with long connections. And all this can go via SSL. And each layer adds its own headers and converts data, especially when moving from batch to streaming and from streaming to packet. This is like a mockery, honestly.


Therefore, games and instant messengers often do not have sufficient interactivity, they podglyuchivayut, there is no necessary responsiveness in delivery, a lot of overhead and delays, and the bandwidth required to achieve is not so easy. Here are interactive applications and got stuck in development, especially due to the fact that there is no wide access to good means of transparent scaling of web sockets into tens of millions of simultaneous connections. Everyone who needs to go beyond tens of thousands of compounds already does it for himself, there is no good recipe.


It's time to admit that HTTP is full of problems and crutches, it was designed by semi-literate CERN physicists in IT who didn’t use their specialty in the past century to send articles with links and is not suitable for applications, especially for interactive applications with intensive data exchange.


HTML is of course an excellent language for page layout, but least suitable for complex user-friendly graphical user interfaces. The speed of the DOM and the lack of transactional change in interfaces lead to web applications sticking. It’s hard to come up with something more redundant and slow to represent user interfaces in memory and dynamically modifying screens. This is solved in Shadow DOM, React, Angular, Web components, but there are still more problems than solutions.


A little more about scaling . REST, which promised everyone a transparent solution, instead took the opportunity to work with the state, and did not make the promise. We have a situation where no one uses REST, but oh well, few people understand what it is, but everyone says that they have a REST API. Do URLs Identify Objects or Methods? PUT, PATCH and DELETE are used? Is the sequence of API calls unimportant? Who through REST work payments, shopping cart, chat, reminders? Mostly all AJAX APIs, which need to be called. And the problem with scaling as it was, and is solved every time anew.


There are too many problems in order to go further in detail, let us briefly summarize the main ones now:



As a result, our software quickly becomes outdated, we get constant problems with the integration of systems. We are not able to use our achievements, the reuse of the code is extremely complicated, we are doomed to Sisyphean labor, constantly reworking the same thing.


What we need:



This is no longer a web of web pages, but a network of applications and databases.
And now I will show that it is possible. I don’t say that this is what needs to be done, but our team will demonstrate the fundamental possibility on the prototype.


Metarhia is a new technological stack that:



Now let's get closer to JavaScript , it also has an excellent syntax for describing data structures, including hashes, arrays, scalar values, functions, and expressions. And we need a universal network protocol that would correspond to the data structures of the language and eliminate unnecessary data repacking. We need an end-to-end protocol and the same serialization format, one for the whole technology stack, in the database, in the client’s memory, in the server’s memory and during the transfer between processes. So why do we need JSON, XML, YAML and others, if such a format can be a subset of the JavaScript language. At the same time, we do not invent any standard or formal grammar, we already have everything. By and large, preparations for parsers are already implemented in all languages ​​and platforms. Moreover, if we take V8 and force it to be a parser of network packets , we get statistical optimization, hidden classes and an order of magnitude faster parsing than JSON. I will publish performance tests later in order not to overload this, and so voluminous, report.


Let the protocol become the central core to which all the components of the system cling. But the protocol is not enough, we need a launch environment, both client and server. Developing the ideas of Node.js, we want not only the language, but also the startup environment to be almost identical on the server and on the client. The same API, which allows you to access the functions of the platform, such as the network, file system, input / output devices. Of course, this API should not be uncontrollably accessible to applications, but must have security restrictions. The memory area and file system should be virtual, closed sandboxes, not having a message with the rest of the launch environment. All libraries should not be loaded by the programs themselves, but implemented by the launch environment as DI through control inversion (IOC). Application rights should be controlled, but management should not be escalated to the user, because he is not able to make an adequate decision on access rights. The user is inclined to either prohibit everything, being guided by paranoia or allowing everything, being guided by carelessness.


We also need a DBMS that will store data in the same JavaScript format. We need a language for describing data models (or schemas). We need a query language for data structures. As the syntax of these two query languages, we can take the same JavaScript syntax. And in the end, we need a language for describing two kinds of interfaces, network APIs and user interfaces GUI. A subset of the same JavaScript is great for both of us.


Now we can list the components of the Metarhia technology stack:



JavaScript Transfer Protocol is not only a presentation format, but also a protocol that has special constructions, headers, packages, delimiters, in general, everything that provides an implementation of transparent for applications interaction over the network. JSTP unites client and server parts so that they become one integral application. You can share interfaces and make function calls in exactly the same way as if the functions were located locally. With the only restriction that locally functions can be synchronous and asynchronous, and functions shared over a network are always asynchronous, i.e. the data is not returned via return, but through a callback, which is taken as the last argument, as is customary in JavaScript. JSTP also supports broadcasting events over the network, wrapped in a network analogue of EventEmitter. JSTP has a lot of features, but for now I’ll give you only its main features and advantages:



Now more about the last paragraph. Metadata will allow us not only to make the system more flexible, but also to optimize it further, besides the fact that we are already using V8 optimization for parsing JSTP. In most serialization formats, the names of the fields are also transferred, in JSON they occupy, according to rough estimates, from 20% to 40% of the total data transferred, in XML this percentage is much higher than half. Field names are needed only for imaginary human readability, although how often do people read XML or JSON? For JSTP, there are two options for serialization, this is a full form and an abbreviated one, built on the basis of arrays with positional key storage. In other words, we once transmit the schema, field names, their types, other auxiliary metadata from which we build a prototype on the client. Then we get multiple instances of objects serialized into arrays , i.e. field names are not transmitted, and the structure of the array positionally corresponds to the structure of the prototype, which has getters and setters. We simply assign the prototype to the array, it remains an array, but we can work with this instance through the field names. Example of conceptual code here: metarhia / JSQL / Examples / filterArray.js


Scaffolding and introspection are used when building a transparent connection between a client and a server (and generally between any processes, two servers or two clients). When connected, we take away metadata describing shared interfaces with methods and their parameters and build proxies on the opposite side that structurally repeat the remote API. Such a proxy can be used locally, actually working with a remote interface. Metaprogramming and dynamic interpretation of metamodels are used to work with the subject area, which I have already done a lot of articles and reports ( in 2014 and even in 2012 ), but I will definitely prepare materials on metaprogramming in the new technological stack.


Now the basic ideas of the technological stack of Metarhia and JSTP, as its basic component, become clear:



As Linus Torvalds says, "Bad programmers are worried about the code. Good programmers are worried about data structures." And Eric Raymond expressed it more precisely: "Smart data structures and stupid code work much better than the other way around."


Now for short JSTP examples:



Compare the description of the same data in XML, CLEAR, JSON and JSTP:


XML example
<oilPump name="PT004" displacement="constant" value="63" control= "automatic" status="working"> <flowMeter substance="fluid" recording="off" role="master" period="00:30" dataOutput="Parent.FT002.Verification"/> <outlet pressure="180" status=working"/> <failureSensors> <row><module>Seatings</module><indication>none</indication> <status>OK</status></row> <row><module>Flap01</module><indication>open</indication> <status>OK</status></row> <row><module>Flap02</module><indication>closed</indication> <status>overload</status></row> <row><module>Joint</module><indication>detach</indication> <status>OK</status></row> </failureSensors> </oilPump>

CLEAR example
 1: PT004:OilPump Displacement[constant] Value[63] Control[automatic] Status[working] 2: #FlowMeter Substance[fluid] Recording[off] Role[master] Period[00:30] DataOutput[Parent.FT002.Verification] 2: #Outlet Pressure[180] Status[working] 2: #FailureSensors:Table [Module,Indication,Status] 3: [Seatings,none,OK] 3: [Flap01,open,OK] 3: [Flap02,closed,overload] 3: [Joint,detach,OK]

JSON example
 { "name": "PT004", "type": "OilPump", "displacement": "constant", "value": 63, "control": "automatic", "status":"working", "flowMeter": { "substance": "fluid", "recording": "off", "role": "master", "period": "00:30", "dataOutput": "Parent.FT002.Verification" }, "outlet": { "pressure": 180, "status": "working", "failureSensors": [ ["Module", "Indication", "Status"], ["Seatings", "none", "OK"], ["Flap01", "open", "OK"], ["Flap02", "closed", "overload"], ["Joint", "detach", "OK"] ] } }

JSTP Example
 { name: "PT004", type: "OilPump", displacement: "constant", value: 63, control: "automatic", status:"working", flowMeter: { substance: "fluid", recording: "off", role: "master", period: "00:30", dataOutput: "Parent.FT002.Verification", }, outlet: { pressure: 180, status: "working", failureSensors: [ ["Module", "Indication", "Status"], ["Seatings", "none", "OK"], ["Flap01", "open", "OK"], ["Flap02", "closed", "overload"], ["Joint", "detach", "OK"] ] } }

Example JSTP minimized using a data scheme:


 ["PT004","OilPump","constant",63,"automatic","working", ["fluid","off","master","00:30","Parent.FT002.Verification"], [180,"working",[["Module","Indication","Status"], ["Seatings","none","OK"],["Flap01","open","OK"], ["Flap02","closed","overload"], ["Joint","detach","OK"]]]]

And now attention, the simplest JSTP parser on Node.js looks like this:


 api.jstp.parse = (s) => { let sandbox = api.vm.createContext({}); let js = api.vm.createScript('(' + s + ')'); return js.runInNewContext(sandbox); };

there are only 5 lines, and its use is simple and obvious:


 api.fs.readFile('./person.record', (e, s) => { let person = api.jstp.parse(s); console.dir(person); });

Now let's look at a more complex JSTP example, with functions and expressions:


 { name: ['Marcus', 'Aurelius'].join(' '), passport: 'AE' + '127095', birth: { date: new Date('1990-02-15'), place: 'Rome' }, age: () => { let difference = new Date() - birth.date; return Math.floor(difference / 31536000000); }, address: { country: 'Ukraine', city: 'Kiev', zip: '03056', street: 'Pobedy', building: '37', floor: '1', room: '158' } }

As you can see, functions can refer to the fields of the object itself, but for this you need to slightly modify the parser:


 api.jstp.parse = (s) => { let sandbox = vm.createContext({}); let js = vm.createScript('(' + s + ')'); let exported = js.runInNewContext(sandbox); for (let key in exported) { sandbox[key] = exported[key]; } return exported; };

My development team and independent developers are currently preparing the JSTP SDK for more than a dozen languages, and for some there are already several alternative implementations. Testing the functionality and performance of JSTP we will conduct in the near future. On the specification page below there are collected links, but not all, the links will be updated: https://github.com/metarhia/JSTP


Now there are such implementations:



Links



Conclusion


And next time I will talk about other components of the technology stack. Despite the seeming simplicity of solutions, there is still a lot to be done. I promise to periodically publish the results and conduct tests on the loads and reliability, comparison with analogues in functionality and efficiency. So far I can not recommend all this for mass implementation, but in the winter of 2016-2017 we will release the SDK with examples. A few dozen people work on this in a specially created R & D center at the Kiev Polytechnic Institute with the support of SinceTV. If you follow the project, I hope that it will still be useful to you. Even partial implementation has already yielded results. Thank you for your attention.


')

Source: https://habr.com/ru/post/306584/

More articles:


All Articles