Node.js Part 8 Tutorial: HTTP and WebSocket Protocols

Node.js is a server platform. The main task of the server is to process requests from clients, in particular - from browsers, as quickly and efficiently as possible. The eighth part of the translation guide for Node.js, which we publish today, is devoted to the protocols HTTP and WebSocket.

What happens when making HTTP requests?

Let's talk about how browsers perform requests to servers using the HTTP / 1.1 protocol.

')

If you have ever been interviewed in the IT field, then you might be asked about what happens when you type something into the address bar of the browser and press Enter. Perhaps this is one of the most popular questions that is encountered at such interviews. The one who asks such questions wants to know if you can explain some fairly simple concepts and find out if you understand the principles of the Internet.



This question involves many technologies, to understand the general principles of which is to understand how one of the most complex systems ever built by mankind, which covers the whole world, is arranged.

HTTP HTTP Protocol

Modern browsers are able to distinguish real URLs entered in their address bar from search queries, which are usually processed using the default search engine. We will talk about URLs. If you enter a site address, like flaviocopes.com , into a browser line, the browser converts this address to the form http://flaviocopes.com , assuming that the HTTP protocol will be used to exchange data with the specified resource. Please note that on Windows, what we are going to talk about here may look a little different than on macOS and Linux.

DNS DNS Lookup Phase

So, the browser, starting work on downloading data from the address requested by users, performs a DNS lookup (DNS Lookup) operation in order to find out the IP address of the corresponding server. Character resource names entered in the address bar are convenient for people, but the Internet device implies the ability to exchange data between computers using IP addresses, which are sets of numbers like 222.324.3.1 (for IPv4).



First, figuring out the IP address of the server, the browser looks in the local DNS cache in order to find out if a similar procedure has been performed recently. In the Chrome browser, for example, there is a convenient way to view the DNS cache by typing the following address in the address bar: chrome://net-internals/#dns .



If the cache does not find anything, the browser uses the POSIX gethostbyname system call to find out the IP address of the server.

GetGethostbyname feature

The gethostbyname function first checks the hosts , which, on macOS or Linux, can be found at /etc/hosts , in order to find out whether it is possible to dispense with local information when figuring out the server address.



If it is not possible to resolve the request for finding the IP address of the server by local means, the system executes the request to the DNS server. Addresses of such servers are stored in the system settings.



Here are a couple of popular DNS servers:

• 8.8.8.8: Google's DNS server.
• 1.1.1.1: CloudFlare DNS Server.

Most people use the DNS servers provided by their providers. The browser performs DNS queries using the UDP protocol.



TCP and UDP are two basic protocols used in computer networks. They are located at the same conceptual level, but TCP is a connection-oriented protocol, and for exchanging UDP messages, the processing of which creates a small additional load on the systems, the connection establishment procedure is not required. We will not talk about exactly how data is exchanged via UDP.



The IP address corresponding to the domain name in question may be in the cache of the DNS server. If this is not the case, it will contact the root DNS server. The root DNS server system consists of 13 servers on which the entire Internet depends.



It should be noted that the root DNS server is unknown correspondences between all existing domain names in the world and IP addresses. But such servers know the addresses of top-level DNS servers for domains such as .com, .it, .pizza, and so on.



After receiving the request, the root DNS server redirects it to the DNS server of the top-level domain, to the so-called TLD server (from the Top-Level Domain).



Suppose a browser is looking for an IP address for the flaviocopes.com server. By accessing the root DNS server, the browser will receive from it the TLD server address for the .com zone. Now this address will be stored in the cache, as a result, if you need to know the IP address of another URL from the .com zone, the root DNS server will not have to be contacted again.



TLD servers have IP addresses of name servers (Name Server, NS), which can be used to find out the IP address from the URL we have. Where do NS servers get this information from? The fact is that if you buy a domain, the domain registrar sends data about it to the name servers. A similar procedure is performed, for example, when changing hosting.



The servers in question are usually owned by hosting providers. As a rule, to protect against failures, several such servers are created. For example, they may have the following addresses:

• ns1.dreamhost.com
• ns2.dreamhost.com
• ns3.dreamhost.com

To find out the IP address at the URL, in the end, refer to such servers. It is they who keep current data about IP addresses.



Now, after we managed to find out the IP address behind the URL entered in the address bar of the browser, we proceed to the next step of our work.

▍ TCP connection establishment

Having learned the server’s IP address, the client can initiate a TCP connection procedure to it. In the process of establishing a TCP connection, the client and the server send each other some service data, after which they will be able to exchange information. This means that after the connection is established, the client will be able to send a request to the server.

Sending a request

A request is a structured text according to the rules of the protocol used. It consists of three parts:

• Query string
• Request header
• Body request.

Query string

The query string is a single text string that contains the following information:

• HTTP method.
• Resource address
• Protocol version.

For example, it may look like this:

 GET / HTTP/1.1 

Request header

The request header is represented by a pair of : . There are 2 required header fields, one of which is Host , and the second is Connection . The remaining fields are optional.



The headline might look like this:

 Host: flaviocopes.com Connection: close 

The Host field indicates the domain name that the browser is interested in. The Connection field set to the close value means that the connection between the client and the server does not need to be kept open.



Other commonly used query headers include the following:

• Origin
• Accept
• Accept-Encoding
• Cookie
• Cache-Control
• Dnt

In fact, there are many more.



The request header ends with an empty string.

Request body

The request body is optional, it is not used in GET requests. The request body is used in POST requests, as well as in other requests. It may contain, for example, data in JSON format.



Since now we are talking about a GET request, the request body will be empty, we will not work with it.

▍ Answer

After the server receives the request sent by the client, it processes it and sends a response to the client.



The response starts with a status code and a corresponding message. If the request is successful, the beginning of the response will look like this:

 200 OK 

If something went wrong, there may be other codes. For example, the following:

• 404 Not Found
• 403 Forbidden
• 301 Moved Permanently
• 500 Internal Server Error
• 304 Not Modified
• 401 Unauthorized

The response further contains a list of HTTP headers and the response body (which, since the request is executed by the browser, will be HTML code).

HTML parsing

After the browser receives a response from the server, the body of which contains the HTML code, it begins to parse it, repeating the above process for each resource that is needed to form the page. These resources include, for example, the following:

• CSS files.
• Images.
• Webpage icon (favicon).
• Javascript files

The way the browser displays the page does not apply to our conversation. The main thing that interests us here is that the above-described process of requesting and receiving data is used not only for the HTML code, but also for any other objects transmitted from the server to the browser using the HTTP protocol.

About creating a simple server using Node.js

Now, after we have analyzed the interaction between the browser and the server, you can take a fresh look at the First Node.js application section from the first part of this series of materials, in which we described the simple server code.

Making HTTP requests using Node.js

To execute HTTP requests using Node.js, the corresponding module is used . The examples below use the https module. The fact is that in modern conditions, whenever possible, it is necessary to use the HTTPS protocol.

▍ GET requests

Here is an example of performing a GET request using Node.js:

 const https = require('https') const options = { hostname: 'flaviocopes.com', port: 443, path: '/todos', method: 'GET' } const req = https.request(options, (res) => { console.log(`statusCode: ${res.statusCode}`) res.on('data', (d) => {   process.stdout.write(d) }) }) req.on('error', (error) => { console.error(error) }) req.end() 

▍POST request execution

Here's how to perform a POST request from Node.js:

 const https = require('https') const data = JSON.stringify({ todo: 'Buy the milk' }) const options = { hostname: 'flaviocopes.com', port: 443, path: '/todos', method: 'POST', headers: {   'Content-Type': 'application/json',   'Content-Length': data.length } } const req = https.request(options, (res) => { console.log(`statusCode: ${res.statusCode}`) res.on('data', (d) => {   process.stdout.write(d) }) }) req.on('error', (error) => { console.error(error) }) req.write(data) req.end() 

▍PUT and DELETE requests

The execution of such requests looks the same as the execution of POST requests. The main difference, in addition to the semantic content of such operations, is the value of the method property of the options object.

▍Executing HTTP requests in Node.js using the Axios library

Axios is a very popular JavaScript library that also works in a browser (this includes all modern browsers and IE, starting with IE8), and in Node.js, which can be used to perform HTTP requests.



This library is based on promises, it has some advantages over standard mechanisms, in particular, over API Fetch. Among its advantages are the following:

• Support for older browsers (you need a polyfill to use Fetch).
• Ability to interrupt requests.
• Support setting timeouts for requests.
• Built-in protection against CSRF attacks.
• Support for uploading data providing progress information on this process.
• Support JSON data conversion.
• Work at Node.js

Installation

To install the Axios, you can use npm:

 npm install axios 

The same effect can be achieved when working with yarn:

 yarn add axios 

You can connect the library to the page using unpkg.com :

 <script src="https://unpkg.com/axios/dist/axios.min.js"></script> 

API Axios

You can execute an HTTP request using the axios object:

 axios({ url: 'https://dog.ceo/api/breeds/list/all', method: 'get', data: {   foo: 'bar' } }) 

But it is usually more convenient to use special methods:

• axios.get()
• axios.post()

This is similar to how jQuery uses $.get() and $.post() instead of $.ajax() $.post() .



Axios offers separate methods for performing other types of HTTP requests that are not as popular as GET and POST, but are still used:

• axios.delete()
• axios.put()
• axios.patch()
• axios.options()

The library has a method for executing a query designed to get only HTTP headers, without the response body:

• axios.head()

GET requests

Axios is convenient to use with the use of the modern syntax async / await. In the following sample code for Node.js, the library is used to load a list of dog breeds from the Dog API . Here the axios.get() method is used and the rocks are calculated:

 const axios = require('axios') const getBreeds = async () => { try {   return await axios.get('https://dog.ceo/api/breeds/list/all') } catch (error) {   console.error(error) } } const countBreeds = async () => { const breeds = await getBreeds() if (breeds.data.message) {   console.log(`Got ${Object.entries(breeds.data.message).length} breeds`) } } countBreeds() 

The same can be rewritten without using async / await, using promises:

 const axios = require('axios') const getBreeds = () => { try {   return axios.get('https://dog.ceo/api/breeds/list/all') } catch (error) {   console.error(error) } } const countBreeds = async () => { const breeds = getBreeds()   .then(response => {     if (response.data.message) {       console.log(         `Got ${Object.entries(response.data.message).length} breeds`       )     }   })   .catch(error => {     console.log(error)   }) } countBreeds() 

Using Parameters in GET Requests

A GET request may contain parameters that look like this in a URL:

 https://site.com/?foo=bar 

With Axios, this kind of query can be done like this:

 axios.get('https://site.com/?foo=bar') 

The same effect can be achieved by setting the params property in an object with parameters:

 axios.get('https://site.com/', { params: {   foo: 'bar' } }) 

POST requests

The execution of POST requests is very similar to the execution of GET requests, but here, instead of the axios.get() method, the axios.post() method is used:

 axios.post('https://site.com/') 

As a second argument, the post method takes an object with query parameters:

 axios.post('https://site.com/', { foo: 'bar' }) 

Using the WebSocket protocol in Node.js

WebSocket is an alternative to HTTP, it can be used to organize data exchange in web applications. This protocol allows you to create long-lived bidirectional communication channels between the client and the server. After the connection is established, the communication channel remains open, which gives the application a very fast connection, characterized by low latency and a small additional load on the system.



The WebSocket protocol is supported by all modern browsers.

▍Difference from HTTP

HTTP and WebSocket are very different protocols that use different approaches to data exchange. HTTP is based on the request-response model: the server sends some data to the client after it has been requested. In the case of WebSocket, everything is different. Namely:

• The server can send messages to the client on its own initiative, without waiting for a request from the client.
• Client and server can exchange data simultaneously.
• When sending a message is used extremely small amount of service data. This, in particular, leads to low data transfer delays.

The WebSocket protocol is very well suited for real-time communication through channels that remain open for a long time. HTTP, in turn, is great for organizing episodic communication sessions initiated by the client. At the same time, it should be noted that, from the point of view of programming, implementing data exchange via the HTTP protocol is much easier than using the WebSocket protocol.

Secure version of WebSocket protocol

There is an insecure version of the WebSocket protocol (URI scheme ws:// ), which resembles http:// in terms of security. The use of ws:// should be avoided, preferring the protected version of the protocol - wss:// .

▍ Creating WebSocket connections

To create a WebSocket connection, you need to use the appropriate constructor :

 const url = 'wss://myserver.com/something' const connection = new WebSocket(url) 

After a successful connection is established, the open event is raised. You can listen to this event by assigning the callback function to the onopen property of the connection object:

 connection.onopen = () => { //... } 

For error handling, the onerror event handler is used:

 connection.onerror = error => { console.log(`WebSocket error: ${error}`) } 

▍Sending data to server

After opening the WebSocket connection to the server, you can send data to it. This can be done, for example, in the onopen onopen :

 connection.onopen = () => { connection.send('hey') } 

▍Receiving data from the server

To receive data sent from the server using the WebSocket protocol, you can assign an onmessage onmessage to be called when the message event is received:

 connection.onmessage = e => { console.log(e.data) } 

▍ Implementing a WebSocket server in Node.js environment

In order to implement a WebSocket server in the Node.js environment, you can use the popular ws library. We will use it for server development, but it is suitable for creating clients and for organizing interaction between two servers.



Install this library by initializing the project:

 yarn init yarn add ws 

The WebSocket server code that we need to write is rather compact:

 constWebSocket = require('ws') const wss = newWebSocket.Server({ port: 8080 }) wss.on('connection', ws => { ws.on('message', message => {   console.log(`Received message => ${message}`) }) ws.send('ho!') }) 

Here we create a new server that listens on port 8080, which is standard for the WebSocket protocol, and describe a callback that, when the connection is established, sends the ho! Message to the client ho! and displays in the console the message received from the client.



Here is a working example of a WebSocket server, and here is a client that can interact with it.

Results

Today we talked about the networking mechanisms supported by the Node.js platform, drawing parallels with similar mechanisms used in browsers. Our next topic will be working with files.



Dear readers! Do you use the WebSocket protocol in your web applications, the server part of which was created using Node.js?