Scalable and high-performance web applications. Ch. 1. The HTTP protocol. Draft.

Topics covered: HTTP requests; HTTP responses; Optimization using HTTP keep-alive; Optimize with HTTP pipelining; Optimizing using HTTP caching: Caching in modern clients, revalidation strategies, caching turned off and interactive pages; Optimization with compression; Specific optimization of the content delivered; Combining content.

Comments are welcome.


')
<div class = "chapter" lang = "ru" xml: lang = "ru">

Chapter 1. The HTTP Protocol

Content HTTP requests HTTP responses Optimization using HTTP keep-alive Optimization using HTTP-pipelining Optimization using HTTP caching Caching in modern clients Revalidation strategies Disabling caching and interactive pages Optimization using compression
content Combining content Clients (i.e. browsers or robots) communicate with web servers
it is via HTTP. Fine details of http work and its
interactions with the lower and upper layers of technology - one of
keys to high performance. https stands for Hyper Text Transport Protocol.
Of course, it is used not only to transfer from server to client
HTML files, but also for any other objects: images, scripts,
CSS files, data files. It also works in reverse.
side - for uploading files to the server, sending forms and
etc. AJAX applications also obviously communicate with the server over
HTTP Sometimes HTTP is used for more specific things.
for example, to manage server content via WebDAV and
etc. HTTP communication consists of a series of interleaved
HTTP requests and HTTP responses between client and servers. <Div class = "sect1" lang = "ru" xml: lang = "en">

HTTP requests

An HTTP request consists of three parts: the query string, headers, and
message body. Here is an example of an HTTP request that retrieves a file.
www.habrahabr.ru/robots.txt :

  GET /robots.txt HTTP / 1.1
 Host: www.habrahabr.ru
 User-Agent: Wget 1.8.3

The vast majority of HTTP requests in the modern web -
these are GET requests, whose request body is empty (zero length), that we
and observe in this case. POST requests, for example, body
almost always non-empty. Let's look at the sample query in detail. The first word in the request is
GET . This is the so-called "method"
HTTP request. The most common method is just
GET . The GET method means that we simply
request content (page or file) available at specified
URL'u. An important fact is that GET requests can
cached. Another frequently encountered request is POST .
POST requests almost always contain the request body encoded
in one of the special formats. POST requests are usually sent to
the result of the user clicking on the “Submit” button or similar
Action. The HEAD method (a neighbor of the GET method) is also used, and
a whole series of more rare methods like PUT ,

DELETE , as well as a large set of methods specific to
WebDAV protocol. More about them is in other chapters. The second word in our query: /robots.txt -
This is the URL of the page or file we want to get. The third word in our request is HTTP/1.1 . it
proper protocol name and version number. HTTP 1.1
almost universally used, but for some it's enough
specific goals can sometimes be useful to reduce the version to HTTP
1.0. The main differences between HTTP 1.1 and HTTP 1.0 are mandatory.
support for virtual hosts, improved caching support,
HTTP support for keep-alive, and some more important
capabilities. The query string is terminated with a CR-LF (\ r \ n) character combination.
Next come the request headers, separated from each other as well.
CR-LF combination. The only required (in HTTP 1.1)
the header is the Host containing the domain name on
which is a page or file. The HTTP standard describes
many different request headers, many of which play
key role in ensuring high performance. AT
The example shows another header, the User-Agent , containing
client identifier line. The list of headers ends with another character combination
CR-LF (i.e., empty string). After the heading is the body
request. In our example, the request body is empty (zero length). <Div class = "sect1" lang = "ru" xml: lang = "ru">

HTTP responses

By sending an HTTP request to the server, the client is waiting for a response. HTTP response
It looks generally similar to a query: status bar, list
headers and response body.

  HTTP / 1.1 200 OK
 Server: nginx / 0.5.35
 Date: Tue, Apr 22, 2008 10:18:08 GMT
 Content-Type: text / plain;  charset = windows-1251
 Connection: close
 Last-Modified: Fri, 30 Nov 2007 12:46:53 GMT
 ETag: "27e74f-43-4750063d"
 Accept-Ranges: bytes
 Content-Length: 34

 User-agent: *
 Disallow: / people

The first word in the response is HTTP/1.1 , version
of the protocol. The second word is 200 . This is the most common
response code indicating that the requested file or page is detected and
given to the client. There are many different response codes, some of
which plays a key role in ensuring high
performance. The rest of the line is
the human-readable description of the response code (in this case,
OK ). After the status bar, there is a list of headers containing
additional, sometimes extremely important information about the file or
page. In this case, of interest is the MIME type of content
the last modified date of the file ( Last-Modified )
length ( Content-Length ), and also ETag (this
special token used for caching). Also in response
there is some server information ( Server and

Date ) and about the connection ( Connection ). After the list of titles, we see another empty line and,
finally, the actual content of the response (in this case, these are two
text lines). It can be concluded that HTTP is simple, flexible due to
text syntax protocol. Careful use of the set
request and response headers will allow to achieve significant improvements in
areas of high performance web system. Detailed analysis
we will deal with the following technologies in the following chapters. <div class = "sect1" lang = "en" xml: lang = "en">

Optimize with HTTP keep-alive

The HTTP protocol runs on top of the TCP protocol (Transmission
Control Protocol). TCP is a reliable protocol.
two-way data stream transmission. TCP works by sending packets
data from the client to the server and back. TCP packet consists of a header
and data. The title indicates, among other things, the client’s IP address and
server numbers, TCP port numbers used on the client and server, and dialing
flags. The server typically uses a standard TCP port.
The number 80.TCP-connection between the client and the server is established with
using the classic "TCP three-way handshake". First client
sends a packet with a SYN flag to the server. In response, the server sends a packet with
SYN + ACK flags. Finally, the client sends another packet, with the flag
ACK and from now on the connection is considered established, and the client can
send your data, in our case - an HTTP request. It can be seen that this part of the protocol interaction requires one
just wait for the response packet from the server before you could
send an HTTP request. That is, the operation of establishing a TCP connection
may be relatively expensive, and on some channels with high
latency and quite expensive. In other words, we
interested in reducing the number of TCP connections. In HTTP 1.1, this optimization is enabled by default. Once
having connected to the server, the client exchanges requests with it and
answers without breaking the connection. This is called HTTP
keep-alive. It is clear that if a dozen thousand browsers install with the server
keep-alive connection, they quickly exhaust its resources.
Therefore, all servers have a configurable timeout after expiration
which keep-alive connection is broken if it was not
no activity. The client may request a disconnection after a response by sending to
request header Connection: close . Similarly
the server may indicate in the response that it does not wish to support keep-alive
connection, passing the exact same header:
Connection: close . Generally speaking, all these
splicing with mutual notification, strictly speaking, do not impose
no duties. Both server and client must be fully prepared.
to the fact that the connection is interrupted at any time on the initiative
the other party without any notification. In order to maintain the integrity of the keep-alive connection,
the server must know the length of the response. The easiest way -
specify it in the Content-Length header. If the length
the response is not specified by the handler, the server is forced before sending
response set header Connection: close and
close the connection for its part after sending the response. Make sure your web application correctly exposes
Content-Length header, otherwise keep-alive connections
will not work. Sometimes the size of the response is difficult to determine at the stage of creation. AT
In this case, you can either donate a keep-alive connection, or
use a special chunked response encoding (it is given with
using the Transfer-Encoding: chunked header.
Details can be found in the HTTP standard. <Div class = "sect2" lang = "ru" xml: lang = "ru">

Optimize with HTTP pipelining

When we make a series of requests and responses within one
keep-alive connections, time plays an important role in performance
latency between request and response. Delay may be
caused by both high channel latency and long time
processing requests on the server. Before sending the next request, we
Must wait for the processing of the next. To cope with
this problem, technology can be used
HTTP-pipelining. The meaning of HTTP-pipelining is simply that the client
sends several requests in succession, and then starts gradually
to rake incoming answers. Server support in general
good, but unfortunately at the moment, customers supporting
HTTP-pipelining, make up a very small percentage of the market. <Div class = "sect1" lang = "en" xml: lang = "en">

Optimize with HTTP caching

Over the years, a lot of caching has accumulated around HTTP caching.
myths and prejudices. Chapter 13 RFC2616 (Caching in HTTP)
pretty messy. The main reason for this is that the authors
tried to explain everything to everyone at once: both to the developers of the servers, and
developers of clients, and developers of caching proxies. Besides
Moreover, the confusion adds a discussion of the fight against obsolete
(HTTP 1.0) clients, servers and proxies. In this chapter we will try to accurately tell the whole story with
point of view of a modern developer (that is, supporting
HTTP 1.1) server. First we discuss the simplest situation: how
work with a modern client (without caching proxies on the way).
Then - how to work with a modern client through one or
several modern cache proxies. Finally, we will discuss
Additionally, which (obsolete) headers should be added to the response,
to achieve acceptable results from old and outdated customers
proxy on the way (note, however, that the existence of these two types
animals prove extremely difficult). <div class = "sect2" lang = "ru" xml: lang = "ru">

Caching in modern clients

In general, the caching process consists of two parts:
the expiration date of the content and the revalidation procedure. To set the expiration date of the content, it is used
Cache-Control: max-age=300 response header Cache-Control: max-age=300 . Here

300 is the expiration date of the content in seconds
(i.e., five minutes). This means that within five minutes the client
can freely use the received object without asking at all
re-server. If an object is required after the expiration date, its
must be revalidated. Revalidation is just
repeated request of the same object, but with additional
headings. Revalidation may result in either retransmission
object, or to sending a special response code
304 Not Modified . Obviously, the second option
is preferred. There are three main methods of revalidation:

• by last modified time
  ( Last-Modified );
• by object identifier ( ETag );
• unconditional (just give the contents of the object);

In order to work revalidation on the last
changes, there should be a special header among the response headers
Last-Modified , containing, naturally, the last time
object changes. When revalidation, this value is transmitted by the client to
special request header: If-Modified-Since .
The request handler can check if the object has changed, and if
no - return the answer with an empty body and a response code

304 Not Modified . The content itself is not
transmitted, and the client will use the content that
stored in his cache. Use revalidation by last modified time is easier
all for static files, and all default servers automatically
include this feature. For the case of dynamically generated pages
the question of choosing the right architecture for processing the request and in general
The possibility and expediency of such a revalidation is discussed.
below. In order for revalidation to work by object identifier,
among the response headers there should be a special header
ETag . It returns some unspecified
object version identifier (for example, for Apache static files
by default it uses a combination of the last modified date of the file,
its size and inode number in the file system). With revalidation
the client sends a special header in the request
If-None-Match containing the same identifier.
The server compares the current value of the version identifier, and if
matches, returns the response with the empty body and the response code
304 Not Modified As in the case of revalidation on the last modification time,
This revalidation is easiest to use for static files. For
the case of dynamically generated pages the question of choosing the right
Content architecture is discussed below. Both ways of revalidation can
shared. For compatibility with mythological browsers and proxies, not
support HTTP 1.1, add a header to the response
Expires , which indicates the required expiration time
file or page. The format of this header is:
Tue, 15 Nov 1994 08:12:31 GMT . <div class = "sect2" lang = "ru" xml: lang = "ru">

Revalidation strategies

As already mentioned, the easiest and most understandable is caching.
static files like design images, javascript files and
CSS files. For static files it is very easy to generate and
check the value of both conditions: as Last-Modified , so
and ETag . The ETag difficult question in the case of static
files - choose a convenient expiration date. Developers
Yahoo.com advises a simple strategy: a) put on static
files have a one-year shelf life (read: "forever") and b)
introduce a system for changing urls when posting a new version of the project
on a production server. For example, in Ruby on Rails you can set up a project so that the URL
CSS files will look like this:

/stylesheets/main.css?31415 , where
31415 is the latest revision number in the system.
version control. A similar mechanism should be easy to implement in
any engine. Another conservative strategy is to cache
static files for a short time, about half an hour. So
Thus, we speed up a typical user session. When will he go
to the site next time, the first time you just happen
Mass revalidation of all statics. This strategy is not required
make changes to the system of removal of code for production (which, in principle,
can be considered a disadvantage of this strategy - it is often
may mean that there is no such system at all). In general, in order to achieve some significant increase
performance, you need a happy combination of conditions.
First, it should be relatively easy for a page to be calculated.
validator (last modification time or ETag). Secondly,
info-page design should allow client download enough
obsolete copies. By experience, rare classes of pages
provide this combination, not to mention some kind of measurable effect
from the introduction of revalidation. It is possible that for modern
interactive projects in general no longer makes sense to think in this
direction and concentrate on extremely high-speed generation
these same pages. We repeat, however, that the above caching of static
files plays a crucial role in the user experience of each particular
user, and such caching should be given attention (good
this is not difficult). Another aspect of this problem is that search engines
systems are highly recommended to set a more or less correct time.
Last-modified ( Last-Modified ) on all pages,
who are you going to feed the search robot. It is possible that
the process of making corrections to the engine on this topic you will succeed without
special effort to introduce also the strategy itself
caching. <div class = "sect2" lang = "ru" xml: lang = "ru">

Disable caching and interactive pages

Often it is necessary to solve also the inverse problem: how to make
so that certain pages of the site are reloaded from the site each
time the user logs in to them. There is one on this topic.
of the common myths associated with caching in HTTP. He is bound
with the ritual repetition of the magical incantation of the three magical
headers:

 Cache-Control: no-cache
 Pragma: no-cache
 Expires: now

We will not dwell on the detailed analysis of errors,
collected in these three lines. We just say that the first line
not completely intended for this purpose, the second line never
it did not make sense, and the third one was catastrophically obsolete (although it does
assigned task). The correct modern way is to use
Cache-Control: max-age=0 header Cache-Control: max-age=0 . He means
that the contents of the page become obsolete immediately upon receipt, and
the browser must revalidate it. The server, having received a request for
revalidation, simply returns to the user a new version
pages. For compatibility with mythological old browsers and
proxies that do not support HTTP 1.1 should also be answered
add an Expires: now header. He has the same
sense as Cache-Control: max-age=0 <div class = "sect1" lang = "en" xml: lang = "en">

Optimization with compression

An excellent way to significantly reduce traffic and speed up the response.
server is the compression of the given files and pages. Practically
all modern web servers somehow support this
functionality. The compression ratio varies depending on
each specific page and can reach values ​​of the order of 10. For
web application developer compression is completely transparent and does not require
no effort. The ability to take compressed client content
announces server using header

Accept-Encoding: gzip . If the server is configured to
compressing the relevant content, then it can add a header
Content-Encoding: gzip response (not to be confused with
Transfer-Encoding ) and send the client compressed
Content. Historically different browsers had many different errors.
in the processing of compressed content. Now (mid 2008)
It seems that the percentage of actually used clients with errors has decreased to
negligible. Usually the documentation for the server is set forth.
An exemplary set of combinations of content types and browsers that
work, as well as recommendations to work around errors in old clients.
<div class = "sect2" lang = "ru" xml: lang = "ru">

Specific optimization given
the contents

Simply put, it removes extra spaces and line breaks.
from html files and css files, minification of javascript as well
optimization of file sizes with pictures. Besides influencing the size
transmitted data, this optimization can also lead to an increase in
client-side performance: the browser will have less
parsing and taking up less memory for page source and DOM nodes
processed page.JavaScript minification can also occur simultaneously with
obfuscation (obscuring the meaning of the code), which may have some meaning in
terms of intellectual property protection policies. Libraries

Multi-purpose tools such as prototype.js contain built-in support for minification, as well as the ability to remove a
subset of components that are not used in the project. <div class = "sect2" lang = "ru" xml: lang = "en">

Combining content

In order to reduce the number of HTTP requests, you can simply
combine several files into one. Often, with a
combination, you can also minify the resulting file. The CSS files can simply be concatenated into one. JavaScript files
require caution in this regard, but in general we can assume that
if your JavaScript is not concatenated, you have some
problem with the JS code architecture. Icons, navigation graphics, etc. can be combined into
one file using t n. CSS spriting. See the numerous
guides on this topic on the Internet.