Railgun protocol for traffic compression

CDN provider CloudFlare last year developed the Railgun protocol for compressing network traffic. The binary protocol, written in the programming language Google Go, transfers the hashes of the web pages and the diffs between them, with versioning support. After the introduction of this technology, the compression ratio in some cases reaches 99.6%, which is unattainable with gzip. Sites 4Chan and Imgur after the introduction of Railgun reduced non-cached traffic by about 50% .

Now, CloudFlare has announced that protocol support has also been implemented at Amazon Web Services and about 30 of the largest hosting companies. Putting Railgun is now easier than ever: there are plugins for WordPress, Joomla, Drupal and other CMS, packages for most of the popular Linux and BSD distributions have been released.

Any CDN provider tries to cache the maximum amount of content. But in reality, only about 66% of objects can be cached, and the remaining 34% have to be re-requested from the server in case of an update. To compress this traffic, Railgun was created.
')


Railgun uses roughly the same algorithms as when processing a sequence of frames in high-definition video. A high degree of compression in video codecs is achieved by compressing not individual frames, but differences between adjacent frames. This allows you to compress the frame size of millions of pixels in several kilobytes. Theoretically, you can compress it at all in one byte, if it is no different from the previous frame. The same with web pages. When a web page changes, only a small part of it changes in reality, and it is enough to transfer the change between the current version and the one that the client received the previous time.

Technically, a Railgun consists of two components: a sender (sender) and a receiver (listener). Senders are installed in every CloudFlare CDN-network worldwide. Recipient - software component that is provided to commercial partners of CloudFlare. A TLS-protected TCP connection is established between the sender and the receiver, using which the binary Railgun protocol asynchronously transmits HTTP requests. For the web client, the Railgun system looks like a proxy server, although in fact it is a specialized system with specific functions. One of them is the compression of content that cannot be cached, due to the synchronization of page versions. When updating a page version over the network, only the change between the previous and the new version is transferred.



Testing has shown that maximum compression is achieved on news sites with high traffic. For example, a binary comparison of the reddit.com home page shows an average change of 2.15% within 5 minutes and 3.16% within an hour. The home page of The New York Times changes binary by 0.6% in five minutes and by 3% within an hour. For the BBC News homepage, these figures are 0.4% and 2%, respectively. In general, for the most popular sites, when you re-request diff, it is sometimes so small that it fits in a single TCP packet.