WebService with gzip C support

In the post I will discuss my experience of embedding the XML-RPC interface in a utility written in C. The interface should provide access to the statistics and the results of the utility. One of the requirements for the interface - support responses in gzip format, in order to save traffic. I really wanted to do a little blood and that's what came out of it.

First of all tests

Let's start with the tests. The XML-RPC client in python fits into 4 lines. By the way, he just understands the answers in gzip format.

import xmlrpclib if __name__ == '__main__': proxy = xmlrpclib.ServerProxy("http://localhost:8080/", verbose=True) print proxy.sayHello() 

Fine! Now we know what HTTP headers are received by the client. And if the format is incorrect - we get an exception with a detailed call stack. In case of an error, all this will help us shed light on the cause of its occurrence.

Zlib

The wiki says that the gzip format is based on the deflate compression algorithm, which is implemented in the zlib library. This library has a great compress method.
')

 int compress (Bytef *dest, uLongf *destLen, const Bytef *source, uLong sourceLen); 

Having enjoyed the discovery, I immediately decided to try this method and sketched a simple backbone of the application, but this was not enough. The client refused to understand the contents of the server responses and fell out with the exception. I had to study the gzip format more.

Gzip

It's all pretty simple .



Compressed data is framed with ten bytes of a special format header and eight bytes of a suffix containing the checksum of the source data and their length.

The header starts with the magic constants ID1 = 31 (0x1f, \ 037), ID2 = 139 (0x8b, \ 213), which indicate the beginning of the data in gzip format. Next is the CM (Compress Method), in the case of deflate CM = 8. Zanim followed flags, in our case FLG = 1, which means text data. Then there are 4 bytes of the date of the last change in the source data, in our case MTIME = 0. Then there are additional flags XFL = 2 (high compression ratio). The name of the operating system let us leave undefined OS = 255.

To calculate the checksum, we use the function from the same zlib

 uLong crc32 (uLong crc, const Bytef *buf, uInt len); 

But this is not enough. Our client is still not satisfied with the server responses.

And again zlib

Let's see in what format zlib data is returned to us.



It turned out that zlib adds a special 2-byte prefix and 4-byte suffix to the compressed data ( more ). Get rid of them and add a header and a gzip suffix.

And, lo and behold! The client finally understood us!

Note: in the Qt library there is a qCompress () method, which returns data compressed by the zlib library, but also with a 4-byte prefix for the length of the compressed data.

Total

To form data in gzip format, compress the source data with the compress function, in the resulting array, replace the first 2 bytes with the 10-byte gzip header, instead of the last 4 bytes, put the checksum and length of the original data.

An example of a working XML-RPC server returning data in gzip format is shown below.

 #include <zlib.h> #include <stdio.h> #include <stdlib.h> #include <string.h> #include <netinet/in.h> #include <sys/socket.h> #define PORT 8080 #define MAXCONN 5 #define BUF_SZ 1024 #define ZLIB_PREFIX_SZ 2 #define ZLIB_SUFFIX_SZ 4 #define GZIP_PREFIX_SZ 10 #define GZIP_SUFFIX_SZ 8 // Returns listen socket handle int create_srvsock(int port, int maxconn); // Returns response to be sent back int get_response(int clisock, char *response); // Writes given data range to socket void write_range(int sock, const char *begin, const char *end); // Write int value to socket void write_int(int sock, int value); // Prints error message and exit void error(const char *msg); int main(int argc, const char *argv[]) { fprintf(stderr, "HTTP Server with gzip encoding support using zlib (%s)\r\n", ZLIB_VERSION); char httpheaders[BUF_SZ] = {0,}; char response[BUF_SZ] = {0,}; char compressed[BUF_SZ] = {0,}; int srvsock = create_srvsock(PORT, MAXCONN); fprintf(stderr, "Server is started on port %d\r\n", PORT); while (true) { struct sockaddr_in addr = {0,}; socklen_t addrlen = sizeof(addr); // 1. Accepting connection int clisock = accept(srvsock, (struct sockaddr *)&addr, &addrlen); // 2. Retreiving response int responselen = get_response(clisock, response); // 3. Compressing response long unsigned int compressedlen = BUF_SZ; if (compress((unsigned char *)compressed, &compressedlen , (const unsigned char *)response, responselen) != Z_OK) error("Can not compress"); // substract zlib prefix and suffix: http://www.ietf.org/rfc/rfc1950.txt compressedlen -= ZLIB_PREFIX_SZ + ZLIB_SUFFIX_SZ; // 4. Writing HTTP headers int contentlen = GZIP_PREFIX_SZ + compressedlen + GZIP_SUFFIX_SZ; int httpheaderslen = sprintf(httpheaders, "HTTP/1.1 200 OK\r\n"\ "Content-Type: text/xml\r\n"\ "Content-Encoding: gzip\r\n"\ "Content-Length: %d\r\n\r\n", contentlen); write_range(clisock, httpheaders, httpheaders + httpheaderslen); // 5. Writing gzip headers: http://www.gzip.org/zlib/rfc-gzip.html const char gzipheader[] = { 0x1f, 0x8b // gzip magic number , 8 // compress method "defalte" , 1 // text data , 0, 0, 0, 0 // timestamp is not set , 2 // maximum compression flag , 255 // unknown OS }; write_range(clisock, gzipheader, gzipheader + sizeof(gzipheader)); // 6. Write compressed data write_range(clisock, compressed + ZLIB_PREFIX_SZ , compressed + ZLIB_PREFIX_SZ + compressedlen); // 7. Append crc32 write_int(clisock, (int)crc32(0, (unsigned char *)response, responselen)); // 8. Append initial size write_int(clisock, responselen); } return EXIT_SUCCESS; } // Returns listen socket handle int create_srvsock(int port, int maxconn) { int sock = 0; struct sockaddr_in addr = {0,}; addr.sin_family = AF_INET; addr.sin_port = htons(port); addr.sin_addr.s_addr = INADDR_ANY; if ((sock = socket(AF_INET, SOCK_STREAM, 0)) < 0) error("Can not open socket"); if (bind(sock, (struct sockaddr *)&addr, sizeof(addr)) < 0) error("Can not bind socket"); if (listen(sock, maxconn) < 0) error("Can not listen socket"); return sock; } // Returns response to be sent back int get_response(int clisock, char *response) { return sprintf(response, "<?xml version=\"1.0\"?>\r\n"\ "<methodResponse>\r\n"\ " <params><param><value>Hello there!</value></param></params>\r\n"\ "</methodResponse>"); } // Writes given data range to socket void write_range(int sock, const char* begin, const char *end) { for (const char *it = begin; it != end;) { int written = write(sock, it, end - it); if (written < 0) error("Can not write to socket"); it += written; } } // Write int value to socket void write_int(int sock, int value) { const char *data = (const char *)&value; write_range(sock, data, data + sizeof(int)); } // Prints error message and exit void error(const char *msg) { perror(msg); exit(EXIT_FAILURE); }