RFC for wimps or the story of a single investigation
Preamble : one day we decided to connect a CDN to our site in order to please users with faster loading pages. After some searches, the choice fell on Highwinds, because they stated that they supported all the necessary functionality and managed to agree on a very tasty price. After the successful transfer of the site to work through Highwinds, we decided - why not switch to them and our REST API for mobile applications. And then began the fun.
Switched API on test devices to work via CDN, check: iOS is working, Android also seems to work, although wait. In the Android application, only GET and HEAD requests work, and POST, PUT and so on fall from 502. After a brief trial and comparing the traffic of iOS and Android applications, we find out that Android sends the header “Transfer-Encoding: chunked” in the requests.
We try to pull the API page with curl:
Works. And what if you try this:
')
Yeah, it does not work, despite the fact that without using a CDN, such requests are excellent.
In the access logs of our nginx we see that requests fell with code 400 “Bad request”.
But there may be a problem in that curl sends the “Transfer-Encoding: chunked” header, but does not generate the data properly. Check this option by writing a small Python script that sends data in chunks.
The script hangs 30 seconds (30 seconds is the request write timeout in the CDN settings) and falls out with an error.
The output shows the following:
It can be seen that the request is correct, after the last chunk there is a zero-length message “0 \ r \ n \ r \ n”, informing the web server that all chunks have been transmitted. But the CDN server continues to wait for more chunks and after 30 seconds it falls off by timeout.
But it is still too early to shift the blame on the CDN. As we remember, the request reaches our nginx, but it falls off with code 400, is it possible that our nginx is to blame? Check this by dumping the traffic and selecting the “Follow TCP Stream” option in Wireshark to see the data in a readable format:
As you can see, nginx received headers, but POST data did not reach it in any way, and when the CDN server gives the client 502 and terminates the connection with nginx, there is nothing left for it but to write a message to the log stating that it received an invalid request.
Consider the last possibility, maybe the CDN is not required to work with “Transfer-Encoding: chunked” and we are to blame for using it in the application? Read what RFC 7230 is thinking about. What we are looking for was found in sections 3.3.1 and 4.1 . By standard, using Transfer-Encoding: chunked is allowed in both requests and responses. It is separately stated that this is a mandatory part of HTTP / 1.1 and it should be supported in all applications implementing this standard.
We have collected all the evidence that the problem is in the incorrect operation of the HTTP server on the CDN side. We write a ticket to the support and after a long clarification of all the details of the problem and communication with their engineers we get a wonderful answer.
Switched API on test devices to work via CDN, check: iOS is working, Android also seems to work, although wait. In the Android application, only GET and HEAD requests work, and POST, PUT and so on fall from 502. After a brief trial and comparing the traffic of iOS and Android applications, we find out that Android sends the header “Transfer-Encoding: chunked” in the requests.
We try to pull the API page with curl:
curl https://cdn.api.example.com -XPOST -d 'test=data' Works. And what if you try this:
')
curl https://cdn.api.example.com -XPOST -d 'test=data' -H 'Transfer-Encoding: chunked' Yeah, it does not work, despite the fact that without using a CDN, such requests are excellent.
In the access logs of our nginx we see that requests fell with code 400 “Bad request”.
But there may be a problem in that curl sends the “Transfer-Encoding: chunked” header, but does not generate the data properly. Check this option by writing a small Python script that sends data in chunks.
import requests import logging import httplib as http_client http_client.HTTPConnection.debuglevel = 1 logging.basicConfig() logging.getLogger().setLevel(logging.DEBUG) requests_log = logging.getLogger("requests.packages.urllib3") requests_log.setLevel(logging.DEBUG) requests_log.propagate = True def test(): yield 'data' yield 'test' s = requests.Session() data = s.post('https://cdn.api.example.com', data=test()) The script hangs 30 seconds (30 seconds is the request write timeout in the CDN settings) and falls out with an error.
The output shows the following:
send: 'POST cdn.api.example.com HTTP/1.1\r\nHost: cdn.api.example.com\r\nConnection: keep-alive\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nUser-Agent: python-requests/2.18.4\r\nTransfer-Encoding: chunked\r\n\r\n' send: '4' send: '\r\n' send: 'data' send: '\r\n' send: '4' send: '\r\n' send: 'test' send: '\r\n' send: '0\r\n\r\n' reply: 'HTTP/1.1 502 Bad Gateway\r\n' header: Date: Mon, 11 Dec 2017 22:05:04 GMT header: Connection: Keep-Alive header: Accept-Ranges: bytes header: Cache-Control: max-age=10 header: Content-Length: 0 header: X-HW: 1 It can be seen that the request is correct, after the last chunk there is a zero-length message “0 \ r \ n \ r \ n”, informing the web server that all chunks have been transmitted. But the CDN server continues to wait for more chunks and after 30 seconds it falls off by timeout.
But it is still too early to shift the blame on the CDN. As we remember, the request reaches our nginx, but it falls off with code 400, is it possible that our nginx is to blame? Check this by dumping the traffic and selecting the “Follow TCP Stream” option in Wireshark to see the data in a readable format:
POST / HTTP/1.1 Date: Tue, 12 Dec 2017 07:19:48 GMT Host: cdn.api.example.com Connection: Keep-Alive Accept-Encoding: gzip, deflate Accept: */* User-Agent: python-requests/2.18.4 Transfer-Encoding: chunked As you can see, nginx received headers, but POST data did not reach it in any way, and when the CDN server gives the client 502 and terminates the connection with nginx, there is nothing left for it but to write a message to the log stating that it received an invalid request.
Consider the last possibility, maybe the CDN is not required to work with “Transfer-Encoding: chunked” and we are to blame for using it in the application? Read what RFC 7230 is thinking about. What we are looking for was found in sections 3.3.1 and 4.1 . By standard, using Transfer-Encoding: chunked is allowed in both requests and responses. It is separately stated that this is a mandatory part of HTTP / 1.1 and it should be supported in all applications implementing this standard.
We have collected all the evidence that the problem is in the incorrect operation of the HTTP server on the CDN side. We write a ticket to the support and after a long clarification of all the details of the problem and communication with their engineers we get a wonderful answer.
It’s not working by design.After this, even add nothing special. Separately, I want to note that the problem did not arise if Highwinds used any Open Source implementation of the HTTP server, for example Varnish or Nginx, and did not write its own with such features “by design”. Beware of HTTP protocol fakes.
Source: https://habr.com/ru/post/344872/
All Articles