Gixy - open source from Yandex, which will make configuring Nginx safe
Nginx is definitely one of the coolest web servers. However, being rather simple, rather expandable and productive, it requires respect for yourself. However, this applies to almost any software on which the security and performance of the service depends. I admit, we like Nginx. In Yandex, it is represented by a huge number of installations with various configurations: from simple reverse proxy to full-fledged applications. Due to this diversity, we have accumulated some experience of its [not] safe configuration, which we want to share.
But first things first. We have long been tormented by the issue of the secure configuration of Nginx, because it is a full cube of the web application, which means that its configuration requires no less control on our part than the code of the application itself. Last year it became obvious to us that this process requires serious automation. Thus began the in-house Gixy project, the requirements for which we outlined as follows:
')
- be simple;
- but expandable;
- with the possibility of convenient integration into testing processes;
- it would be nice to be able to rekolvit incluses;
- and work with variables;
- and about regular expressions not to forget.
Frankly, we have hesitated to the last with the choice of language (between Golang and Python). As a result, Python was chosen with the hope that it is more common, which means it will be a little easier with development.
On this, we’ll finish the introduction and go over to examples of common problems :) To avoid confusion in the future, the current mainline version of Nginx, 1.13.0, was used in all examples.
Server-Side-Request-Forgery
Server Side Request Forgery is a vulnerability that allows to perform various kinds of requests on behalf of a web application (in our case, on behalf of Nginx). Occurs when an attacker can control the address of the proxied server, for example, in case of incorrect configuration of the XSendfile .
From my own experience I can say that often the vulnerability is associated with several errors:
- lack of directive internal . Its meaning is to indicate that a particular location can only be used for internal requests;
- unsafe internal redirection.
If everything is clear with the first case, then with the internal redirection things are not so simple. I believe many of you have seen / written a similar configuration:
Unfortunately, in this configuration you need to check at least all the rewrite and try_files directives, since according to the documentation:
It turns out that any careless rewrite will allow you to make a request in the internal location. This is fairly easy to verify:
- configuration:
- exploitation:
In this situation, we usually recommend several practices:
- use only internal location for proxying;
- if possible, prohibit the transfer of user data;
- secure the address of the proxied server:
• if the number of proxied hosts is limited (for example, you have S3), then it is better to hard-code them and select using map or in any other way convenient for you;
• if for some reason it is not possible to list all possible hosts for proxying, it is worth signing.
Bad regular expressions for validating a referrer or origin
Often, validation of the “Referer” or “Origin” request header is done using a regular expression. Often this is necessary for conditional exposure of the X-Frame-Options header (ClickJacking protection) or Cross-Origin Resource Sharing ( CORS ) implementation. And if with the validation of “Referer” everything is a bit simpler and, with some conditions, you can refuse the regular expression in favor of the ngx_http_referer_module module, then with “Origin” everything is not so simple.
We distinguish two main classes of problems:
- errors in the compilation of a regular expression;
- permission of untrusted third-party domains.
Problem configuration is as follows:
In fact, I have simplified the regular expression very much, but even in this example, seeing the problem from the first time is not so easy. It is easier for people to write regular expressions than to read them.
Fortunately, this problem is not typical for the machine, so Gixy is able to independently determine that this regular expression is www.matdex.ru.evil.com as a valid origin and will inform you about it:
Or, if you assume ya.ru is not trusted enough, it will report the origins of ya.ru and www.yandex.ru.evil.com :
HTTP Splitting
HTTP Splitting is used to attack an application that is behind Nginx (HTTP Request Splitting) or application clients (HTTP Response Splitting). Vulnerability occurs when an attacker can embed a newline character \ n in a request or response generated by Nginx.
I have no reliable advice (except how to be attentive), but you should always pay attention to several things:
- which variables are used in the directives responsible for creating queries (can they contain CRLF), for example: rewrite, return, add_header, proxy_set_header and proxy_pass;
- whether the variables $ uri and $ document_uri are used, and if so, in which directives, since they are guaranteed to contain the url-decoded value;
- pay special attention to variables derived from groups with an exclusive range: (? P [^.] +).
Exclusive Range Example:
- configuration:
- exploitation:
As you can see, we were able to add an x-crlf-header response header: injected. This happened due to the coincidence of several circumstances:
- add_header does not encode / validate the values ​​passed to it, considering that the author is aware of the consequences;
- the value of the path is normalized before processing the location;
- the $ action variable was selected from the regular expression group with the exclusive range: [^.] *;
- thus, the value of the $ action variable became equal to see below \ r \ nx-crlf-header: injected and got into the HTTP response.
Fortunately, Gixy successfully copes with this task:
- he knows about "dangerous" variables - more precisely, he knows about the allowable set of characters in most embedded variables. Thus, the difference between $ request_uri and $ uri is obvious to him;
- able to select variables from regular expression groups;
- able to determine whether any character (in our case, \ n) can be matched with a regular expression (or a single group).
Another interesting example is rewright with try_files:
- configuration:
- operation (on 127.0.0.1:9000 listens to the debug echo-server):
What to do?
- Try to use more secure variables, such as $ request_uri instead of $ uri.
- Disallow line breaks in the exclusion range, for example, / some / (? [^ / \ S] +) instead of / some / (? [^ /] +.
- It may be a good idea to add $ uri validation (only if you know what you are doing).
Override "parent" response headers with the add_header directive
This is a well-known feature of Nginx that many of us have stumbled over and will continue to stumble over. The point is extremely simple - if you set headings at the same level (for example, in the server section), and a lower level (for example, at the location) sets up some more, the first one will not be applied.
The most simple example is as follows:
In this case, the X-Content-Type-Options response header will not be set when processing a locale /.
Gixy successfully tells you about it:
I know several ways to solve this problem:
- duplicate important headers;
- set the headers at the same level, for example, in the server section;
- consider the option of using the ngx_headers_more module.
Each of them has its advantages and disadvantages. Which one to choose is up to you.
I hope I convinced you that the Nginx configuration requires more attention. I also believe that static analysis of Nginx configurations can work (this also confirms the experience of Nginx Amplify ). Unfortunately, it is not always possible to automatically determine all borderline cases or specific features of the application behind Nginx. So, for example, I didn’t include the X-Forwarded- * request header override check in the standard set, since the response to them depends on the application, and in some cases they cannot be touched at all (for example, in case of multiple proxying). But at home you can do the checks you need based on a deeper understanding of the application. Yes, now Gixy does not know how to identify the whole range of known problems, but he is learning and, perhaps, with your help, he will start doing it better and more fully.
If we talk about usage scenarios, then we have identified several typical cases for ourselves:
- run in a test environment where nginx is installed;
- web application for checking a single block. This is useful when you encounter a suspicious part of the config;
- HTTP API for integration with CI or thin clients.
It seems to us that the most interesting option is using the HTTP API for thin clients. Indeed, in this case, we can centrally manage the checks we need, update them, and so on. Fortunately, modern versions of nginx have the -T switch to test the configuration and dump it, and Gixy can parse this format.
Finally, I would like to emphasize the fact that this is the first public alpha version of Gixy, so the API can be changed without maintaining backward compatibility. In this regard, if you need to implement your own plug-in, it is better to write an Issue or send a Pull Request - then we will come up with something together.
I hope our experience was interesting and useful to you, and, perhaps, even made you reconsider our configurations again;)
But first things first. We have long been tormented by the issue of the secure configuration of Nginx, because it is a full cube of the web application, which means that its configuration requires no less control on our part than the code of the application itself. Last year it became obvious to us that this process requires serious automation. Thus began the in-house Gixy project, the requirements for which we outlined as follows:
')
- be simple;
- but expandable;
- with the possibility of convenient integration into testing processes;
- it would be nice to be able to rekolvit incluses;
- and work with variables;
- and about regular expressions not to forget.
Frankly, we have hesitated to the last with the choice of language (between Golang and Python). As a result, Python was chosen with the hope that it is more common, which means it will be a little easier with development.
About problems
On this, we’ll finish the introduction and go over to examples of common problems :) To avoid confusion in the future, the current mainline version of Nginx, 1.13.0, was used in all examples.
Server-Side-Request-Forgery
Server Side Request Forgery is a vulnerability that allows to perform various kinds of requests on behalf of a web application (in our case, on behalf of Nginx). Occurs when an attacker can control the address of the proxied server, for example, in case of incorrect configuration of the XSendfile .
From my own experience I can say that often the vulnerability is associated with several errors:
- lack of directive internal . Its meaning is to indicate that a particular location can only be used for internal requests;
- unsafe internal redirection.
If everything is clear with the first case, then with the internal redirection things are not so simple. I believe many of you have seen / written a similar configuration:
location ~* ^/internal-proxy/(?<proxy_proto>https?)/(?<proxy_host>.*?)/(?<proxy_path>.*)$ { internal; proxy_pass $proxy_proto://$proxy_host/$proxy_path ; proxy_set_header Host $proxy_host; } Unfortunately, in this configuration you need to check at least all the rewrite and try_files directives, since according to the documentation:
Internal requests are:
- queries redirected by the directives error_page, index, random_index and try_files ;
- requests redirected using the “X-Accel-Redirect” field of the parent server's response header;
- subqueries formed by the “include virtual” command of the ngx_http_ssi_module module and the ngx_http_addition_module module directives;
- queries modified by the rewrite directive.
It turns out that any careless rewrite will allow you to make a request in the internal location. This is fairly easy to verify:
- configuration:
location ~* ^/internal-proxy/(?<proxy_proto>https?)/(?<proxy_host>.*?)/(?<proxy_path>.*)$ { internal; return 200 "proto: $proxy_proto\nhost: $proxy_host\npath: $proxy_path"; } rewrite ^/(?!_api)(.*)/\.files/(.*)$ /$1/.download?file=$2 last; - exploitation:
GET /internal-proxy/http/evil.com/.files/some HTTP/1.0 Host: localhost HTTP/1.1 200 OK Content-Length: 42 Content-Type: application/octet-stream Date: Fri, 28 Apr 2017 13:55:51 GMT Server: nginx/1.13.0 proto: http host: evil.com path: .download In this situation, we usually recommend several practices:
- use only internal location for proxying;
- if possible, prohibit the transfer of user data;
- secure the address of the proxied server:
• if the number of proxied hosts is limited (for example, you have S3), then it is better to hard-code them and select using map or in any other way convenient for you;
• if for some reason it is not possible to list all possible hosts for proxying, it is worth signing.
Bad regular expressions for validating a referrer or origin
You have a problem. You decide to use regular expressions to solve it.
- Now you have two problems.
Often, validation of the “Referer” or “Origin” request header is done using a regular expression. Often this is necessary for conditional exposure of the X-Frame-Options header (ClickJacking protection) or Cross-Origin Resource Sharing ( CORS ) implementation. And if with the validation of “Referer” everything is a bit simpler and, with some conditions, you can refuse the regular expression in favor of the ngx_http_referer_module module, then with “Origin” everything is not so simple.
We distinguish two main classes of problems:
- errors in the compilation of a regular expression;
- permission of untrusted third-party domains.
Problem configuration is as follows:
if ($http_origin ~* ((^https://www\.yandex\.ru)|(^https://ya\.ru)/)) { add_header 'Access-Control-Allow-Origin' "$http_origin"; add_header 'Access-Control-Allow-Credentials' 'true'; } In fact, I have simplified the regular expression very much, but even in this example, seeing the problem from the first time is not so easy. It is easier for people to write regular expressions than to read them.
Fortunately, this problem is not typical for the machine, so Gixy is able to independently determine that this regular expression is www.matdex.ru.evil.com as a valid origin and will inform you about it:
$ gixy --origins-domains yandex.ru,ya.ru /etc/nginx/nginx.conf ==================== Results =================== Problem: [origins] Validation regex for "origin" or "referrer" matches untrusted domain. Description: Improve the regular expression to match only trusted referrers. Additional info: https://github.com/yandex/gixy/blob/master/docs/ru/plugins/origins.md Reason: Regex matches "https://www.yandex.ru.evil.com" as a valid origin. Pseudo config: include /etc/nginx/sites/default.conf; server { server_name _; if ($http_origin ~* ((^https://www\.yandex\.ru)|(^https://ya\.ru)/)) { } } Or, if you assume ya.ru is not trusted enough, it will report the origins of ya.ru and www.yandex.ru.evil.com :
$ gixy --origins-domains yandex.ru /etc/nginx/nginx.conf ==================== Results =================== Problem: [origins] Validation regex for "origin" or "referrer" matches untrusted domain. Description: Improve the regular expression to match only trusted referrers. Additional info: https://github.com/yandex/gixy/blob/master/docs/ru/plugins/origins.md Reason: Regex matches "https://www.yandex.ru.evil.com", "https://ya.ru/" as a valid origin. Pseudo config: include /etc/nginx/sites/default.conf; server { server_name _; if ($http_origin ~* ((^https://www\.yandex\.ru)|(^https://ya\.ru)/)) { } } HTTP Splitting
HTTP Splitting is used to attack an application that is behind Nginx (HTTP Request Splitting) or application clients (HTTP Response Splitting). Vulnerability occurs when an attacker can embed a newline character \ n in a request or response generated by Nginx.
I have no reliable advice (except how to be attentive), but you should always pay attention to several things:
- which variables are used in the directives responsible for creating queries (can they contain CRLF), for example: rewrite, return, add_header, proxy_set_header and proxy_pass;
- whether the variables $ uri and $ document_uri are used, and if so, in which directives, since they are guaranteed to contain the url-decoded value;
- pay special attention to variables derived from groups with an exclusive range: (? P [^.] +).
Exclusive Range Example:
- configuration:
server { listen 80 default; location ~ /v1/((?<action>[^.]*)\.json)?$ { add_header X-Action $action; return 200 "OK"; } } - exploitation:
GET /v1/see%20below%0d%0ax-crlf-header:injected.json HTTP/1.0 Host: localhost HTTP/1.1 200 OK Content-Length: 2 Content-Type: application/octet-stream Date: Fri, 28 Apr 2017 13:57:28 GMT Server: nginx/1.13.0 X-Action: see below x-crlf-header: injected OK As you can see, we were able to add an x-crlf-header response header: injected. This happened due to the coincidence of several circumstances:
- add_header does not encode / validate the values ​​passed to it, considering that the author is aware of the consequences;
- the value of the path is normalized before processing the location;
- the $ action variable was selected from the regular expression group with the exclusive range: [^.] *;
- thus, the value of the $ action variable became equal to see below \ r \ nx-crlf-header: injected and got into the HTTP response.
Fortunately, Gixy successfully copes with this task:
- he knows about "dangerous" variables - more precisely, he knows about the allowable set of characters in most embedded variables. Thus, the difference between $ request_uri and $ uri is obvious to him;
- able to select variables from regular expression groups;
- able to determine whether any character (in our case, \ n) can be matched with a regular expression (or a single group).
Another interesting example is rewright with try_files:
- configuration:
server { listen 80 default; location / { try_files $uri $uri/ /index.php?q=$uri; } location ~ \.php { proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $remote_addr; proxy_set_header Host $host; proxy_pass http://127.0.0.1:9000; } } - operation (on 127.0.0.1:9000 listens to the debug echo-server):
GET /request%20HTTP/1.0%0aInjection: HTTP/1.0 Host: localhost HTTP/1.1 200 Ok Content-Length: 244 Content-Type: text/plain Date: Fri, 28 Apr 2017 13:59:18 GMT Server: nginx/1.13.0 GET /index.php?q=/request HTTP/1.0\n Injection: HTTP/1.0\r\n X-Real-IP: 127.0.0.1\r\n X-Forwarded-For: 127.0.0.1\r\n Host: localhost\r\n Connection: close\r\n User-Agent: HTTPie/0.9.8\r\n Accept-Encoding: gzip, deflate\r\n Accept: */*\r\n \r\n What to do?
- Try to use more secure variables, such as $ request_uri instead of $ uri.
- Disallow line breaks in the exclusion range, for example, / some / (? [^ / \ S] +) instead of / some / (? [^ /] +.
- It may be a good idea to add $ uri validation (only if you know what you are doing).
Override "parent" response headers with the add_header directive
This is a well-known feature of Nginx that many of us have stumbled over and will continue to stumble over. The point is extremely simple - if you set headings at the same level (for example, in the server section), and a lower level (for example, at the location) sets up some more, the first one will not be applied.
The most simple example is as follows:
server { listen 80 default; server_name _; add_header X-Content-Type-Options nosniff; location / { add_header X-Frame-Options DENY; } } In this case, the X-Content-Type-Options response header will not be set when processing a locale /.
Gixy successfully tells you about it:
$ gixy /etc/nginx/nginx.conf ==================== Results =================== Problem: [add_header_redefinition] Nested "add_header" drops parent headers. Description: "add_header" replaces ALL parent headers. See documentation: http://nginx.org/en/docs/http/ngx_http_headers_module.html#add_header Additional info: https://github.com/yandex/gixy/blob/master/docs/ru/plugins/addheaderredefinition.md Reason: Parent headers "x-content-type-options" was dropped in current level Pseudo config: include /etc/nginx/sites/default.conf; server { server_name _; add_header X-Content-Type-Options nosniff; location / { add_header X-Frame-Options DENY; } } I know several ways to solve this problem:
- duplicate important headers;
- set the headers at the same level, for example, in the server section;
- consider the option of using the ngx_headers_more module.
Each of them has its advantages and disadvantages. Which one to choose is up to you.
About Gixy
I hope I convinced you that the Nginx configuration requires more attention. I also believe that static analysis of Nginx configurations can work (this also confirms the experience of Nginx Amplify ). Unfortunately, it is not always possible to automatically determine all borderline cases or specific features of the application behind Nginx. So, for example, I didn’t include the X-Forwarded- * request header override check in the standard set, since the response to them depends on the application, and in some cases they cannot be touched at all (for example, in case of multiple proxying). But at home you can do the checks you need based on a deeper understanding of the application. Yes, now Gixy does not know how to identify the whole range of known problems, but he is learning and, perhaps, with your help, he will start doing it better and more fully.
If we talk about usage scenarios, then we have identified several typical cases for ourselves:
- run in a test environment where nginx is installed;
- web application for checking a single block. This is useful when you encounter a suspicious part of the config;
- HTTP API for integration with CI or thin clients.
It seems to us that the most interesting option is using the HTTP API for thin clients. Indeed, in this case, we can centrally manage the checks we need, update them, and so on. Fortunately, modern versions of nginx have the -T switch to test the configuration and dump it, and Gixy can parse this format.
Judge for yourself how comfortable it is.
$ nginx -T | http -v https://gixy/api/check Content-Type:'application/nginx'
POST /api/check HTTP/1.1
Accept: application/json, */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Content-Length: 959
Content-Type: application/nginx
Host: gixy
User-Agent: HTTPie/0.9.8
# configuration file /etc/nginx/nginx.conf:
user http;
worker_processes 1;
#daemon on;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 65;
gzip on;
access_log /var/log/nginx/access.log combined;
error_log /var/log/nginx/error.log debug;
include sites/*.conf;
}
# configuration file /etc/nginx/mime.types:
types {
text/html html htm shtml;
text/css css;
text/xml xml;
image/gif gif;
image/jpeg jpeg jpg;
application/javascript js;
application/atom+xml atom;
application/rss+xml rss;
}
# configuration file /etc/nginx/sites/default.conf:
server {
listen 80;
return 301 https://some$uri;
}
HTTP/1.1 200 OK
Connection: keep-alive
Content-Encoding: gzip
Content-Type: application/json
Date: Tue, 24 Apr 2017 19:45:57 GMT
Keep-Alive: timeout=120
Server: nginx
Transfer-Encoding: chunked
{
"result": [
{
"auditor": "http_splitting",
"config": "\ninclude /etc/nginx/sites/default.conf;\n\n\tserver {\n\t\treturn 301 https://some$uri;\n\t}",
"description": " (\"\\n\") nginx. : rewrite, return proxy_pass.",
"help_url": "https://wiki/product-security/gixy/httpsplitting/",
"reason": "At least variable \"$uri\" can contain \"\\n\"",
"recommendation": " , (eg \"$request_uri\" \"$uri\").",
"severity": "HIGH",
"summary": " HTTP Splitting"
}
],
"status": "ok",
"warnings": []
}Finally, I would like to emphasize the fact that this is the first public alpha version of Gixy, so the API can be changed without maintaining backward compatibility. In this regard, if you need to implement your own plug-in, it is better to write an Issue or send a Pull Request - then we will come up with something together.
I hope our experience was interesting and useful to you, and, perhaps, even made you reconsider our configurations again;)
Source: https://habr.com/ru/post/327590/
All Articles