Watch vBulletin or attempt to cache dynamic content
There are several VPS in my charge, on which it is spinning ... in general, it is not my area of responsibility, and therefore it is spinning there, which is spinning, moderately slowing down, moderately working. And it turned out that some forum was spinning on one of them, and began to slow down the forum. And I wanted to figure it out ...
I lived a forum for myself, did not hurt it, showed xm top load in the region of 30-40 percent. And then the hour "X" came and the load jumped to a flat shelf 90 percent with peaks higher, which, in general, is not gud. DDOS suspicion was not confirmed. The logs were observed normal workload. Well, before stupidly increasing resources, the idea arose to understand what is happening and try to cache everything that is possible.
Investigation. Part One - What Does
Since I was not familiar with the ideology and features of this software, I began to study the problem of analyzing the logs and traffic between visitors and the server. First of all, I was surprised to find that attachments to messages in the forum are given exclusively by the attachment.php script, while the files themselves can be stored in the database, can be on a local disk, but the return is only through a script. And nothing else. That is, we get 8-10 extra twitching php-interpreter on the message branch with 8-10 photos. And this is for every visitor. Since this forum does not require registration to view attachments, attachments can be cached, for example, for a couple of days. Like this:
')
The second “revelation” for me was that there are archives on the forum, and they do not just exist, but almost half of the requests fall on them. The appearance of the pages also allows you to cache their contents:
On the formation of the key "$ psUID" I will tell further.
From the point of view of the visitor of the forum, the visitor can be either a registered user or a guest. But the situation is completely different if we observe the situation “came, walked, logged in, walked, logged off, walked” in terms of the appearance and disappearance of cookies in the browser. So, we clear cookies for the domain and its subdomains, open HTTPfox and see what happens:
Everything is clear with uid and PHPSESSID - these are the intrigues of nginx and the php interpreter with the session.auto_start option set, and the rest are the witnesses for forum activity. But the main session cookie vBulletin is not yet observed. Looking ahead, I will say that vBulletin does not use the standard php session (or, more precisely, ALMOST does not use), but maintains its own, which identifier stores in the bbsessionhash cookie . So, the user has logged in, but there is no session - that is, he is an anonymous user without a session. In this case, links to the forum can then be of two kinds (meaning all the links on the page, and not one way, and the other):
forum.domain.com/forumdisplay.php?s=12b66e447be52ebc84ab16d3f39626fb&f=69
forum.domain.com/forumdisplay.php?f=69
And if you follow the link of the first type, then the next answer from the forum will come in a session cookie, and if the link of the second type is not. If you didn’t come across the forum with the second answer from the session, then you can wander around the forum unsessionally and restless until you come across a link of the first type (the pattern of their appearance didn’t come out), or you want to login. With a successful login Cook session will come in any way. If the guest was anonymous-with-session before login, then the session will be replaced. It looks like this:
After login, the session is “stable” and there is no link messaging. The logout procedure is not different by originality - all existing forum cookies are deleted (even those that have not been registered) and a new ("anonymous") session is written:
That is, at the output we get an anonymous user (guest), but one hundred percent having a session.
As a result, from the point of view of forum software and HTTP headers, we have three types of users: a guest without a session , a guest with a session , a logged in visitor . And at the level of nginx'a to distinguish the second from the third extremely problematic.
Now, by understanding what cookies and how to ply between the visitor and the server, you can approach the issue of caching dynamic content. As is known, the caching functionality for fastcgi-backend responses in nginx is built into the ngx_http_fastcgi_module module. To do this, you need to register globally in the http-section a caching zone, and in the correct location'e - the key. And if for conditionally static content (images, archives) the key for caching could be considered a URI with minor additions, then for caching the dynamics it is necessary to take into account the user as well. It seemed to be a type rule
could satisfy both guests and logged in users; however, in practice, visitors began to receive the contents of someone else's cache. Caching the "true" dynamics had to be turned off. I hope the sentence is not final.
However, this information is not useless. Based on it, we can generate a key to limit the frequency of requests based not only on the visitor’s IP address, but also on its status.
This config fragment is placed in the server section of the nginx config file to describe all the locations. As a result, we get the original key for a user with a session and a key based on the IP address for session visitors who do not (for example, a search crawler).
As a result of the efforts made, the total load on the virtual machine dropped from the shelf by 90 percent to 40% with a surge to 80 percent.
Ixd
- Forum under vBulletin 3.8.x
- Rendered to the subdomain forum.domain.com
- Nginx 1.1.13, php 5.3.x (fpm)
- In addition to the forum on this server, nothing is spinning. ( this is important ).
- Mysql on a separate server, communication via TCP / IP.
Prehistory
I lived a forum for myself, did not hurt it, showed xm top load in the region of 30-40 percent. And then the hour "X" came and the load jumped to a flat shelf 90 percent with peaks higher, which, in general, is not gud. DDOS suspicion was not confirmed. The logs were observed normal workload. Well, before stupidly increasing resources, the idea arose to understand what is happening and try to cache everything that is possible.
Investigation. Part One - What Does a Visitor Woman Want?
Since I was not familiar with the ideology and features of this software, I began to study the problem of analyzing the logs and traffic between visitors and the server. First of all, I was surprised to find that attachments to messages in the forum are given exclusively by the attachment.php script, while the files themselves can be stored in the database, can be on a local disk, but the return is only through a script. And nothing else. That is, we get 8-10 extra twitching php-interpreter on the message branch with 8-10 photos. And this is for every visitor. Since this forum does not require registration to view attachments, attachments can be cached, for example, for a couple of days. Like this:
location = /attachment.php { expires max; limit_req zone=lim_req_1s_zone burst=5; fastcgi_pass forum__php_cluster; include /etc/nginx/fastcgi_params; include /etc/nginx/fastcgi_params_php-fpm; fastcgi_cache forum_att__cache; fastcgi_ignore_headers Cache-Control Expires Set-Cookie; fastcgi_hide_header Set-Cookie; fastcgi_hide_header Pragma; fastcgi_cache_key "$request_method:$http_if_modified_since:$http_if_none_match:$host:$request_uri:"; fastcgi_cache_use_stale updating error timeout invalid_header http_500; fastcgi_cache_lock on; fastcgi_cache_lock_timeout 2m; fastcgi_cache_valid 2d; } - http- forum_att__cache: fastcgi_cache_path /var/cache/nginx/att levels=1:2 keys_zone=forum_att__cache:4m max_size=2g inactive=2d; ')
The second “revelation” for me was that there are archives on the forum, and they do not just exist, but almost half of the requests fall on them. The appearance of the pages also allows you to cache their contents:
location /archive/ { expires 10d; limit_req zone=lim_req_1s_zone burst=2; location ~ \.css$ { expires max; } fastcgi_pass forum__php_cluster; fastcgi_index index.php; include /etc/nginx/fastcgi_params; include /etc/nginx/fastcgi_params_php-fpm; fastcgi_param SCRIPT_FILENAME $document_root/archive/index.php; fastcgi_param SCRIPT_NAME $fastcgi_script_name; fastcgi_cache forum_arc__cache; fastcgi_hide_header Set-Cookie; fastcgi_ignore_headers Cache-Control Expires Set-Cookie; fastcgi_cache_key "$request_method:$http_if_modified_since:$http_if_none_match:$host:$request_uri:"; fastcgi_cache_use_stale updating error timeout invalid_header http_500; fastcgi_cache_valid 2d; } http-: fastcgi_cache_path /var/cache/nginx/arc levels=1:2 keys_zone=forum_arc__cache:4m max_size=2g inactive=2d; DDOS-: limit_req_zone "$psUID" zone=lim_req_1s_zone:2m rate=1r/s; On the formation of the key "$ psUID" I will tell further.
Investigation. Part Two - Authorization in vBulletin
From the point of view of the visitor of the forum, the visitor can be either a registered user or a guest. But the situation is completely different if we observe the situation “came, walked, logged in, walked, logged off, walked” in terms of the appearance and disappearance of cookies in the browser. So, we clear cookies for the domain and its subdomains, open HTTPfox and see what happens:
HTTP/1.1 200 OK Set-Cookie: PHPSESSID=cdme9rrptft67tbo97p4t1cua5; expires=Wed, 22-Feb-2012 15:04:12 GMT; path=/; domain=.domain.com Set-Cookie: bblastvisit=1329059052; expires=Mon, 11-Feb-2013 15:04:12 GMT; path=/; domain=.domain.com Set-Cookie: bblastactivity=0; expires=Mon, 11-Feb-2013 15:04:12 GMT; path=/; domain=.domain.com Set-Cookie: uid=XCuiGU831OyC8VLqAx/QAg==; expires=Thu, 31-Dec-37 23:55:55 GMT; domain=.domain.com; path=/ Everything is clear with uid and PHPSESSID - these are the intrigues of nginx and the php interpreter with the session.auto_start option set, and the rest are the witnesses for forum activity. But the main session cookie vBulletin is not yet observed. Looking ahead, I will say that vBulletin does not use the standard php session (or, more precisely, ALMOST does not use), but maintains its own, which identifier stores in the bbsessionhash cookie . So, the user has logged in, but there is no session - that is, he is an anonymous user without a session. In this case, links to the forum can then be of two kinds (meaning all the links on the page, and not one way, and the other):
forum.domain.com/forumdisplay.php?s=12b66e447be52ebc84ab16d3f39626fb&f=69
forum.domain.com/forumdisplay.php?f=69
And if you follow the link of the first type, then the next answer from the forum will come in a session cookie, and if the link of the second type is not. If you didn’t come across the forum with the second answer from the session, then you can wander around the forum unsessionally and restless until you come across a link of the first type (the pattern of their appearance didn’t come out), or you want to login. With a successful login Cook session will come in any way. If the guest was anonymous-with-session before login, then the session will be replaced. It looks like this:
HTTP/1.1 200 OK Set-Cookie: bbsessionhash=85745bc6110db5221e159087bf037f24; path=/; domain=.domain.com; HttpOnly After login, the session is “stable” and there is no link messaging. The logout procedure is not different by originality - all existing forum cookies are deleted (even those that have not been registered) and a new ("anonymous") session is written:
HTTP/1.1 200 OK Set-Cookie: bbsessionhash=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; path=/; domain=.domain.com Set-Cookie: bblastvisit=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; path=/; domain=.domain.com Set-Cookie: bblastactivity=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; path=/; domain=.domain.com Set-Cookie: bbthread_lastview=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; path=/; domain=.domain.com Set-Cookie: bbreferrerid=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; path=/; domain=.domain.com Set-Cookie: bbuserid=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; path=/; domain=.domain.com Set-Cookie: bbpassword=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; path=/; domain=.domain.com Set-Cookie: bbthreadedmode=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; path=/; domain=.domain.com Set-Cookie: bbstyleid=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; path=/; domain=.domain.com Set-Cookie: bblanguageid=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; path=/; domain=.domain.com Set-Cookie: bbsessionhash=3d0bdc5dbe8dabae361deebe8f6048d2; path=/; domain=.domain.com; HttpOnly That is, at the output we get an anonymous user (guest), but one hundred percent having a session.
As a result, from the point of view of forum software and HTTP headers, we have three types of users: a guest without a session , a guest with a session , a logged in visitor . And at the level of nginx'a to distinguish the second from the third extremely problematic.
Now, by understanding what cookies and how to ply between the visitor and the server, you can approach the issue of caching dynamic content. As is known, the caching functionality for fastcgi-backend responses in nginx is built into the ngx_http_fastcgi_module module. To do this, you need to register globally in the http-section a caching zone, and in the correct location'e - the key. And if for conditionally static content (images, archives) the key for caching could be considered a URI with minor additions, then for caching the dynamics it is necessary to take into account the user as well. It seemed to be a type rule
fastcgi_cache_key "$request_method:$http_if_modified_since:$http_if_none_match:$host:$request_uri:$cookie_bbsessionhash:"; could satisfy both guests and logged in users; however, in practice, visitors began to receive the contents of someone else's cache. Caching the "true" dynamics had to be turned off. I hope the sentence is not final.
However, this information is not useless. Based on it, we can generate a key to limit the frequency of requests based not only on the visitor’s IP address, but also on its status.
set $psUID "anon"; set $psUCL "anon"; if ($cookie_bbsessionhash) { set $psUID "$cookie_bbsessionhash"; set $psUCL "user"; } if ($psUCL = "anon") { set $psUID "anon:$remote_addr"; } This config fragment is placed in the server section of the nginx config file to describe all the locations. As a result, we get the original key for a user with a session and a key based on the IP address for session visitors who do not (for example, a search crawler).
results
As a result of the efforts made, the total load on the virtual machine dropped from the shelf by 90 percent to 40% with a surge to 80 percent.
Source: https://habr.com/ru/post/138057/
All Articles