Locking sessions in web projects - choosing an effective weapon
Hello!
Recently, due to the rapid growth and complication of front-end, Ajax, etc. - Increasingly, the problem of blocking sessions during the operation of sites on PHP. PHP by default creates a file for the session and the process blocks it exclusively. The rest of the processes that are trying to open a session (Ajax, tabs in the browser) - are lined up. Not always the application logic, especially if it is complex, allows you to effectively limit the blocking time of processes competing for a session.
The situation is aggravated by the fact that 3-5 such clients are able to quickly score the PHP-workers who are stuck and idle while waiting for the processes and the site becomes bad, if not very much said.
')
Unfortunately, developers / system administrators are not always able to immediately understand that it is a matter of blocking the session - and they are looking for problems in other parts of the project, losing time.
The article will tell you what tools allow you to quickly diagnose a problem, give a working code and give some combat survival recommendations :-)
I deliberately do not complicate the article and do not talk about the theory and practice of writing custom PHP session handlers - this is a separate interesting topic. Focus on a specific problem and try to solve it.
Consider what happens inside the operating system, if you simultaneously try to open one falling asleep file and several scripts that simply start a session in the browser (in different tabs):
Pages will wait for the release of the session (30 seconds) and it will take oh oh oh oh time, the slots of the web server will be clogged. Approximately the same thing happens when an AJAX launches a heavy task in a web client session and the remaining AJAXs and other interface elements hang pending (or when several tabs are opened under one authorization).
Web server processes, in this case httpd, but the same thing happens with php-fpm — they try to exclusively block the session file, which we see with lsof:
Pay attention to the 4 column. The number is the number of the file descriptor in the process, and then the type of lock. “UW” ​​- the web server has locked the file exclusively for writing. The rest are waiting and smoking nervously on the sidelines :-) As soon as process 7079 finishes its work, the “uW” lock will be taken by another process. At this time, of course, a queue is being built and the web interface is noticeably slow. Even more fun, if several processes block a session for a few seconds, the interface will generally become a stake.
Now let's look at the other side of what the processes do:
In the second column, we see that all but one are occupied in the function “flock_lock_file_wait” . With what?
That's right, in a system call with an exclusive lock request.
To keep track of the emergence of such a “train” that clogs PHP-workers at web servers, I wrote a simple script on AWK:
Runs like this:
Displays the length of the "train" and the process - creating a traffic jam.
It is clear that there should be no such plugs in the battle - you need either to alter the logic of working with the session, write custom PHP handlers - to do everything so that the client doesn’t slow down as much as possible, and you, as a system administrator, slept well and for a long time.
If it's very lazy (am I really the only one :-)), you can uncomment “kill” and shoot web server processes that create collapseand enjoy the reaction of customers and technical support managers :-) But it is more correct to buy 2-3 cans of beer and visit the developers - with statistics collected in a similar way via cron and agree on refactoring :-)
Good luck and success to all!
PS
At the request of teachers of the Russian language and programmers with a philological education, replaced the word "lock" with "block".
Supporting large projects of our company, we have to constantly create tools and techniques for quick analysis of performance problems and their solutions. GNU / Linux contains a large set of useful tools, but, unfortunately, not all of them know how to use them. I hope these practical articles will be useful not only for system administrators, but also for web developers.
Recently, due to the rapid growth and complication of front-end, Ajax, etc. - Increasingly, the problem of blocking sessions during the operation of sites on PHP. PHP by default creates a file for the session and the process blocks it exclusively. The rest of the processes that are trying to open a session (Ajax, tabs in the browser) - are lined up. Not always the application logic, especially if it is complex, allows you to effectively limit the blocking time of processes competing for a session.
The situation is aggravated by the fact that 3-5 such clients are able to quickly score the PHP-workers who are stuck and idle while waiting for the processes and the site becomes bad, if not very much said.
')
Unfortunately, developers / system administrators are not always able to immediately understand that it is a matter of blocking the session - and they are looking for problems in other parts of the project, losing time.
The article will tell you what tools allow you to quickly diagnose a problem, give a working code and give some combat survival recommendations :-)
I deliberately do not complicate the article and do not talk about the theory and practice of writing custom PHP session handlers - this is a separate interesting topic. Focus on a specific problem and try to solve it.
Diagnostics
Consider what happens inside the operating system, if you simultaneously try to open one falling asleep file and several scripts that simply start a session in the browser (in different tabs):
<?php session_start(); sleep(30);// ?> Pages will wait for the release of the session (30 seconds) and it will take oh oh oh oh time, the slots of the web server will be clogged. Approximately the same thing happens when an AJAX launches a heavy task in a web client session and the remaining AJAXs and other interface elements hang pending (or when several tabs are opened under one authorization).
Web server processes, in this case httpd, but the same thing happens with php-fpm — they try to exclusively block the session file, which we see with lsof:
lsof -n | awk '/ sess_ /' httpd 7079 nobody 52uW REG 8.1 2216 809832 / tmp / sess_f629a13b4b0920a21042c86d17f4a6a6 httpd 10406 ​​nobody 52u REG 8.1 2216 809832 / tmp / sess_f629a13b4b0920a21042c86d17f4a6a6 httpd 10477 nobody 52u REG 8.1 8.116 809832 / tmp / sess_f629a13b4b0920a21042c86d17f4a6a6 httpd 10552 nobody 52u REG 8.1 2216 809832 / tmp / sess_f629a13b4b0920a21042c86d17f4a6a6 httpd 11550 nobody 52u REG 8.1 2216 809832 / tmp / sess_f629a13b4b0920a21042c86d17f4a6a6 httpd 11576 nobody 52u REG 8.1 8.116 809832 / tmp / sess_f629a13b4b0920a21042c86d17f4a6a6
Pay attention to the 4 column. The number is the number of the file descriptor in the process, and then the type of lock. “UW” ​​- the web server has locked the file exclusively for writing. The rest are waiting and smoking nervously on the sidelines :-) As soon as process 7079 finishes its work, the “uW” lock will be taken by another process. At this time, of course, a queue is being built and the web interface is noticeably slow. Even more fun, if several processes block a session for a few seconds, the interface will generally become a stake.
Now let's look at the other side of what the processes do:
ps -e -o pid, comm, wchan = WIDE-WCHAN-COLUMN | grep httpd 7079 httpd - 10406 ​​httpd flock_lock_file_wait 10477 httpd flock_lock_file_wait 10552 httpd flock_lock_file_wait 11550 httpd flock_lock_file_wait 11576 httpd flock_lock_file_wait
In the second column, we see that all but one are occupied in the function “flock_lock_file_wait” . With what?
strace -p 10406 Process 10406 ​​attached - interrupt to quit flock (52, LOCK_EX)
That's right, in a system call with an exclusive lock request.
LOCK_EX Place an exclusive lock. Only one process may hold an
exclusive lock for a given file at a given time.
Useful script
To keep track of the emergence of such a “train” that clogs PHP-workers at web servers, I wrote a simple script on AWK:
/sess_/ { load_sessions[$9]++; if (load_sessions[$9]>max_sess_link_count){ max_sess_link_count = load_sessions[$9]; max_sess_link_name = $9; }; if ($4 ~ /.*uW$/ ){ locked_id[$9]=$2 }; } END { print max_sess_link_count, max_sess_link_name,locked_id[max_sess_link_name]; if (locked_id[max_sess_link_name] && max_sess_link_count>3) { # r=system("kill "locked_id[max_sess_link_name]); # if (!r) print "Locking process "locked_id[max_sess_link_name]" killed" system("ls -al "max_sess_link_name); } } Runs like this:
lsof -n | awk -f sess_view.awk 5 / tmp / sess_f629a13b4b0920a21042c86d17f4a6a6 24830
Displays the length of the "train" and the process - creating a traffic jam.
It is clear that there should be no such plugs in the battle - you need either to alter the logic of working with the session, write custom PHP handlers - to do everything so that the client doesn’t slow down as much as possible, and you, as a system administrator, slept well and for a long time.
If it's very lazy (am I really the only one :-)), you can uncomment “kill” and shoot web server processes that create collapse
Good luck and success to all!
PS
At the request of teachers of the Russian language and programmers with a philological education, replaced the word "lock" with "block".
Supporting large projects of our company, we have to constantly create tools and techniques for quick analysis of performance problems and their solutions. GNU / Linux contains a large set of useful tools, but, unfortunately, not all of them know how to use them. I hope these practical articles will be useful not only for system administrators, but also for web developers.
Source: https://habr.com/ru/post/179803/
All Articles