Recently, they called me and asked for help to deal with very strange symptoms on the site. And the symptoms looked like this. Different customers called and told that they were on the site in the personal accounts of other random customers. Login by themselves. Log in to your account, and then bam - and already in the account of another client. True, they did not realize that someone else could also be in their personal account.
Let me tell you a “failure story” so that others do not step on this rake.
I will try, as in a detective, to reveal the details of what happened. Particularly clever will guess in the middle of the story where the problem crept in.
The project has user authentication. A classic scheme is used with a randomly generated session ID, which is set in the cookie of the client browser. On the server, session data is stored in memcached, since Application servers are several.
Obviously, clients did not crack each other; they all had the same password suddenly not established. At first glance, some kind of difficulty happened with the user sessions - they started to get into each other’s sessions. But how? If it is not yet clear, then you can try restarting memcached so that all sessions are reset. Yes, not very humane - users will have to re-login. But restarting memcached did not solve the problem.
They began to dig logs and it is really clear that, for example, a certain specific client always logged on to the same IP address, and now suddenly his actions are logged from a heap of different IPs. Also, and more importantly, in the logs for authentication (when the client enters the username and password), it is clear that this client logged in only from his usual IP address. We conclude that when a client becomes another client, this does not happen through authentication, but it specifically falls into the session of another client (facepalm).
Code changed? No, they didn't release anything. Servers reconfigured? Well…. only included in Nginx caching of images that were not given as static (static has long been cached), but generated on the fly. But it does not change. Those. there is an endpoint of the form: / path / image? id = NNNN In fact, this is almost like static, because a picture with a specific id will never change. Therefore, in order to optimize the resources, the Application servers decided to cache such requests.
Since I wrote about it, it is important. And since nothing else has changed, it is almost obvious that the problem is this caching. Of course, caching was disabled, but the problem did not go away.
So how did such caching arrange a leapfrog on the site?
Here are some more introductory (implementation details).
Here you can already guess what happened ...
And the following happened. An unauthorized client came without a session cookie. The GET request for the image / path / image? Id = NNNN , which in response generated the session ID and, “ tear the hair on itself ”, has been cached! Yes, I was cached along with the Set-Cookie header, which sets a cookie with a specific session ID. Further other victims come, receive the same picture together with ONE AND THE SAME session ID , which they are forcibly rewritten in the browser. As a result, we have a bunch of users who have the same session ID in browsers in cookies.
It turns out that the client is authorized - its session. Then another client is authorized and now in this session we have a second client. The first is enough to update the page and it is already in the session of the second client. Now imagine the madness that was observed when a bunch of clients "lived" in one session.
Well, you understand that one of the goals of viruses is to steal cookies from session IDs so that an attacker could remotely replace my session and work on behalf of the victim. And then you do not need to steal anything - the session ID is the same for everyone. The site went into "single user" mode.
Yes, of course, you also need to bind the session to an IP address, User Agent, etc., then such a mess would not happen. But what was not, that was not done.
But the story does not end there. An inquisitive reader had to go read the Nginx documentation about caching and find this:
http://nginx.org/ru/docs/http/ngx_http_proxy_module.html#proxy_cache_valid
A response with a “Set-Cookie” in its header will not be cached.
Yeah, why in this case is Set-Cookie cached? Here is one more nuance of the project: the engine in each answer makes a Set-Cookie for other needs, not related to the session ID. Including when gives the picture. Therefore, to make caching work, the following parameter was present in the Nginx config:
http://nginx.org/ru/docs/http/ngx_http_proxy_module.html#proxy_ignore_headers
proxy_ignore_headers Cache-Control Expires Set-Cookie;
It is because of this cookie that it was cached.
In order for the story to be instructive, you need to talk about how to do it right in terms of Nginx settings. Specifically in this case, the addition
http://nginx.org/ru/docs/http/ngx_http_proxy_module.html#proxy_hide_header
proxy_hide_header Set-Cookie;
should solve the problem.
PS What is Backend written on? I think it does not matter. "Auto-session" can be present on any platform. Sometimes it is useful and simplifies life, and sometimes it is dangerous. Think with your head and everything will be fine. The story is more about the features of setting up Nginx.
PPS This is not a success story. This is a story of stuffing cones. The project developers are aware of the weaknesses in the architecture that led to the incident described. Therefore, a big request in the comments not to develop the theme of what developers and admins are stupid, and you with 20 years of experience would have foreseen everything and did not step on this rake.
Source: https://habr.com/ru/post/341590/
All Articles