Hello. On Friday, we launched the race for the social network vKontakte - TurboGonki. You can play them here:
http://vkontakte.ru/turboraces , and they look something like this:

It is still a game in beta test mode, so you may feel hatred for the author of the topic, an irreconcilable desire to throw out your browser and even be sure that you will find some bugs, but you are welcome to contact us at light.
What is it written on?
')
The client part is ActionScript, the implementation of physics is Box2D. The server side that controls the race process is Qt, the menu, the store and other trivia is PHP (CodeIgniter), the database is MySQL.
How we started
The game was written for a long time. Indecently long. I am even ashamed to call these terms, so I will not. Several developers changed, in my memory the project was frozen 2 times and the release date was postponed 3 times. In the end, they decided to be released and launched last Thursday. To be precise, on Friday, because Day application hung on moderation.
Users started to come to the game little by little, they started to create races and play and we realized that we were hit. The number of bugs turned out to be 4 times higher than we expected and even those that couldn’t manifest at all got out - Murphy's law is in force. The main thing was that it was possible to play, despite them (we fixed the critical ones, stabilized the game a little).
After looking at it, we decided to run around a small number of users over the weekend (established thousands and a half during the first day), identify most of the problems, and since Monday fix them and in a week have already been actively promoting.
But on Saturday, we decided to try to test ads on vKontakte. The traffic crawled little by little, the experiments brought the cost of installing the application from 1.5 rubles to 0.4, which is just great, it seems to me, and then it was 11 pm and the traffic flowed like a river.
What is the reason - hard to say, because there was no time to analyze. I pass on the sad experience to the next generations - NEVER run with social networking apps when Apache is on the server. Hmm, in general, because we did not expect a sharp influx of traffic, the server was with the usual default settings, there was even a shop spinning there. Yes, 8 GB of RAM, 80 SSD and some 4-core processor, but all this did not help, because hanged a web server. We directly imagined how Apache sadly looks out of the window, sees a crowd of enraged pale faces, turns around and sadly throws a noose on a hook.
Quickly deployed nginx, at least as a proxy in front of Apache - it helped, it got a little prettier. In the morning I already understood that it was necessary for workers to clearly not install 2, and to increase the number of processes, but the default config also saved. For a while.
For the next hour traffic went tight, then slept. As a result, I don’t remember how in the first hour, but when I went to bed at 5 am, having already decided to move to the clouds tomorrow, there were 7,000 users in the application.
At 10 o'clock I was woken up by a call that the server had gone down again. Having updated the page with the counter, I, to put it mildly, was surprised - there were already 14,000 users (x2 in 5 hours) and they were streaming further. It turned out to be easier here - I just adjusted the nginx configs and the server boldly ran on. Over the next 5 hours it was x2 again - at 3 pm we had 28 thousand users. Further already let.
What does architecture look like now?
On Sunday we settled in the “Oversan-Skalaksi”. Why there? Yes, I simply have no idea about other Russian clouds, and in the States or Europe I was confused by pings. They put a load balancer and 2 servers behind it, it looks like this in the control panel:

The base and the game server are not yet in the clouds, while on a more powerful dedicated server in another data center.
On the balancer put HAProxy, which distributes the load between the servers behind it. The config is almost default and lies here:
http://pastebin.com/xZ7Prc3u , I saw it for the first time on Sunday, so there was no time to understand very deeply, if someone tells me how to do better, I will listen with pleasure.
The web backends are nginx + php-cgi. scaling by means of the Skalaxi panel is only still being understood, so it’s hard to say how it works)
Thanks to everyone who read these 8 kilometers of the text, did not say “fu, VKontakte” and did not put a minus :)
I am pleased to hear comments, suggestions, recommendations, advice.
Finally a screenshot of the game process at the request of the comments:
