Introduction

It's funny, but when a programmer develops a product, he rarely thinks about the question of whether 2,000 people can simultaneously press one button at a time. And in vain. It turns out they can. Oddly enough, most of the engines written by such programmers behave very badly under heavy loads. Who would have thought, and just one extra INSERT, not marked by an index, or a curve, a recursive function can raise the load averages by almost an order of magnitude.

In this article I will describe how we, the developers of the project, managed to squeeze out of a single server with a Pentium 4 HT / 512Mb RAM, maximum, holding simultaneously 700+ users on the forum and 120,000 on the tracker. Yes, this project is a torrent tracker. I propose to immediately leave aside the talk about copyrights and rights, I'm not interested in it, which is really interesting - this is HighLoad.

To begin, I will describe the project as it was:

The usual torrent tracker on the TorrentPier engine (it is also in phpbb 2.x)

• Server on FreeBSD 6.0
• Pentium 4 HT / 512Mb RAM
• Apache web server
• MySQL database
• All PHP logic

That is almost LAMP
')
In short, I will immediately write down the steps we have taken:

• Installing on the opcode cache server
• Replacing apache with nginx
• Cache some intermediate samples in NOT RDBMS
• Translation of the key part (read tracker) to C ++
• Optimization of the FreeBSD network stack, as well as its update to the latest -STABLE
• MySQL optimization
• BB caching
• Correspondence code to use SphinxSearch
• Code profiling and installation of monitoring tools
• Disassembling queries from MySQL slow query log

Now about each item in more detail

Installing on the opcode cache server

He is always needed! Installing php-cache gave 300% + performance, spending 15 minutes of time.
Caches are different: eAccelerator , xCache , APC , etc ... We stopped at the last, because of the good speed and the ability to store user data in it

Replacing apache with nginx

Apache is heavy and slow, first stood as the main web server, then nginx was put in front of it, giving static and compressing answers with gzip. Further, apache was abandoned altogether in favor of the nginx + php-fpm bundle (to be exact at that time, it was spawn_fcgi, but now this option is better). A bunch in those days was not the most popular for production, but it worked great for us!

Cache some intermediate samples in NOT RDBMS

RDBMS is evil. It is convenient, but you have to pay for convenience. In this case, speed. And we just need it. So, we backed up some of the results of the most popular and not critical to the relevance of muscle queries to APC. Immediately anticipating a lot of questions why not memcached ... How would you answer ... I'm already tired of even hearing the word memcached, memcached, memcached as if it were a panacea for everything. It does not offer the last time except that only from diarrhea. In our case, the choice fell on the APC because it does not use a TCP connection and because of this it works many times faster. For the time being, everything is fine on one server and distributed storage is not so necessary for us.
You can choose any other key / value storage that does not necessarily store data in RAM .
But it is highly likely that in your case memcached / memcachedb / memcacheQ would be the best option.

In general, there was an idea to make a multilevel cache layer in which php looked for the value in global variables, then in APC, then in memcached, and only then crawls into the database with a SELECT. But since we are engaged in the project in free time from studies / work / family time, it has not yet reached this point.

Translation of the key part (read tracker) to C ++

120000 nginx, , php, . ? . XBTT TorrentPier'. , 2710 , , , . . . +100000% .
, — 1
input (rl0) output
packets errs bytes packets errs bytes colls drops
20K 0 2.5M 16K 0 1.5M 0 0

PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
10 root 1 171 52 0K 8K RUN 1 538.6H 47.12% idle: cpu1
6994 root 1 108 0 98140K 96292K CPU0 0 3:57 33.98% xbt_tracker
11 root 1 171 52 0K 8K RUN 0 595.0H 31.20% idle: cpu0
35 root 1 -68 -187 0K 8K WAIT 0 17.1H 21.14% irq21: rl0
12 root 1 -44 -163 0K 8K WAIT 0 482:57 9.96% swi1: net

[root@****] /usr/ports/devel/google-perftools/> netstat -an | wc -l
24147
100 30% . , 8

FreeBSD, -STABLE

FreeBSD 6 4BSD, 7 ULE, SMP
FreeBSD sysctl,

MySQL

, , .
myisam

• -
• FULLTEXT

. tuning-primer.sh
Xtradb. — =)

BB-

phpbb « » bbcod' html. . html /. 2 , .

SphinxSearch

- , . innodb Master-MultipleSlaves myisam, . , , . , , SphinxSearch. . .
-- rss , .

, . . bottleneck . php hook' , munin.

MySQL slow query log

! 20% 80% . . , FORCE INDEX php , 10(!!) .
- . mysqlsla.

LAMP . Core2Duo 2 3 , «», , 1.5 200000 ~500 . LAMP ?
.

, . , !

UPD: