Architecture and Odnoklassniki project platform

Architecture and Odnoklassniki project platform

In this post we will tell about the experience accumulated over 5 years in maintaining a high-load project. We hope that our fellow developers will be interested to know what we are doing and how, what problems and difficulties we face and how we cope with them.

Basic statistics

Up to 2.8 million users online during peak hours
7.5 billion requests per day (150,000 requests per second during peak hours)
2,400 servers, storage systems
Network traffic during peak hours: 32 Gb / s

Architecture

Puff architecture:

• presentation layer (presentation layer or simply WEB servers forming HTML)
• business services layer (servers providing data collection and processing)
• caching layer (caching frequently used data)
• persistence layer (database server)
• common infrastructure systems (statistics logging systems, application configurations, resource localization, monitoring)
')

Presentation Layer:

• We use our own framework, which allows you to build a composition of pages in JAVA, using your own GUI factories (text design, lists, tables, portlets).
• Page composition consists of independent blocks (usually portlets), which allows you to update the information on the screen in parts using AJAX requests. This approach to navigation allows you to get rid of the constant page reloads, thus the important functions of the site (Messages, Discussions and Alerts) are always available to the user. Without javascript, the page is fully functional, except for the functionalities written in GWT - when you follow links, it just completely redraws.
• Functional components like Messages, Discussions and Alerts, as well as all the dynamic parts (shortcut menu, photo tags, photo sorting, gift wrapping) are written using the Google Web Toolkit framework.

Selection, processing and caching data:

The code is written in Java. There are exceptions - some modules for data caching are written in C and C ++.
Java because it is a convenient language for development, a lot of developments in various fields, libraries, open source projects in Java.
At the level of business logic, there are about 25 types of servers / components and caches that communicate with each other through remote interfaces. Every second, there are about 3,000,000 remote requests between these modules.
For caching data, the odnoklassniki-cache self-written module is used. It provides the ability to store data in memory using Java Unsafe. Cache all data that is accessed frequently. For example: information from user profiles, user groups, information about the groups themselves, of course, a graph of user connections, a graph of user connections and groups, user holidays, some meta information about photos, etc.
For example, one of the servers that caches the graph of user connections during peak hours can handle about 16,600 requests per second. The CPU is busy up to 7%, the maximum load average for 5 minutes is 1.2. The number of vertices of the graph is> 85 million, with connections of 2,500 million (two and a half billion). In memory, the graph is 30 GB.

Distribution and load balancing:

• weighted round robin inside the system;
• vertical and horizontal partitioning of data both in databases and at the cache level;
• servers at the level of business logic are divided into groups. Each group handles a different event. There is an event routing mechanism, i.e. Any event (or group of events) can be selected and sent for processing to a specific group of servers.
Services are managed through a centralized configuration system. The system is self-written. Through the WEB interface, you can change the location of portlets, cluster configuration, change the logic of some services, etc. The changed configuration is saved in the database. Each server periodically checks to see if there are updates for applications running on it. If is - applies them.

Data, database server, backup copies:

The total amount of data without reservation is 160 TB. Two solutions are used for storing and serving data - MS SQL and BerkeleyDB. Data is stored in at least two copies. Depending on the types of data, there may be two to four copies. There is a daily backup of all data. Every 15 minutes, backup copies of the accumulated data are made. As a result of this backup strategy, the maximum possible data loss is 15 minutes.

Equipment, data centers, network:

Used dual-processor, 4-core server. Memory capacity from 4 to 48 GB, depending on the functionality. Depending on the types and uses of data, they are stored either in the memory of servers, or on server disks, or on external storage systems.
All equipment is located in 3 data centers. A total of about 2,400 servers and storage systems. Data centers are combined into an optical ring. At the moment, on each of the routes capacity is 30 Gb / s. Each route consists of physically independent fiber pairs. These pairs are aggregated into a common “pipe” on root routers.
The network is divided into internal and external. Networks are physically separated. Different server interfaces are connected to different switches and work in different networks. On the external network of the WEB server, communicate with the world. On the internal network, all servers communicate with each other.
The internal network topology is a star. Servers are connected to L2 switches (access switches). These switches are connected by at least two gigabit links to the agregation stack of routers. Each link goes to a separate switch in the stack. In order for this architecture to work, we use the RSTP protocol. If necessary, access switches connect to the agregation stack with more than two links. Then link aggregation of ports is used.
Agregation switches are connected by 10Gb links to the root routers, which provide both communication between data centers and communication with the outside world.
Switches and routers from Cisco are used. For communication with the outside world, we have direct connections with several major telecom operators.
Network traffic during peak hours - 32 Gb / s

Statistics system:

There is a library responsible for logging events. The library is used in all modules. It allows you to aggregate statistics and save them in a temporary database. The saving itself takes place using the log4j library. Usually we store the number of calls, the maximum, minimum and average execution time, the number of errors that occurred during execution.
From temporary databases, all statistics are stored in DWH. Every minute DWH servers go to temporary databases in production and take data. Temporary databases are periodically cleared of data.

An example of code that stores statistics about sent messages:

public void sendMessage(String message) {
   long startTime = LoggerUtil.getMeasureStartTime();
   try {
       /**
        * business logic - send message
        */
        LoggerUtil.operationSuccess(LogFactory.getLog({log's appender name}), startTime, "messageService", "sendMessage");
    } catch (Exception e) {
        LoggerUtil.operationFailure(LogFactory.getLog({log's appender name}), startTime, "messageService", "sendMessage");
    }
}

DWH . Microsoft. – MS SQL 2008, – Reporting services. DWH – 13 , production . (.. ). . — 13 TB.
(multi-dimension) OLAP.

:

1.
2. (, )
. , DWH. , - .
, “” (, : CPU, RAM, ..), ( CPU, RAM, ..). “” Zabbix, Cacti.
, .

:

• : MS Windows, openSUSE
• Java, C, C+. Java. + .
• GWT WEB . GWT ,
• WEB – Apache Tomcat
• JBoss 4
• WEB – LVS. IPVS Layer-4
• Apache Lucene
• :
MS SQL 2005 Std edition. , . MS SQL failover . , standby
BerkeleyDB – BDB , . BDB, C , 4.5. master-slave . BDB . master, . tmpfs, transaction . 15 . .
. .
• , JBoss Remoting
• SQL JDBC

:

70 . 40 , 20 , 8 .
(1-3 ). - , . . , . , .
, . , ( , , ). , , , . – .

. 3- :
0 —
1 — ,
2 — pre-production , production

«». «» :
1. production
2. . ,
3. , . , , , , , 1-

Best practices, tricks & tips

:

• , , .. ( ), ( ). — .
• — . , 1 , , .
• JOIN CPU. JOIN . JOIN , .
• , . CPU .
• SQL DELETE — DML. — , .
• . , . .

:

• , Java. , , , .. .
• . .
• , , .
• , -, , , . map-reduce . , , C. .
• Java (off heap memory) Java GC.
• , .

• (Expires Cache-Control ). CSS JavaScript (gzip).
• HTTP , JavaScript CSS . .
• , .
• CSS . ( ).
• CSS expressions, «». .
• DOM , , reflow. DOM «».
• GWT UIBinder HTMLPanel .

! .