Evernote service architecture overview

As promised, we began to translate some posts from the English- language Evernote Technolog , in which our engineers and developers talk about some of the details of the technical implementation of the service, share stories and just interesting facts from their work.

Today we would like to start the overview of the architecture of Evernote with a general idea of ​​the structure of the service. Now we will not go into details regarding each component, leaving it for future posts.

Let's start with the scheme presented above. All statistics listed there are shown on May 17, 2011.
')
Network: Almost all traffic to / from Evernote goes to www.evernote.com via HTTPS port 443. It includes both all “web activity” and synchronization with all software clients through our Thrift- based API. In total, about 150 million HTTPS requests per day are generated with peak traffic of about 250 Mbit / s. Our night shift of administrators is especially “glad” to this circumstance, since the weekly peak is at about 6:30 am Pacific time.

We use BGP to send traffic directly through a fully independent network of channels provided by our main ( NTT ) and additional ( Level 3 ) providers. It is filtered through Vyatta on the way to the A10 load balancers , which we installed in January, when we reached the SSL performance limit for our old balancers. It is still quite convenient for us to handle existing traffic using one AX 2500 plus a fault-tolerant server, but we are preparing to test their N + 1 configuration when creating our cluster in order to be ready for future growth.

Shards: The core of the Evernote service is the server farm, which we call shards. Each shard processes all data and all traffic (web and API) for a cohort of 100,000 Evernote users. Since we already have more than 9 million users, it turns out about 90 shards.

Each shard is a pair of SuperMicro servers with Intel dual-core processors, a huge amount of RAM, and fully clogged Seagate industrial drives configured in mirrored RAID arrays. At the head of each server, we have a Debian host running a couple of Xen virtual machines. The core virtual machine runs the kernel from the following set of applications: Debian + Java 6 + Tomcat + Hibernate + Ehcache + Stripes + GWT + MySQL (for metadata) + hierarchical local file systems (for file data).

All user data in the master virtual machine on one server is synchronously duplicated in an additional machine on another server using DRBD . This means that each byte of user data is stored on at least four different industrial drives physically located on two different servers. And add to this nightly backup. If we have any problem with the server, we can transfer work from the main virtual machine to an additional one located on another server with minimal downtime using Heartbeat .

Since each user's data is stored completely locally on one (virtual) shard host, we can run each shard as an independent island with little or no interference and regardless of the situation in the rest of the farm. It also means that problems with one shard do not affect the work of the others.

The lion's share of work in the direction of users to their shards falls on load balancers, which have a set of instructions for finding a shard using URLs and / or cookies.

UserStore: Although the vast majority of all data is stored on logically independent shards (“NoteStore”), all of them are associated with a single user account database “UserStore” (also running on MySQL) with a small amount of information about each account, such as user name, MD5 password and user's shard id. This database is small enough to fit in RAM, but at the same time we provide the same reliable backup using a similar combination of mirrored RAID and replications via DRBD for secondary and night backups.

Image Recognition Farm: In order to ensure the search for words within images in your notes, we have selected a pool of 28 servers that use their 8-core processors daily to process new images. On busy days, the flow can reach 1.3 or 1.4 million individual images. Currently, a combination of Linux and Windows is used there, but the other day we plan to complete the migration to Debian to get rid of unnecessary dependencies.

These servers provide the recognition process, which we call AIR (“Advanced Imaging and Recognition”). The software for it was developed by our R & D team. This software processes each image, identifies areas containing text, and then tries to make a weighted list of possible matches for each word using “recognition engines” that generate sets of assumptions. The work uses both the mechanisms developed by our team, which specializes in AIR (for example, handwriting recognition), and licensed technologies from commercial partners who have the best development in their field.

Other services: All of the above servers are installed in pairs in the racks of our data center in Santa Clara, California. In addition to the hardware that provides the basic operation of the service, we also have small groups of servers for simpler tasks that require one or two servers or Xen virtual machines. For example, our SMTP gateway for incoming mail is provided by a couple of servers on Debian with Postfix and a special mail processor written in Java, installed on top of Dwarf . Our Twitter gateway for sending notes to your account using @myen is just a small script using twitter4j .

Our corporate site works on Apache , blogs - on WordPress . Most of our internal backup topology is provided by HP products. We use Puppet for configuration management, and Zabbix , Opsview and AlertSite for monitoring. Night backup is provided by a combination of different software that transmits data over a dedicated gigabit channel to a backup data center.

Wait, why? We understand that this post raises many obvious questions about why we chose X rather than Y in a variety of situations. Why did we choose our own servers instead of using the services of a "cloud" provider? Why did we prefer old software (Java, SQL, local storage, etc.) instead of using the latest recipes?

We will try to answer these questions in more detail in the next few months.