Qihoo 360 and Go

I bring to your attention the translation of the guest post from Go blog on behalf of Yang Zhou, who is currently in the position of engineer in Qihoo 360 .

Qihoo 360 is a leading supplier of antivirus products for the Internet and mobile devices in China, controls a large distribution platform for mobile applications for Android (application store). At the end of June 2014, Qihoo was used by 500 million active PC users per month and over 640 million mobile device users. Qihoo also has its own browser and search engine, both equally popular among Chinese.

My team, the push messaging department, provides a fundamental messaging service for more than 50 applications among the company's products (for both PCs and mobile devices), as well as thousands of third-party applications that use our open platform.
')
Our “novel” with Go originates in 2012, when we tried to set up push work. The very first option was a bundle of nginx + lua + redis, but performance under a serious load did not fit our requirements. During the next search for the right stack, our attention was attracted by the fresh release of Go 1.0.3. We made a prototype in just a few weeks, thanks in large part to the gorutines (lightweight flows) and the go-channels (typed queues), which are primitives of the language.

Initially, our goffered system was deployed on 20 servers, serving a total of 20 million active connections. She coped with sending only 2 million messages per day. Now the system is deployed on 400 servers, supports 200+ million active connections and provides sending more than 10 billion messages daily.

In parallel with the rapid growth of the business and increasing demands for the service of push-messages, the initial system on Go quickly rested in limits: the heap reached 69G, the GC pauses were 3-6 seconds each. Moreover, we rebooted our servers weekly to free up memory. To be honest, they thought even to get rid of Go and rewrite the entire core to C. However, soon the plans changed: the plug-in occurred during the transfer of business logic. It was impossible for one person (me) to master the support of the system on Go, while at the same time ensuring porting business logic to C.

So I decided to stay with Go (and in my opinion, this is the wisest of all that I could accept) and qualitative progress came pretty soon.

Here are some techniques and optimizations:

• Reuse TCP connections (connection pool) to avoid creating new objects and buffers during the interaction;
• Reuse objects and allocated memory to reduce the load on the GC;
• Use groups of long-lived gorutin to process task queues or messages received from
  gorutin connections. Those. classic pub / sub instead of spawning multiple gorutins, one for each incoming request;
• Monitor and control the amount of gorutin in the process. Lack of control can lead to an unbearable load on the GC due to a jump in the number of generated gorutin. For example, if you allow the uncontrolled creation of gorutin to process requests from the outside, blocking newly created as a result of accessing internal services, for example, will lead to the creation of new gorutin and so on;
• Do not forget to specify the deadlines for reading and writing for network connections when working with a mobile network, otherwise the gorutiny may be blocked. However, use them carefully with a LAN network, otherwise the efficiency of the RPC interaction may suffer;
• Use Pipelining for calls if necessary (if Full Duplex for TCP is available).

The result was three iterations of our architecture, two iterations of the RPC framework, even with a limited number of people to implement. I would attribute this achievement to the convenience of developing on Go. I give a fresh diagram of our architecture:



The result of continuous improvements and improvements in the form of a table:



And, no less steeply, at the same time we developed a platform for profiling Go programs in real time (Visibility Platform). Now we have access to the system status and diagnostic information, anticipating any problems.

Screenshots:




From the great thing: with this tool we can simulate the connection and behavior of millions of users using the Distributed Stress Test Tool module (also written in Go), monitor the results in real time and with visualization. This allows us to study the effectiveness of any innovation or optimization, preventing any performance problems in combat conditions. Virtually any possible optimization of the system has already been tested by us. And we are looking forward to good news from the team responsible for GC at Go, they could save us from doing too much work with code optimization under GC in the future. I also admit that our tricks will soon become simply rudimentary, as Go continues to evolve.

I want to end this story with gratitude for the opportunity to participate in Gopher China . It was a solemn event that allowed us to learn a lot, share our knowledge, feel the popularity and success of Go in China. A large number of teams in Qihoo have already managed to get acquainted with Go and even try. I am convinced that soon more Chinese Internet companies will join the trend and rewrite their systems on Go, so the efforts of the team behind Go will benefit a huge number of developers and companies in the near future.