I present to you the translation of my article on Medium.com: part 1 , part 2 . Since the first part of the article contains mostly already stated in this post , I cite the translation of only the second part.

Lose weight and move to containers

In the first part of the article, I talked about simple approaches that allow building a scalable Selenium cluster without writing code. In this part we will consider more subtle issues of working with Selenium:

1. How to create highly scalable work nodes using standard Selenium Hub
2. Why it is possible and necessary to run most browsers in containers and how it is done
3. What open-source tools for this exist

What's inside the work node

All the new tools described in the first part are in fact smart lightweight proxies that redirect user requests to real Selenium hubs and nodes. If you reflect a little, then questions arise:

1. How to organize hubs and nodes to effectively consume hardware resources and scale well?
2. What operating system to use?
3. What programs should be installed?
4. Can I work without a monitor?

One way would be to use hardware with one Selenium hub and multiple nodes with different browsers. It looks reasonable, but actually uncomfortable:

1. As it was said, the Selenium hub with a large number of loaded nodes works very slowly. Not sure about the real reasons, but practice shows exactly that. My advice - do not read the source code of Selenium at night, if you do not want to have nightmares. Thus, we cannot use dozens of Selenium nodes with the same hub. It remains to use one hub and only a few nodes. To effectively use iron, you need to reduce the total number of cores per hub — a good reason to move to the clouds. For example, our working grid for a long time used small virtual machines with 2 cores and 4 GB of memory.
2. It is not clear how to install different versions of the same browser in a simple way (for example, from packages).
3. It is not clear how easy it is to take into account the total number of browsers of one version available.
4. Different versions of Selenium nodes are compatible with different versions of browsers, i.e. a newer Selenium node may not support an older browser.

The easiest way to have the same number of nodes per hub is to run them inside a single virtual machine. If each browser version is a separate virtual machine, then counting the total number of available browsers becomes an elementary school task. You can easily add and remove virtual machines that contain compatible versions of the node and browser. We recommend this approach when installing a Selenium cluster in the cloud with a constantly available number of each browser version.

What else besides Selenium hub and node is located inside the virtual machine for everything to work?

• First, we recommend using Linux as the primary operating system wherever possible. Using Linux, you can cover 80% of your browser needs. It is easier to list what is not covered:
  
  1. Internet Explorer and Microsoft Edge. These browsers work only under Windows and deserve a separate article. There is no sadder story in the world ...
  2. Desktop Safari. Does anyone use it at all? Selenium quite poorly supports this browser.
  3. iOS and Apple mobile phones. To work with these devices, you need to use Apple hardware, for example, MacMini and Appium .
• To run Selenium, you need to install Java (JDK or JRE), and also download the correct version of Selenium as a JAR archive.

• Virtual machines do not have a monitor, so Selenium must be running in a special version of the X server that emulates the display. This implementation is called Xvfb . It starts like this:

  
  

  xvfb-run -l -a -s '-screen 0 1600x1200x24 -noreset' \ java -jar /path/to/selenium-server-standalone.jar -role node <... > 

  
  

  Please note that Xvfb is only needed for the Selenium node process.

  
  
• You can also install additional font packages, for example, Microsoft True Type fonts .
• If you want to play a sound in your tests, then you need to configure the sound card support. For Ubuntu, it can do something like this:
  

   #!/bin/bash apt-get -y install linux-sound-base libasound2-dev alsa-utils alsa-oss apt-get -y install --reinstall linux-image-extra-`uname -r` modprobe snd-dummy if ! grep -Fxq "snd-dummy" /etc/modules; then echo "snd-dummy" >> /etc/modules fi adduser $(whoami) audio 

  
  

  Lose weight

  
  

  As you can see, Selenium is a Java application. To run Selenium, you need to install Java Virtual Machine (JVM). The smallest Java installation package, called JRE, is about 50 megabytes in size. Selenium JAR latest version 3.0.1 adds another 20 megabytes. Now add the size of the operating system, the necessary fonts, the size of the browser itself and you easily reach several hundred megabytes. And although hard drives are cheap now, we can do better. Selenium versions 2.0 and 3.0 are also called Selenium Webdriver. This is due to the fact that support for different browsers is implemented using separate applications, called web drivers.
  
  

  Here's how it works:
  

  1. Browser developers can write their product as they like. In order for the browser to be supported in Selenium, they need to provide a web server application that provides the same API as the Selenium Server itself and supports the JSONWire protocol. This application should be able to start the browser process, execute Selenium commands as per the specification and stop the browser on demand. Any details of the interaction of the driver with the browser can be implemented at the discretion of the developers. The only requirement is to support the same Selenium API. For example, Chrome has Chromedriver , Opera Blink provides OperaDriver, and so on.
  2. When installing Selenium, you only need to specify the path to the driver application.
  3. When you request a browser from Selenium, it actually starts the driver process and proxies all requests to the driver. The driver does the rest of the work. You can get the same result if you manually start the driver process on the required port and aim your tests at it.
  
  
  
  

  Now that we’ve figured it out, the question is: isn’t it too expensive to spend hundreds of megabytes for simple proxying? A year ago, the answer was definitely not, because there was no driver application for Firefox - the most frequently used browser in Selenium. The responsibility of Selenium was to launch Firefox, load a special extension into it, and proxy requests to the port opened by this extension. Over the past year the situation has changed. Starting with Firefox 48.0, Selenium interacts with the browser using a separate binary driver, Geckodriver . This means that now for most desktop browsers we can completely remove Selenium Server and proxy requests directly to the drivers.

  
  

  We move to containers

  
  

  In the previous sections, I described how you can build a Selenium cluster using virtual machines in the cloud. In this approach, virtual machines are always running and constantly spend your money. In addition, the total number of browsers available for each version is limited and can lead to the complete exhaustion of available browsers during peak loads. I heard about working and even patented complex solutions that launch and warm up a pool of virtual machines, depending on the current load, in order to always have available browsers. It works, but can you do better? The main problem of hypervisor virtualization is speed. Starting a new virtual machine may take a few minutes. But let's think a bit - do we need a separate operating system for each browser? - No, only simple isolation by disk and network is needed. This is why container virtualization is becoming relevant. At the moment, containers work mostly only under Linux, but, as I said, Linux covers 80% of the most popular browsers. Browser containers start in seconds and stop even faster.

  
  
  

  What should be inside the container? - Almost the same as inside the virtual machine: the browser itself, fonts, Xvfb. For older versions of Firefox (<48.0), you still need to install Java and Selenium Server, but for Chrome, Opera, and newer versions of Firefox, we can use the driver application as the main container process. If you use a lightweight Linux distribution (for example, Alpine), you can get very small and lightweight containers.

  
  

  Selenoid

  
  

  At the moment, the most popular and well-known container platform is Docker . Selenium developers provide a set of ready-made Docker containers for running Selenium in Standalone or Grid mode in Docker. In order to start a cluster of such images, you need to start and stop containers manually or with the help of tools like Docker Compose . This approach is much better than installing Selenium from packages, but it would be even better if there was a server with the following behavior:

  
  
  1. The administrator starts the server daemon instead of the usual Selenium hub.
  2. The daemon "knows" (from the configuration) that, for example, to launch Firefox 48.0, you need to download and run container X, and for Chrome 53, container Y.
  3. The user requests the Selenium session in the usual way, but from this new demon.
  4. The daemon analyzes the desired capabilities, starts the desired container, and then proxies requests to the main container process (Selenium server or web driver).
  
  

  We made such a demon ... and even more.

  
  

  Over the years of using the Selenium server on a large scale, we realized that it is very inefficient to use the JVM and the "thick" Selenium JAR for simple proxying requests. Therefore, we were looking for a more lightweight technology. Our choice was in the Go programming language, also known as Golang. Why is Go better for our goals?

  
  
  1. Static linking. The result of the compilation is one file - a binary that can be copied to the server and immediately run. For start it is not necessary to install dependences and additional programs like JVM for Java.
  2. Cross compilation. On the same machine with the same compiler, we can build binaries for different operating systems.
  3. Rich standard library. For us, the most important was the support from the box reverse-proxying and HTTP / 2.
  4. A large community of developers. The language is already becoming one of the most popular for a particular class of tasks.
  5. Well maintained in IDE. For Go there is a good plugin for IntellijIDEA and an alpha version of the full-fledged IDE Goglang from the same developers.
  
  

  We never came up with a good name for the demon described above. Therefore, we called it simply Selenoid . To try Selenoid you need to follow 3 simple steps:

  
  
  • Create a JSON file containing information about which container to launch for each browser version:
  
  

   { "firefox": { "default": "latest", "versions": { "49.0": { "image": "selenoid/firefox:49.0", "port": "4444" }, "latest": { "image": "selenoid/firefox:latest", "port": "4444" } } }, "chrome": { "default": "54.0", "versions": { "54.0": { "image": "selenoid/chrome:54.0", "port": "4444" } } } } 

  
  

  As in the XML file for Gridrouter, a list of available browser versions is provided. Since Selenoid runs containers on the same machine or through the Docker API, there is no need to specify host names and regions. For each browser version, you need to specify the name of the container, its version and the port on which the main process of the container is listening.

  
  
  • Run Selenoid:
  
  

   $ selenoid -limit 10 -conf /etc/selenoid/browsers.json 

  
  

  By default, Selenoid starts on port 4444, just like a regular Selenium hub.

  
  
  • Run your tests on the Selenoid host, as if it were a regular Selenium hub.
  
  

  Our experiments show that even containers with a standard Selenium server inside start in a few seconds. In return, you get guaranteed disk and memory status. The browser is always in a state like after installation on a clean operating system. In addition, you can install Selenoid on a large cluster of hosts with the same set of supported browsers saved as Docker images. This gives you a large cluster of Selenium, which automatically scales with the consumption of browsers. For example, if current user requests require more Chrome, more containers automatically launch. When there are no requests for Chrome, containers with Chrome are stopped and the vacant hosts can be used for other browsers.

  
  

  In order to ensure a better distribution of the load across the cluster, Selenoid limits the total number of concurrent sessions on a host and queues all requests that exceed the limit. Requests from the queue are processed as earlier sessions are completed on the same host.

  
  
  

  But Selenoid allows you to run not only containers. It can also run on-demand web driver processes. The main application of this functionality is the replacement of Selenium Server with Windows. In this case, Selenoid starts the IEDriverServer process, which saves memory and avoids proxying errors in Selenium itself.

  
  

  Go Grid Router (also known as ggr)

  
  

  You may remember that the original GridRouter is a Java application. We wrote a lightweight implementation of this proxy on Go from scratch and simply called the Go Grid Router (or ggr). What are the advantages of the new version compared to the old one?

  
  
  1. Increased performance. It can serve at least 25% more requests.
  2. Lower memory consumption. With a load of 150 rps it consumes only 100-200 megabytes of memory and this number does not change.
  3. Tracked client disconnection. If a client disconnects (for example, due to a timeout), the Java version of GridRouter continues to iterate over the hosts, trying to create a session. This clogs the network with unnecessary packets and reduces the performance of the GridRouter when many hubs become unavailable. The new implementation on Go stops trying to get the browser as soon as the client is disconnected.
  4. Server restart without loss of connections (graceful restart). If the server is used outside the Docker container, you can reboot it without losing connections by sending a SIGUSR2 signal to the process.
  5. Reload quotas on request. When using multiple GridRouter instances behind a balancer, it is important to update quotas at the same time. When new hosts are added to the XML quota files and the quotas are not updated at the same time on a running Selenium cluster, a situation may occur when one “head” of the GridRouter already knows about new hosts and redirects requests there, and the other does not know about these hosts and returns a 404 error. The implementation on Go reloads quotas on the SIGHUP signal, rather than automatically, as it does in the Java version. This functionality works both with and without Docker.
  6. Encrypted passwords. Ggr uses Apache htpasswd text files to store logins and passwords. Logins are stored in clear text and passwords are encrypted.
  7. Small size of executable file. Only 6 megabytes. Does not require installing Java. If installed in Docker, the Alpine Linux-based container occupies only 11 megabytes.
  
  

  In conjunction with Selenoid, allows you to create a scalable, reliable Selenium cluster:

  
  
  

  Conclusion

  
  

  In this part, I talked about the latest technologies that can be used to organize a modern Selenium cluster:

  
  
  1. Why Selenium packs well and runs in containers
  2. What should be inside the container
  3. What open-source tools for running Selenium in containers exist
  
  

  Links

  
  

  In conclusion, in one place are collected links to the products mentioned in the article:

  
  
  1. Selenoid
  2. Go grid router
  3. Grid router