MapReduce without brakes: bypassing bottlenecks using machine learning

When performing calculations in distributed computing systems, including those with the MapReduce architecture, there are often tasks that are performed slowly or late on a single node (stragglers). The reason for the appearance of such problems may be cluster heterogeneity, hardware or software problems. Because of such lateness, the speed of the system as a whole falls. The creators of Hadoop try to deal with this by duplicating slow task runs on other nodes of the cluster (speculative execution), but this approach does not allow defining slow tasks in a timely manner.

On September 20, a scientific and technical seminar will be held at Yandex’s Moscow office , where Eduard Bortnikov, the chief engineer of the Yahoo! Research center, will speak. In the first half of his report, he will tell you how to solve the bottleneck problem of MapReduce using machine learning. This method, unlike the technology of Hadoop, allows you to predict a slowdown in the execution of tasks on a specific node. The predictor can be integrated with the existing MapReduce system, thus increasing the efficiency of the system.

The second part of the report will be devoted to Sailfish - a new implementation of the MapReduce model from Yahoo! .. The new product is based on the principle of combining intermediate data and batch processing of disk I / O operations. Sailfish brilliantly experimented on real-world data and tasks at Yahoo !, showing truly champion results - task efficiency increased to 400% compared to Hadoop. In addition, Sailfish allows you to automatically configure task parameters when changing volumes or distributing data. Sailfish is easier to use than Hadoop, where every launch requires painstaking, manual parameter tuning.
The seminar will be held in Russian, beginning at 19:00 .
')
Registration is required to attend the seminar.

For those who will not be able to come to the seminar, a video broadcast will be organized.