Intel Distribution for Apache Hadoop: for “big data” to be comfortable

Habrazhiteli interested in "big data", probably does not make sense to talk about the project Apache Hadoop. Recently, this framework has become very popular and often mentioned on Habré - more recently, Mail.ru experts told about the migration of their search engine to Hadoop.
How to make a good software system even better? For example, add hardware components to it. This is exactly how Intel went after releasing Intel Distribution for Apache Hadoop - “localizing” Hadoop under the Intel Xeon platform, equipped with many useful features to facilitate its management and operation.

In addition to the basic Apache Hadoop applications, as shown in the figure, the Intel package includes an additional component, Intel Manager for Hadoop Software, which makes it easy to deploy, configure, and monitor new applications. In addition, the Intel Active Tuner for Apache Hadoop module automatically configures the system performance configuration. As an additional component is available the library of graph reports construction Intel Graph Builder for Apache Hadoop.
Optimizing the framework for the Intel Xeon platform has achieved two important results:

• Implementation of data encryption without slowing down disk operations in the HDFS file system due to the hardware implementation of the AES encryption algorithm in Intel Xeon processors;
• Significant increase in overall system performance due to optimal interaction between Hadoop software and Xeon I / O subsystems. According to the results of Intel's internal testing, the developers of the company managed to achieve a five-fold acceleration of the system compared to the non-optimized version.

Intel is developing several projects related to cloud technologies and storage networks. Thus, Intel Intelligent Systems Framework, a framework of systems for collecting, analyzing and transmitting information from end devices to data centers, was recently released. But another project is only at the very beginning. In late February, Intel and the Russian company Mirantis, the vendor of the OpenStack platform, signed an agreement on cooperation in software development. It means that in the future we will also see the optimization of OpenStack for Intel processors, which, as is the case with Hadoop, will certainly allow us to get a noticeable gain.