IBM Expands Apache Spark for zSystems Mainframes

IBM has already announced that Apache Spark for Linux will be supported by zSystems. Such support will be provided in the framework of the analytics on mainframe project. Thanks to this, data mining specialists will be able to use Apache Spark on zSystems' powerful mainframes.

In addition, it was stated that Apache Spark will work not only as a service on the Bluemix platform, but also integrate the system with other cloud and analytical solutions, including the Cloudant NoSQL solution and the cloud storage platform SashDB. Developers, using Bluemix, will be able to integrate their projects with analytical solutions and databases from IBM.

Now IBM has already fulfilled most of its promises regarding Apache Spark. First, the corporation has made it easier and faster for organizations to access data analysis capabilities using zSystems mainframes. This creates new ways for data scientists and developers.
')
The IBMz / OS Platform for ApacheSpark allows the open-source Spark framework to work natively on z / OS. And this, in turn, provides the possibility of studying the received data in real time “in the field”, that is, without the need to extract, transform and load (ETL) source information. For example, business representatives can analyze corporate data (sales, market trends, etc.), changing and adjusting their work to market needs on the fly.

Scientists can work with the data in the course of any experiment, receiving detailed reports on the progress of such work in real time. That is, there is practically no delay between receiving information and analyzing it with the output of the processed data.

Now zSystems work in many areas, including science, banking, transportation, insurance business. The mainframe and its software analyze transactions and data instantly, simultaneously building a predictive model in the framework of the current operation. Spark and zSystems help save time, effort and money. Since Spark supports both machine learning, and natural language recognition, and image processing technology, as well as offering a large number of other features, IBM sees Spark as a complete environment for working with data. For example, using the IBM Datacap service, which is part of Insight Cloud Services, a client can automatically classify and recognize the content of a document, including its format and structure, text and numeric information.



There are other advantages of the new platform:

• Simplification of the development process : data processing specialists and developers will be able to use their experience in programming languages ​​such as Scala, Python, RandSQL to reduce development time and get results faster.
• Simplified data access : fast, constant data access in traditional formats, including IMS, VSAM, DB2 z / OS, PDSE, or SMF with familiar tools through the Apache Spark API.
• In-place data analysis : Apache Spark uses for data processing, which allows you to quickly get results. This method reduces the cost of data processing, plus a fairly high level of security is maintained.
• Opensource : Apache Spark within the framework of the platform is provided as open-source, which opens up broad opportunities for third-party developers.
• In addition, IBM continues to work with three major data processing partners . These are Zementis (predictive analysis), Rocket Software (data visualization) and Elite Analythics (working with projects running on zSystems).
• Cooperation with Zementis will allow specialists to build models using SPSS, R, Python, SAS and other commercial or open-source tools. Such models can then be run in the z / OS operating environment. Users can work with Zementis to create IMS, DB2 for z / OS, and VSAM operating models.
• Collaboration with Rocket Software makes available Data Virtualization Service by Rocket Service for Spark at z Systems. The service allows you to combine various data sources into a single system. Plus, this company announced its intention to add support for the analytical platform R on z / OS.
• Working with Elite Analythics includes the provision of new services for the development and management of various projects on zSystems. These include the real-time capabilities of Zementis or SPSS, plus projects developed on ApacheSpark on z / OS.

Overall, z / OSPlatform for Apache Spark allows data processing specialists and developers to use their own formats and tools for collecting and analyzing information. If necessary, the provided tool can be customized.

The project is now quite a developed ecosystem. One way or another, the activity of 3,500 IBM researchers and developers who create their own projects on this framework is connected with the platform. Experts can post their work on GitHub .

The IBMz / OS Platform for Apache Spark is already available for download .