5 trends in data analysis and processing in 2015

The beginning of the year is the time for predictions and predictions. In this post I collected a review of trends in analytics and data processing systems for 2015.

Every year more and more companies come to the conclusion that the proper use of analytical systems and systems for processing large data arrays provides a significant competitive advantage in the market. This situation leads to the active development of analytical departments in companies and significant investments in data processing infrastructure.

According to various reports, the annual increase in costs for the development of analytics and data processing systems is 20% - 30%. The projected total cost of data processing systems and analytical services in 2015 will be ~ 3B $.
')
The strong demand for analytical systems and substantial investments are a strong driver for the rapid development of the big data industry and the emergence of a huge number of new products and solutions.

According to research by large consulting companies, such as Deloitte and Forrester , in 2015, the following main trends in the field of data analysis and processing can be identified.

• Automation and commodity data processing
• Development of real-time analytical systems
• The emergence of new tools for analyzing sensor data and various devices
• Development of cognitive systems and systems of automatic decision making
• Changes in legislation related to the storage and processing of data

1. Automation and commodity data processing

The high maintenance costs of data storage and storage systems are forcing companies to look for cheaper solutions. One of such solutions is the rejection of its own data centers and the transition to cloud storage and data analysis technologies such as Amazon WS , Microsoft Azure .

The acute shortage of specialists in the field of data analysis serves as a good incentive for the development of automated tools that do not require in-depth knowledge of technology and are focused on solving business problems. Examples of such tools include Tableau , IBM Watson, and Azure Machine Learning .

Separately, it is worth noting the development of automatic preprocessing of raw and unstructured data such as OpenRefine and DataWrangler .

2. Development of real-time analytical systems

Traditional analytical solutions based on relational databases and OLAP technologies that require a relatively large amount of time to process queries and generate reports are not enough to solve modern business problems.

Companies need to analyze real-time data to make effective decisions. This will lead to the development of technologies for stream data processing and quick access to data stored in RAM. One of the first products designed to solve these problems are Apache Storm and Cloudera Impala .

3. The emergence of new tools for analyzing sensor data and various devices

Wearable electronics and the Internet of things will continue to grow rapidly in 2015. Such gadgets are already widely used in the field of entertainment and health monitoring. Analyzing information from these devices, you can increase the efficiency of their use.

The wide distribution of gadgets leads to the emergence of new tasks associated with the analysis of signals of various kinds of sensors and sensors. Unlike transactional data, sensor data may be noisy and subject to distortion, which in turn requires a special approach to the processing of such signals. For the tasks related to the collection and analysis of information from the sensors, special tools will be created.

Standards on the integration of sensors and analytical systems will also be actively developed, which will significantly speed up the development of the Internet of things in general.

Separately, it is worth noting that the data wearable electronics will collect more information about the behavior and preferences of users, which is likely to lead to the emergence of new players in the data market .

4. Development of cognitive systems and systems of automatic decision making

Experts predict the active development of technology in the field of human interaction and artificial intelligence. In particular, the development of automatic decision-making systems in business processes, systems for analyzing textual information, voice, images and video. Among the already existing products can be noted such as Google Now and Word Lense .

5. Changes in legislation related to the storage and processing of data

Already today we are generating so much data every day that we can accurately enough make up a portrait of each person, reveal his preferences and find out personal information. With the development of the Internet of things and the emergence of new gadgets, the amount of data will only increase, and with it the share of personal information stored in data centers will increase.

This development will lead to increased social pressure on services and companies that are directly related to the processing of this kind of data, which will lead to the intervention of the regulator and the emergence of new laws restricting the use of personal information and defining the requirements for data security.

Probably, the new requirements will force to reconsider the architectural solutions of companies and lead to the emergence of hybrid systems in which part of the data instead of the cloud will be stored in secure local data centers.

Conclusion

Despite the great achievements in previous periods, 2015 promises to be very interesting in terms of the development of analytical systems and the emergence of new opportunities for revolutionary solutions in the field of data processing and analysis.

What do you think about trends in analytics and data processing systems for 2015? Which of the existing technologies will be in demand this year? What tasks have yet to be solved? Let's discuss in the comments.

Materials

www.lavastorm.com/assets/Forrester-Predictions-2015-Agile-Features-Keep-the-BI-Market-Moving.pdf
public.deloitte.com/media/analytics/trends/pdf/us-da-analytics-analytics-trends-2015.pdf
research.gigaom.com/report/outlook-big-data-and-analytics-in-2015
channels.theinnovationenterprise.com/articles/big-data-top-trends-in-2015