Analytical review of the Big Data market
“Big Data” is a topic that is actively discussed by technology companies. Some of them managed to give up on big data, others - on the contrary, make the most of their use for business ... A fresh analytical review of the domestic and global Big Data market, prepared by the Moscow Exchange together with IPOboard analysts, shows which trends are most relevant in the market today. . We hope the information will be interesting and useful.
Big Data, at the moment, is one of the key drivers of information technology development. This trend, relatively new to Russian business, has become widespread in Western countries. This is due to the fact that in the era of information technology, especially after the boom of social networks, a significant amount of information began to accumulate on every Internet user, which ultimately led to the development of Big Data.
The term “Big Data” causes a lot of controversy, many believe that it means only the amount of accumulated information, but do not forget about the technical side, this direction includes storage technologies, calculations, as well as services.
It should be noted that it is the processing of a large amount of information that is difficult to handle in traditional ways *.
')
Below is a comparative table of the traditional and the Big Data database.
The scope of Big Data is characterized by the following features:
Volume - the volume, the accumulated database is a large amount of information that is time-consuming to process and store in traditional ways, they require a new approach and improved tools.
Velocity - speed, this attribute indicates both the increasing data accumulation rate (90% of the information was collected over the last 2 years) and the data processing speed; in recent times, real-time data processing technologies have become more in demand.
Variety - diversity, i.e. possibility of simultaneous processing of structured and unstructured multi-format information. The main difference between structured information is that it can be classified. An example of such information is customer transaction information.
Unstructured information includes video, audio files, free text, information coming from social networks. To date, 80% of the information is included in the unstructured group. This information needs a comprehensive analysis to make it useful for further processing.
Veracity - the accuracy of the data, the increasing importance of users began to attach importance to the reliability of the available data. Thus, Internet companies have a problem in separating actions performed by a robot and a person on the company's website, which ultimately leads to difficulty in analyzing data.
Value - the value of the accumulated information. Big Data should be useful to the company and bring some value to it. For example, to help in improving business processes, reporting or optimizing costs.
If the above 5 conditions are met, the accumulated data volumes can be considered as large.
Applications for Big Data
The scope of using Big Data technology is vast. So, with the help of Big Data, you can learn about customer preferences, about the effectiveness of marketing campaigns or conduct a risk analysis. Below are the results of the IBM Institute survey on the use of Big Data in companies.
As can be seen from the diagram, most companies use Big Data in the area of customer service, the second most popular area is operational efficiency, and in the area of risk management, Big Data is less common today.
It should also be noted that Big Data is one of the fastest growing areas of information technology, according to statistics, the total amount of received and stored data doubles every 1.2 years.
For the period from 2012 to 2014, the amount of data transmitted monthly by mobile networks increased by 81%. According to Cisco estimates, in 2014 the volume of mobile traffic was 2.5 exabytes (a unit of information amount equal to 10 ^ 18 standard bytes) per month, and already in 2019 it will be equal to 24.3 exabytes.
Thus, Big Data is already an established sphere of technology, even despite its relatively young age, which has become widespread in many areas of business and plays an important role in the development of companies.
The technologies used for collecting and processing Big Data can be divided into 3 groups:
The most common data processing (software) approaches include:
SQL is a structured query language that allows you to work with databases. Using SQL, you can create and modify data, and the management of the data set is handled by the corresponding database management system.
NoSQL - the term stands for Not Only SQL (not only SQL). It includes a number of approaches aimed at implementing the database, which differ from the models used in traditional, relational database management systems. They are convenient to use with a constantly changing data structure. For example, to collect and store information in social networks.
MapReduce is a computational distribution model. Used for parallel computations over very large data sets (petabytes * and more). In the program interface, not the data is transferred to the program for processing, but the program - to the data. Thus, the request is a separate program. The principle of operation consists in the sequential processing of data by two methods Map and Reduce. Map selects preliminary data, Reduce aggregates them.
Hadoop - is used to implement search and contextual mechanisms of high-load sites - Facebook, eBay, Amazon, etc. A distinctive feature is that the system is protected from the failure of any of the nodes of the cluster, since each block has at least one copy of data on another node.
SAP HANA is a high-performance NewSQL data storage and processing platform. Provides high speed query processing. Another distinguishing feature is that SAP HANA simplifies the system landscape, reducing the cost of supporting analytical systems.
Technological equipment includes:
Servers include data warehouses.
Infrastructure equipment includes platform acceleration facilities, uninterruptible power supplies, server console kits, etc.
Service Services.
Service services include the services of building a database system architecture, arranging and optimizing the infrastructure, and ensuring the safety of data storage.
Software, hardware, and services together form an integrated platform for storing and analyzing data. Companies such as Microsoft, HP, EMC offer services for the development, deployment of solutions for Big Data and their management.
Big Data is widespread in many business sectors. They are used in health care, telecommunications, trade, logistics, financial companies, as well as in public administration.
Below are a few examples of using Big Data in some of the industries.
Retail
In the databases of retail stores can be accumulated a lot of information about customers, inventory management system, the supply of commercial products. This information can be useful in all areas of the stores.
So, with the help of accumulated information, you can manage the supply of goods, its storage and sale. Based on the accumulated information, it is possible to predict the demand and supply of goods. Also, the data processing and analysis system can solve other problems of the retailer, for example, to optimize costs or prepare reports.
Financial services
Big Data provides an opportunity to analyze the borrower's creditworthiness, they are also useful for credit scoring * and underwriting **. The introduction of technology Big Data will reduce the time for consideration of loan applications. With the help of Big Data, you can analyze the operations of a particular customer and offer suitable banking services to him.
Telecom
In the telecommunications industry, Big Data was widely distributed in mobile operators.
Mobile operators along with financial organizations have one of the most extensive databases, which allows them to carry out the most in-depth analysis of the accumulated information.
The main purpose of data analysis is to retain existing customers and attract new ones. For this, companies conduct customer segmentation, analyze their traffic, determine the social identity of the subscriber.
In addition to using Big Data for marketing purposes, technologies are used to prevent fraudulent financial transactions.
Mining and petroleum industry
Big Data is used in both mining and processing and marketing. On the basis of the information received, enterprises can draw conclusions about the effectiveness of the development of the field, track the overhaul schedule and the condition of the equipment, and forecast the demand for products and prices.
According to the Tech Pro Research survey, Big Data was most prevalent in the telecommunications industry, as well as in engineering, IT, financial and state-owned enterprises. According to the results of this survey, Big Data is less popular in education and healthcare. The survey results are presented below:
Today, Big Data is being actively introduced in foreign companies. Companies such as Nasdaq, Facebook, Google, IBM, VISA, Master Card, Bank of America, HSBC, AT & T, Coca Cola, Starbucks and Netflix are already using Big Data resources.
The applications of the processed information are diverse and vary depending on the industry and the tasks that need to be performed.
Below are examples of the application of Big Data technology in practice.
HSBC uses Big Data technology to counter plastic card fraud. With the help of Big Data, the company has increased the effectiveness of the security service 3 times, the recognition of fraudulent incidents - 10 times. The economic effect from the introduction of these technologies exceeded $ 10 million.
Antifraud * VISA allows you to automatically calculate fraudulent transactions, the system currently helps to prevent fraudulent payments in the amount of $ 2 billion annually.
IBM's Watson supercomputer analyzes real-time cash flow data. According to IBM, Watson increased the number of detected fraudulent transactions by 15%, reduced false alarms of the system by 50%, and increased the amount of money protected from transactions of this nature by 60%.
Procter & Gamble uses Big Data to design new products and make up global marketing campaigns. P & G has created specialized offices in the Business Spheres where you can view information in real time.
Thus, the management of the company had the opportunity to instantly test hypotheses and conduct experiments. P & G believes that Big Data helps in predicting company performance.
OfficeMax office supplies retailer uses Big Data technology to analyze customer behavior. Analysis of Big Data allowed to increase B2B revenue by 13%, reduce costs by $ 400,000 per year.
According to Caterpillar , its distributors miss $ 9 billion to $ 18 billion each year just because they don’t adopt Big Data processing technologies. Big Data would allow customers to more effectively manage their fleets, by analyzing information from sensors installed on machines.
To date, it is already possible to analyze the condition of key assemblies, their degree of wear, and manage the costs of fuel and maintenance.
Luxottica group is a manufacturer of sports glasses, such brands as Ray-Ban, Persol and Oakley. The Big Data technology company uses to analyze the behavior of potential customers and "smart" SMS marketing. As a result, the Big Data Luxottica group has identified more than 100 million of its most valuable customers and increased the effectiveness of the marketing campaign by 10%.
Using Yandex Data Factory, World of Tanks game developers analyze player behavior. Big Data technologies allowed analyzing the behavior of 100 thousand World of Tanks players using more than 100 parameters (information about purchases, games, experience, etc.). As a result of the analysis, a forecast of users outflow was obtained. This information allows you to reduce user care and work with participants in the game targeted. The developed model turned out to be 20-30% more efficient than standard tools for analyzing the gaming industry.
The German Ministry of Labor uses Big Data in its work related to the analysis of incoming applications for unemployment benefits. So, after analyzing the information, it became clear that 20% of benefits were paid undeservedly. With the help of Big Data, the Ministry of Labor has cut spending by 10 billion euros.
Toronto Children's Hospital has implemented Project Artemis. This is an information system that collects and analyzes data on infants in real time. The system monitors every second 1260 indicators of the state of each child. Project Artemis allows you to predict the unstable state of the child and begin the prevention of diseases in children.
In 2014, Big Data, according to Data Collective, became one of the priority areas of investment in the field of venture capital industry. According to the information portal Computerra, this is due to the fact that developments from this area have begun to bring significant results for their users. Over the past year, the number of companies with implemented projects in the field of big data management increased by 125%, the market volume grew by 45% compared with 2013.
Most of the revenue of the Big Data market, according to Wikibon, in 2014 were services, their share was equal to 40% in the total revenue (see the diagram below):
If we consider Big Data for 2014 by subtypes, the market will look like this:
According to Wikibon, applications and analytics account for 36% of Big Data revenue in 2014 brought Big Data Analytics applications and analytics, 17% computing equipment and 15% storage technology. The least revenue was generated by NoSQL technologies, infrastructure equipment and the provision of a network of companies (corporate networks).
Big Data technologies such as in-memory platforms from SAP, HANA, Oracle and others are the most popular. T-Systems survey results showed that they were chosen by 30% of surveyed companies. NoSQL platforms were the second most popular (18% of users), companies also used analytical platforms from Splunk and Dell, 15% of companies chose them. The least useful for solving the problems of Big Data, according to the survey results were Hadoop / MapReduce products.
According to an Accenture survey, in more than 50% of companies using Big Data technology, the cost of Big Data ranges from 21% to 30%.
According to the following Accenture analysis, 76% of companies believe that these costs will increase in 2015, and 24% of companies will not change their budget for Big Data technology. This suggests that in these companies, Big Data has become an established IT direction, which has become an integral part of the company's development.
The results of the Economist Intelligence Unit survey survey confirm the positive effect of the implementation of Big Data. 46% of companies say that using Big Data technology they improved customer service by more than 10%, 33% of companies optimized reserves and improved the productivity of fixed assets, 32% of companies improved planning processes.
Today, Big Data technologies are most often being introduced in US companies, but now even other countries around the world have begun to show interest. In 2014, according to IDC, Europe, the Middle East, Asia (excluding Japan) and Africa accounted for 45% of the software market, services and equipment in the field of Big Data.
Also, according to the CIO survey, companies from the Asia-Pacific region are rapidly developing new solutions in the field of Big Data analysis, secure storage and cloud technologies. Latin America is in second place in terms of the number of investments in the development of Big Data technology, ahead of the countries of Europe and the USA.
Next, a description and forecasts of the development of the Big Data market of several countries will be presented.
China
The volume of information in China is 909 exabytes, which is 10% of the total amount of information in the world, by 2020 the amount of information will reach 8060 exabytes, the share of information in global statistics will increase, in 5 years it will be equal to 18%. The potential growth of China's Big Data has one of the fastest growing speakers.
Brazil
Brazil at the end of 2014 has accumulated information for 212 exabytes, which is 3% of the world total. By 2020, the amount of information will increase to 1600 exabyte, which will be 4% of the information from the whole world.
India
According to EMC, India’s accumulated data for 2014 is 326 exabytes, which is 5% of the total information. By 2020, the amount of information will grow to 2,800 exabytes, representing 6% of the information from the whole world.
Japan
Japan’s accumulated data for 2014 is 495 exabytes, which is 8% of the total information. By 2020, the amount of information will increase to 2,200 exabytes, but the share of the Japanese market will decrease and amount to 5% of the total information volume of the whole world.
Thus, the volume of the Japanese market will decrease by more than 30%.
Germany
According to EMC, the amount of accumulated data in Germany at the end of 2014 is 230 exabytes, which is 4% of the total information in the world. By 2020, the amount of information will grow to 1100 exabytes and will be 2%.
In the German market, Experton Group predicts that a larger share of revenue will generate a service segment, whose share in 2015 will be 54%, and in 2019 it will increase to 59%, while the share of software and hardware will, on the contrary, decrease.
In general, the market volume will increase from 1.345 billion euros in 2015 to 3.198 billion euros in 2019, the average growth rate will be 24%.
Thus, based on the CIO and EMC analytics, it can be concluded that the developing countries of the world will become the markets for the active development of Big Data technologies in the coming years.
According to IDG Enterprise, in 2015, the companies' expenditures on the Big Data will average $ 7.4 million for a company, large companies intend to spend about $ 13.8 million, small and medium ones - $ 1.6 million .
Most will be invested in areas such as data analysis and visualization and their collection.
According to current trends and market demand, investments in 2015 will be used to improve data quality, improve planning and forecasting, and also to increase the speed of data processing.
The financial sector companies, according to Bain Company's Insights Analysis, will make significant investments, so in 2015 it is planned to spend $ 6.4 billion on Big Data technologies, the average investment growth rate will be 22% by 2020. Internet companies plan to spend $ 2.8 billion, the average growth rate of the increase in costs for Big Data will be 26%.
During the survey of the Economist Intelligence Unit survey, the priority directions for the development of Big Data in 2014 and in the next 3 years were identified, the distribution of answers is as follows:
IDC forecasts the market development trends are as follows:
IDC experts identified 3 drivers for Big Data Market 2015:
According to the Accenture survey, data security issues are now the main barrier to the implementation of Big Data technologies, more than 51% of respondents confirmed that they are concerned about ensuring data protection and their confidentiality. 47% of companies reported the impossibility of implementing Big Data due to a limited budget, 41% of companies indicated a shortage of qualified personnel as a problem.
Wikibon predicts that the Big Data market will grow in 2015 to $ 38.4 billion and increase by 36% compared with the previous year. In the coming years, there will be a decline in growth rates to 10% in 2017. Taking into account these forecasts, the market volume in 2020 will be equal to 68.7 billion US dollars.
The distribution of the global Big Data Market by business category will be as follows:
As can be seen from the diagram, most of the market will be occupied by technologies from the scope of customer service improvement. Point marketing will be in second place in priority for companies until 2019, in 2020, according to Heavy Reading, it will give way to solutions to improve operational efficiency.
The highest growth rate will also be at the “customer service improvement” segment, an increase of 49% annually.
The market forecast for Big Data subtypes will be as follows:
The prevailing market share, as can be seen from the chart, is occupied by professional services, applications with analytics will have the highest growth rates, their share will grow from the current 12% to 18% in 2020 and the volume of this segment will be equal to 12.3 billion USD. the share of computing equipment, on the contrary, will fall from 20% to 14% and will be about $ 9.3 billion in 2020, the cloud technology market will gradually increase and reach $ 6.3 billion in 2020, the market share of solutions for Data storage, on the contrary, will decrease from 15% in 2014 to 13% in 2020 and in monetary terms. yrazhenii will be equal to US $ 8.9 billion..
According to the forecast of Bain & Company's Insights Analysis, the distribution of the Big Data market by industry in 2020 will be as follows:
Energy companies will invest a relatively small amount of $ 800 million in these technologies, but the growth rate will be one of the highest - 54% annually.
Thus, the largest share of the Big Data market in 2020 will be occupied by companies in the financial industry, and the energy sector will be the fastest growing sector.
Following analysts' forecasts, the total market volume will increase in the coming years. Market growth will be achieved through the implementation of Big Data technologies in developing countries around the world, as can be seen from the graph below.
The projected market volume will depend on how developing countries perceive Big Data technologies, whether they will be as popular as in developed countries. In 2014, the developing countries of the world accounted for 40% of the accumulated information. According to the EMC forecast, the current market structure, with a predominance of developed countries, will change in 2017. According to EMC analytics, in 2020 the share of developing countries will be more than 60%.
According to Cisco and EMC, developing countries of the world will actively work with Big Data, in many respects this will be related to the availability of technology and the accumulation of sufficient information to the level of Big Data. The world map presented on the next page will show a forecast of the increase in volume and growth rate of Big Data by region.
According to a study by CNews Analytics and Oracle, the maturity level of the Russian Big Data market has increased over the past year. Respondents representing 108 large enterprises from different industries demonstrated a higher degree of awareness of these technologies, as well as an established understanding of the potential of such solutions for their business.
As of 2014, according to IDC, Russia has accumulated 155 exabytes of information, which is only 1.8% of world data. The amount of information by 2020 will reach 980 exabyte and will take up 2.2%. Thus, the average growth rate of information will be 36% per year.
IDC estimates the Russian market at $ 340 million, of which $ 100 million is SAP solutions, and about $ 240 million is equivalent to Oracle, IBM, SAS, Microsoft, and others.
The growth rate of the Russian Big Data market is at least 50% per year.
It is predicted that positive dynamics will continue in this sector of the Russian IT market, even in the context of general economic stagnation. This is due to the fact that the business continues to place demands on solutions that will improve work efficiency, as well as optimize costs, improve forecasting accuracy and minimize potential risks for the company.
The main providers of Big Data services on the Russian market are:
According to CNews, in Russia only 10% of companies started using Big Data technologies, when in the world the share of such companies is about 30%. Readiness for Big Data projects is growing in many sectors of the Russian economy, according to a report from CNews Analytics and Oracle. More than a third of the companies surveyed (37%) started working with Big Data technologies, among which 20% are already using such solutions, and 17% are starting to experiment with them. The second third of respondents are currently considering this possibility.
In Russia, Big Data technologies are more popular in the banking and telecom industries, but they are also in demand in the mining, energy, retail, logistics companies and the public sector.
Further examples of the use of Big Data in Russian realities will be considered.
Telecom
Telecom operators have one of the most extensive databases, which allows them to carry out the most in-depth analysis of the accumulated information.
One of the applications of Big Data technology is subscriber loyalty management.
The main purpose of data analysis is to retain existing customers and attract new ones. For this, companies conduct customer segmentation, analyze their traffic, determine the social identity of the subscriber. In addition to using information for marketing purposes, telecom technology is used to prevent fraudulent financial transactions.
. , , , .
Big Data . , , , . 55%.
Alfa-Bank analyzes information from social networks, processes applications for a loan, and analyzes the behavior of users of the company's website.
Sberbank also began processing a data set with the aim of customer segmentation, fraud prevention, cross-selling and risk management. In the future, it is planned to improve the service and analyze customer actions in real time.
The Russian Regional Development Bank analyzes the behavior of plastic card holders. This allows you to identify atypical for a particular client operation, thereby increasing the likelihood of identifying theft of money from plastic cards.
Retail
, . , CNews Analytics, Big Data 20% . 75% . Hadoop 7-10%.
. SAP HANA, , , 10 3, 3 30 .
Wikimart .
One of the first offline stores to implement Big Data analysis in Russia was Lenta. With the help of Big Data, retail began to study information about customers from cash receipts. The retailer collects information for the compilation of behavioral models, which makes it possible to make more informed decisions at the level of operating and commercial activities.
Oil and gas industry
In this industry, the scope of application of Big Data is quite wide. Big Data technologies can be applied to the extraction of minerals from the depths. With their help, you can analyze the process of extraction and the most effective ways to extract it, monitor the drilling process, analyze the quality of raw materials, as well as the processing and marketing of the final product. In Russia, these technologies have already begun to use Transneft and Rosneft.
Government agencies
In countries such as Germany, Australia, Spain, Japan, Brazil and Pakistan, Big Data technology is used to resolve issues of a national scale. These technologies help public authorities to more effectively provide services to the public, provide targeted social support.
In Russia, these technologies began to be developed by such state bodies as the Pension Fund, the Federal Tax Service and the Mandatory Medical Insurance Fund. The potential of implementing projects using Big Data is large, these technologies could help to improve the quality of services, and, as a result, the standard of living of the population.
Logistics and transportation
Big Data . , , .
Big Data SAP. 43,5 ( 14,5 20 ), 40 . Big Data . 300 SAP, 4 -, 220 000.
, , .
:
Big Data :
, . , .
, , .
. IDC , 5 , .
2020 Big Data 1,8% 2,2% . , EMC, 155 980 2020 .
.
CNews Analytics, 44% 100 * 13% 500 .
, , . 2014 IDC 340 . .
50% , , 2018 1,7 . . 3%, 1,2%.
Big Data :
Big Data , , , - , Big Data .
, , , .
, , , , , .
, , Big Data , .
2014 :
:
The Russian market of Big Data is at the stage of formation, the results of 2014 are as follows:
The forecast of the Russian market for the coming years is as follows:
, Big Data , .
, , — !
WHAT IS BIG DATA?
Key features
Big Data, at the moment, is one of the key drivers of information technology development. This trend, relatively new to Russian business, has become widespread in Western countries. This is due to the fact that in the era of information technology, especially after the boom of social networks, a significant amount of information began to accumulate on every Internet user, which ultimately led to the development of Big Data.
The term “Big Data” causes a lot of controversy, many believe that it means only the amount of accumulated information, but do not forget about the technical side, this direction includes storage technologies, calculations, as well as services.
It should be noted that it is the processing of a large amount of information that is difficult to handle in traditional ways *.
')
Below is a comparative table of the traditional and the Big Data database.
The scope of Big Data is characterized by the following features:
Volume - the volume, the accumulated database is a large amount of information that is time-consuming to process and store in traditional ways, they require a new approach and improved tools.
Velocity - speed, this attribute indicates both the increasing data accumulation rate (90% of the information was collected over the last 2 years) and the data processing speed; in recent times, real-time data processing technologies have become more in demand.
Variety - diversity, i.e. possibility of simultaneous processing of structured and unstructured multi-format information. The main difference between structured information is that it can be classified. An example of such information is customer transaction information.
Unstructured information includes video, audio files, free text, information coming from social networks. To date, 80% of the information is included in the unstructured group. This information needs a comprehensive analysis to make it useful for further processing.
Veracity - the accuracy of the data, the increasing importance of users began to attach importance to the reliability of the available data. Thus, Internet companies have a problem in separating actions performed by a robot and a person on the company's website, which ultimately leads to difficulty in analyzing data.
Value - the value of the accumulated information. Big Data should be useful to the company and bring some value to it. For example, to help in improving business processes, reporting or optimizing costs.
If the above 5 conditions are met, the accumulated data volumes can be considered as large.
Applications for Big Data
The scope of using Big Data technology is vast. So, with the help of Big Data, you can learn about customer preferences, about the effectiveness of marketing campaigns or conduct a risk analysis. Below are the results of the IBM Institute survey on the use of Big Data in companies.
As can be seen from the diagram, most companies use Big Data in the area of customer service, the second most popular area is operational efficiency, and in the area of risk management, Big Data is less common today.
It should also be noted that Big Data is one of the fastest growing areas of information technology, according to statistics, the total amount of received and stored data doubles every 1.2 years.
For the period from 2012 to 2014, the amount of data transmitted monthly by mobile networks increased by 81%. According to Cisco estimates, in 2014 the volume of mobile traffic was 2.5 exabytes (a unit of information amount equal to 10 ^ 18 standard bytes) per month, and already in 2019 it will be equal to 24.3 exabytes.
Thus, Big Data is already an established sphere of technology, even despite its relatively young age, which has become widespread in many areas of business and plays an important role in the development of companies.
Big Data Technologies
The technologies used for collecting and processing Big Data can be divided into 3 groups:
- Software;
- Equipment;
- Service Services.
The most common data processing (software) approaches include:
SQL is a structured query language that allows you to work with databases. Using SQL, you can create and modify data, and the management of the data set is handled by the corresponding database management system.
NoSQL - the term stands for Not Only SQL (not only SQL). It includes a number of approaches aimed at implementing the database, which differ from the models used in traditional, relational database management systems. They are convenient to use with a constantly changing data structure. For example, to collect and store information in social networks.
MapReduce is a computational distribution model. Used for parallel computations over very large data sets (petabytes * and more). In the program interface, not the data is transferred to the program for processing, but the program - to the data. Thus, the request is a separate program. The principle of operation consists in the sequential processing of data by two methods Map and Reduce. Map selects preliminary data, Reduce aggregates them.
Hadoop - is used to implement search and contextual mechanisms of high-load sites - Facebook, eBay, Amazon, etc. A distinctive feature is that the system is protected from the failure of any of the nodes of the cluster, since each block has at least one copy of data on another node.
SAP HANA is a high-performance NewSQL data storage and processing platform. Provides high speed query processing. Another distinguishing feature is that SAP HANA simplifies the system landscape, reducing the cost of supporting analytical systems.
Technological equipment includes:
- servers;
- infrastructure equipment.
Servers include data warehouses.
Infrastructure equipment includes platform acceleration facilities, uninterruptible power supplies, server console kits, etc.
Service Services.
Service services include the services of building a database system architecture, arranging and optimizing the infrastructure, and ensuring the safety of data storage.
Software, hardware, and services together form an integrated platform for storing and analyzing data. Companies such as Microsoft, HP, EMC offer services for the development, deployment of solutions for Big Data and their management.
Application in industries
Big Data is widespread in many business sectors. They are used in health care, telecommunications, trade, logistics, financial companies, as well as in public administration.
Below are a few examples of using Big Data in some of the industries.
Retail
In the databases of retail stores can be accumulated a lot of information about customers, inventory management system, the supply of commercial products. This information can be useful in all areas of the stores.
So, with the help of accumulated information, you can manage the supply of goods, its storage and sale. Based on the accumulated information, it is possible to predict the demand and supply of goods. Also, the data processing and analysis system can solve other problems of the retailer, for example, to optimize costs or prepare reports.
Financial services
Big Data provides an opportunity to analyze the borrower's creditworthiness, they are also useful for credit scoring * and underwriting **. The introduction of technology Big Data will reduce the time for consideration of loan applications. With the help of Big Data, you can analyze the operations of a particular customer and offer suitable banking services to him.
Telecom
In the telecommunications industry, Big Data was widely distributed in mobile operators.
Mobile operators along with financial organizations have one of the most extensive databases, which allows them to carry out the most in-depth analysis of the accumulated information.
The main purpose of data analysis is to retain existing customers and attract new ones. For this, companies conduct customer segmentation, analyze their traffic, determine the social identity of the subscriber.
In addition to using Big Data for marketing purposes, technologies are used to prevent fraudulent financial transactions.
Mining and petroleum industry
Big Data is used in both mining and processing and marketing. On the basis of the information received, enterprises can draw conclusions about the effectiveness of the development of the field, track the overhaul schedule and the condition of the equipment, and forecast the demand for products and prices.
According to the Tech Pro Research survey, Big Data was most prevalent in the telecommunications industry, as well as in engineering, IT, financial and state-owned enterprises. According to the results of this survey, Big Data is less popular in education and healthcare. The survey results are presented below:
Examples of using Big Data in companies
Today, Big Data is being actively introduced in foreign companies. Companies such as Nasdaq, Facebook, Google, IBM, VISA, Master Card, Bank of America, HSBC, AT & T, Coca Cola, Starbucks and Netflix are already using Big Data resources.
The applications of the processed information are diverse and vary depending on the industry and the tasks that need to be performed.
Below are examples of the application of Big Data technology in practice.
HSBC uses Big Data technology to counter plastic card fraud. With the help of Big Data, the company has increased the effectiveness of the security service 3 times, the recognition of fraudulent incidents - 10 times. The economic effect from the introduction of these technologies exceeded $ 10 million.
Antifraud * VISA allows you to automatically calculate fraudulent transactions, the system currently helps to prevent fraudulent payments in the amount of $ 2 billion annually.
IBM's Watson supercomputer analyzes real-time cash flow data. According to IBM, Watson increased the number of detected fraudulent transactions by 15%, reduced false alarms of the system by 50%, and increased the amount of money protected from transactions of this nature by 60%.
Procter & Gamble uses Big Data to design new products and make up global marketing campaigns. P & G has created specialized offices in the Business Spheres where you can view information in real time.
Thus, the management of the company had the opportunity to instantly test hypotheses and conduct experiments. P & G believes that Big Data helps in predicting company performance.
OfficeMax office supplies retailer uses Big Data technology to analyze customer behavior. Analysis of Big Data allowed to increase B2B revenue by 13%, reduce costs by $ 400,000 per year.
According to Caterpillar , its distributors miss $ 9 billion to $ 18 billion each year just because they don’t adopt Big Data processing technologies. Big Data would allow customers to more effectively manage their fleets, by analyzing information from sensors installed on machines.
To date, it is already possible to analyze the condition of key assemblies, their degree of wear, and manage the costs of fuel and maintenance.
Luxottica group is a manufacturer of sports glasses, such brands as Ray-Ban, Persol and Oakley. The Big Data technology company uses to analyze the behavior of potential customers and "smart" SMS marketing. As a result, the Big Data Luxottica group has identified more than 100 million of its most valuable customers and increased the effectiveness of the marketing campaign by 10%.
Using Yandex Data Factory, World of Tanks game developers analyze player behavior. Big Data technologies allowed analyzing the behavior of 100 thousand World of Tanks players using more than 100 parameters (information about purchases, games, experience, etc.). As a result of the analysis, a forecast of users outflow was obtained. This information allows you to reduce user care and work with participants in the game targeted. The developed model turned out to be 20-30% more efficient than standard tools for analyzing the gaming industry.
The German Ministry of Labor uses Big Data in its work related to the analysis of incoming applications for unemployment benefits. So, after analyzing the information, it became clear that 20% of benefits were paid undeservedly. With the help of Big Data, the Ministry of Labor has cut spending by 10 billion euros.
Toronto Children's Hospital has implemented Project Artemis. This is an information system that collects and analyzes data on infants in real time. The system monitors every second 1260 indicators of the state of each child. Project Artemis allows you to predict the unstable state of the child and begin the prevention of diseases in children.
REVIEW OF THE WORLD MARKET OF BIG DATA
Current state of the world market
In 2014, Big Data, according to Data Collective, became one of the priority areas of investment in the field of venture capital industry. According to the information portal Computerra, this is due to the fact that developments from this area have begun to bring significant results for their users. Over the past year, the number of companies with implemented projects in the field of big data management increased by 125%, the market volume grew by 45% compared with 2013.
Most of the revenue of the Big Data market, according to Wikibon, in 2014 were services, their share was equal to 40% in the total revenue (see the diagram below):
If we consider Big Data for 2014 by subtypes, the market will look like this:
According to Wikibon, applications and analytics account for 36% of Big Data revenue in 2014 brought Big Data Analytics applications and analytics, 17% computing equipment and 15% storage technology. The least revenue was generated by NoSQL technologies, infrastructure equipment and the provision of a network of companies (corporate networks).
Big Data technologies such as in-memory platforms from SAP, HANA, Oracle and others are the most popular. T-Systems survey results showed that they were chosen by 30% of surveyed companies. NoSQL platforms were the second most popular (18% of users), companies also used analytical platforms from Splunk and Dell, 15% of companies chose them. The least useful for solving the problems of Big Data, according to the survey results were Hadoop / MapReduce products.
According to an Accenture survey, in more than 50% of companies using Big Data technology, the cost of Big Data ranges from 21% to 30%.
According to the following Accenture analysis, 76% of companies believe that these costs will increase in 2015, and 24% of companies will not change their budget for Big Data technology. This suggests that in these companies, Big Data has become an established IT direction, which has become an integral part of the company's development.
The results of the Economist Intelligence Unit survey survey confirm the positive effect of the implementation of Big Data. 46% of companies say that using Big Data technology they improved customer service by more than 10%, 33% of companies optimized reserves and improved the productivity of fixed assets, 32% of companies improved planning processes.
Big Data in different countries of the world
Today, Big Data technologies are most often being introduced in US companies, but now even other countries around the world have begun to show interest. In 2014, according to IDC, Europe, the Middle East, Asia (excluding Japan) and Africa accounted for 45% of the software market, services and equipment in the field of Big Data.
Also, according to the CIO survey, companies from the Asia-Pacific region are rapidly developing new solutions in the field of Big Data analysis, secure storage and cloud technologies. Latin America is in second place in terms of the number of investments in the development of Big Data technology, ahead of the countries of Europe and the USA.
Next, a description and forecasts of the development of the Big Data market of several countries will be presented.
China
The volume of information in China is 909 exabytes, which is 10% of the total amount of information in the world, by 2020 the amount of information will reach 8060 exabytes, the share of information in global statistics will increase, in 5 years it will be equal to 18%. The potential growth of China's Big Data has one of the fastest growing speakers.
Brazil
Brazil at the end of 2014 has accumulated information for 212 exabytes, which is 3% of the world total. By 2020, the amount of information will increase to 1600 exabyte, which will be 4% of the information from the whole world.
India
According to EMC, India’s accumulated data for 2014 is 326 exabytes, which is 5% of the total information. By 2020, the amount of information will grow to 2,800 exabytes, representing 6% of the information from the whole world.
Japan
Japan’s accumulated data for 2014 is 495 exabytes, which is 8% of the total information. By 2020, the amount of information will increase to 2,200 exabytes, but the share of the Japanese market will decrease and amount to 5% of the total information volume of the whole world.
Thus, the volume of the Japanese market will decrease by more than 30%.
Germany
According to EMC, the amount of accumulated data in Germany at the end of 2014 is 230 exabytes, which is 4% of the total information in the world. By 2020, the amount of information will grow to 1100 exabytes and will be 2%.
In the German market, Experton Group predicts that a larger share of revenue will generate a service segment, whose share in 2015 will be 54%, and in 2019 it will increase to 59%, while the share of software and hardware will, on the contrary, decrease.
In general, the market volume will increase from 1.345 billion euros in 2015 to 3.198 billion euros in 2019, the average growth rate will be 24%.
Thus, based on the CIO and EMC analytics, it can be concluded that the developing countries of the world will become the markets for the active development of Big Data technologies in the coming years.
Main market trends
According to IDG Enterprise, in 2015, the companies' expenditures on the Big Data will average $ 7.4 million for a company, large companies intend to spend about $ 13.8 million, small and medium ones - $ 1.6 million .
Most will be invested in areas such as data analysis and visualization and their collection.
According to current trends and market demand, investments in 2015 will be used to improve data quality, improve planning and forecasting, and also to increase the speed of data processing.
The financial sector companies, according to Bain Company's Insights Analysis, will make significant investments, so in 2015 it is planned to spend $ 6.4 billion on Big Data technologies, the average investment growth rate will be 22% by 2020. Internet companies plan to spend $ 2.8 billion, the average growth rate of the increase in costs for Big Data will be 26%.
During the survey of the Economist Intelligence Unit survey, the priority directions for the development of Big Data in 2014 and in the next 3 years were identified, the distribution of answers is as follows:
IDC forecasts the market development trends are as follows:
- In the next 5 years, the costs of cloud solutions in the field of Big Data technologies will grow 3 times faster than the costs of local solutions. Hybrid storage platforms will be in demand.
- The growth of applications using complex and predictive analytics, including machine learning, will accelerate in 2015, the market for such applications will grow 65% faster than applications that do not use predictive analytics.
- Media analytics will triple in 2015 and will be a key driver for the growth of the Big Data technology market.
- The trend towards introducing solutions to analyze the constant flow of information that is applicable to the Internet of Things will accelerate.
- By 2018, 50% of users will interact with services based on cognitive computation.
Drivers and Market Limiters
IDC experts identified 3 drivers for Big Data Market 2015:
- Mass acquisitions of the client base of companies offering mobile applications and other data platforms;
- Development of cloud infrastructure;
- Changes in data privacy laws.
- In addition, it is also worth highlighting:
- Increased interest in the processing of media materials, previously referred to unstructured information;
- The growing popularity of training courses in the field of Big Data;
- Investing in data visualization and active storytelling by data analysts;
- Constant investment in Big Data by web giants such as Google, Amazon, Facebook, etc.
- Among the constraints of the Big Data market there are:
- The still high cost of implementing Big Data technologies;
- The need to ensure the protection of data and their privacy;
- Lack of qualified personnel;
- Mistrust of companies to these technologies;
- Insufficient amount of accumulated information;
- Database support requires constant funding, which creates an additional barrier to the implementation of Big Data;
- The complexity of integration with existing systems;
- Limited number of data providers.
According to the Accenture survey, data security issues are now the main barrier to the implementation of Big Data technologies, more than 51% of respondents confirmed that they are concerned about ensuring data protection and their confidentiality. 47% of companies reported the impossibility of implementing Big Data due to a limited budget, 41% of companies indicated a shortage of qualified personnel as a problem.
Wikibon predicts that the Big Data market will grow in 2015 to $ 38.4 billion and increase by 36% compared with the previous year. In the coming years, there will be a decline in growth rates to 10% in 2017. Taking into account these forecasts, the market volume in 2020 will be equal to 68.7 billion US dollars.
The distribution of the global Big Data Market by business category will be as follows:
As can be seen from the diagram, most of the market will be occupied by technologies from the scope of customer service improvement. Point marketing will be in second place in priority for companies until 2019, in 2020, according to Heavy Reading, it will give way to solutions to improve operational efficiency.
The highest growth rate will also be at the “customer service improvement” segment, an increase of 49% annually.
The market forecast for Big Data subtypes will be as follows:
The prevailing market share, as can be seen from the chart, is occupied by professional services, applications with analytics will have the highest growth rates, their share will grow from the current 12% to 18% in 2020 and the volume of this segment will be equal to 12.3 billion USD. the share of computing equipment, on the contrary, will fall from 20% to 14% and will be about $ 9.3 billion in 2020, the cloud technology market will gradually increase and reach $ 6.3 billion in 2020, the market share of solutions for Data storage, on the contrary, will decrease from 15% in 2014 to 13% in 2020 and in monetary terms. yrazhenii will be equal to US $ 8.9 billion..
According to the forecast of Bain & Company's Insights Analysis, the distribution of the Big Data market by industry in 2020 will be as follows:
- The financial industry will incur Big Data costs of $ 6.4 billion with an average growth rate of 22% per year;
- Internet companies will spend $ 2.8 billion and the average cost growth rate will be 26% over the next 5 years;
- The costs of the public sector will be commensurate with the costs of Internet companies, but the growth rate will be lower - 22%;
- The telecommunications sector will grow at an average growth rate of 40% and will reach $ 1.2 billion in 2020;
Energy companies will invest a relatively small amount of $ 800 million in these technologies, but the growth rate will be one of the highest - 54% annually.
Thus, the largest share of the Big Data market in 2020 will be occupied by companies in the financial industry, and the energy sector will be the fastest growing sector.
Following analysts' forecasts, the total market volume will increase in the coming years. Market growth will be achieved through the implementation of Big Data technologies in developing countries around the world, as can be seen from the graph below.
The projected market volume will depend on how developing countries perceive Big Data technologies, whether they will be as popular as in developed countries. In 2014, the developing countries of the world accounted for 40% of the accumulated information. According to the EMC forecast, the current market structure, with a predominance of developed countries, will change in 2017. According to EMC analytics, in 2020 the share of developing countries will be more than 60%.
According to Cisco and EMC, developing countries of the world will actively work with Big Data, in many respects this will be related to the availability of technology and the accumulation of sufficient information to the level of Big Data. The world map presented on the next page will show a forecast of the increase in volume and growth rate of Big Data by region.
ANALYSIS OF THE RUSSIAN MARKET
Current state of the Russian market
According to a study by CNews Analytics and Oracle, the maturity level of the Russian Big Data market has increased over the past year. Respondents representing 108 large enterprises from different industries demonstrated a higher degree of awareness of these technologies, as well as an established understanding of the potential of such solutions for their business.
As of 2014, according to IDC, Russia has accumulated 155 exabytes of information, which is only 1.8% of world data. The amount of information by 2020 will reach 980 exabyte and will take up 2.2%. Thus, the average growth rate of information will be 36% per year.
IDC estimates the Russian market at $ 340 million, of which $ 100 million is SAP solutions, and about $ 240 million is equivalent to Oracle, IBM, SAS, Microsoft, and others.
The growth rate of the Russian Big Data market is at least 50% per year.
It is predicted that positive dynamics will continue in this sector of the Russian IT market, even in the context of general economic stagnation. This is due to the fact that the business continues to place demands on solutions that will improve work efficiency, as well as optimize costs, improve forecasting accuracy and minimize potential risks for the company.
The main providers of Big Data services on the Russian market are:
- Sap
- Oracle
- Ibm
- EMC
- Microsoft
- IBS
- Cloudera
- Hortonworks
- Teradata.
Market overview by industry and experience of applying Big Data in companies
According to CNews, in Russia only 10% of companies started using Big Data technologies, when in the world the share of such companies is about 30%. Readiness for Big Data projects is growing in many sectors of the Russian economy, according to a report from CNews Analytics and Oracle. More than a third of the companies surveyed (37%) started working with Big Data technologies, among which 20% are already using such solutions, and 17% are starting to experiment with them. The second third of respondents are currently considering this possibility.
In Russia, Big Data technologies are more popular in the banking and telecom industries, but they are also in demand in the mining, energy, retail, logistics companies and the public sector.
Further examples of the use of Big Data in Russian realities will be considered.
Telecom
Telecom operators have one of the most extensive databases, which allows them to carry out the most in-depth analysis of the accumulated information.
One of the applications of Big Data technology is subscriber loyalty management.
The main purpose of data analysis is to retain existing customers and attract new ones. For this, companies conduct customer segmentation, analyze their traffic, determine the social identity of the subscriber. In addition to using information for marketing purposes, telecom technology is used to prevent fraudulent financial transactions.
. , , , .
Big Data . , , , . 55%.
Alfa-Bank analyzes information from social networks, processes applications for a loan, and analyzes the behavior of users of the company's website.
Sberbank also began processing a data set with the aim of customer segmentation, fraud prevention, cross-selling and risk management. In the future, it is planned to improve the service and analyze customer actions in real time.
The Russian Regional Development Bank analyzes the behavior of plastic card holders. This allows you to identify atypical for a particular client operation, thereby increasing the likelihood of identifying theft of money from plastic cards.
Retail
, . , CNews Analytics, Big Data 20% . 75% . Hadoop 7-10%.
. SAP HANA, , , 10 3, 3 30 .
Wikimart .
One of the first offline stores to implement Big Data analysis in Russia was Lenta. With the help of Big Data, retail began to study information about customers from cash receipts. The retailer collects information for the compilation of behavioral models, which makes it possible to make more informed decisions at the level of operating and commercial activities.
Oil and gas industry
In this industry, the scope of application of Big Data is quite wide. Big Data technologies can be applied to the extraction of minerals from the depths. With their help, you can analyze the process of extraction and the most effective ways to extract it, monitor the drilling process, analyze the quality of raw materials, as well as the processing and marketing of the final product. In Russia, these technologies have already begun to use Transneft and Rosneft.
Government agencies
In countries such as Germany, Australia, Spain, Japan, Brazil and Pakistan, Big Data technology is used to resolve issues of a national scale. These technologies help public authorities to more effectively provide services to the public, provide targeted social support.
In Russia, these technologies began to be developed by such state bodies as the Pension Fund, the Federal Tax Service and the Mandatory Medical Insurance Fund. The potential of implementing projects using Big Data is large, these technologies could help to improve the quality of services, and, as a result, the standard of living of the population.
Logistics and transportation
Big Data . , , .
Big Data SAP. 43,5 ( 14,5 20 ), 40 . Big Data . 300 SAP, 4 -, 220 000.
, , .
:
- , ;
- - ;
- , , ;
- . , - .
- , 2 , , .
- , , ;
- -, .
Big Data :
- ;
- ;
- Big Data ;
- ;
- , , ;
- , ;
- , IDC, .
, . , .
, , .
. IDC , 5 , .
2020 Big Data 1,8% 2,2% . , EMC, 155 980 2020 .
.
CNews Analytics, 44% 100 * 13% 500 .
, , . 2014 IDC 340 . .
50% , , 2018 1,7 . . 3%, 1,2%.
Big Data :
- , , ;
- – ;
- – , .;
- – ;
- – , .
Big Data , , , - , Big Data .
, , , .
, , , , , .
, , Big Data , .
2014 :
- 28,5 . , 45% ;
- Big Data , 40% ;
- 36% , 17% — 15% — ;
- in-memory , SAP, HANA Oracle.
- 125% ;
:
- 2015 38,4 . , 2020 – 68,7 . ;
- 16% ;
- 13,8 . 1,6 . ;
- technologies will have the greatest prevalence in the areas of customer service and point marketing;
- in 2017, the global market structure will change towards the predominance of user companies from developing countries.
Russian market
The Russian market of Big Data is at the stage of formation, the results of 2014 are as follows:
- market volume reached $ 340 million;
- the average market growth rate in previous years was 50% annually;
- the total amount of accumulated information was 155 exabytes;
- 10% of Russian companies began to use Big Data technologies;
- The Big Data technology has become more popular in the banking industry, telecom, Internet companies and retail.
The forecast of the Russian market for the coming years is as follows:
- 2015 500 . , 2018 – 1,7 . ;
- 3% 2018 ;
- 2020 980 ;
- 2,2% 2020 ;
- , .
, Big Data , .
, , — !
Source: https://habr.com/ru/post/256747/
All Articles