4 reasons to become a Data Engineer
Hi, Habr! At the moment, there is a huge bias in Data Science towards the data scientist, even those who are in no way connected with IT are now aware of this profession, and new vacancies appear daily. In turn, data engineers do not receive the attention that would correspond to their importance for the company, so in today's post we would like to correct this injustice and explain why developers and administrators should immediately begin to learn Kafka and Spark and build their first pipeline.
Let's look at the typical data scientist working day:
It turns out that about 80% of his time data scientist spends on data collection, their preprocessing and cleaning - processes that are not directly related to his main duty: search for insights and patterns in the data. Of course, data preparation requires the highest level of skill, but this is not data science, this is not why thousands of people today are striving to get into this industry.
')
That is why companies should free data scientists from the least pleasant part of their work and delegate data preprocessing to data to the engineer, whose presence in the data science team, first, will allow the data scientist to do what they truly love - to build models, which in turn will prevent their potential withdrawal from the company and attract the most talented. And secondly, the efficiency of data scientists will increase, since they will spend many times more time searching for valuable insights, which will naturally benefit business.
Also, do not forget about the principle of garbage in - garbage out: if the models are fed with poor-quality data, it is pointless to expect an adequate result from them. Therefore, in order to maximize the effectiveness of the data science department of a company, it is necessary to hire data engineers who, unlike data scientists, specialize in organizing the process of collecting, cleaning and preprocessing data.
That's what Big Pilot thinks about this in Mail.ru Group, Anton Pilipenko: “At the moment, most companies have learned to store a large amount of data and build different models on their basis. However, often, the issues of efficient storage and processing of accumulated data are not given sufficient attention. As a result, constantly here and there questions arise about sizing, scaling applications, streaming and near-realtime processing. As experience shows, the division of specialists into Data Science and Data Engineer did not appear from scratch. Data Engineer is, first of all, an engineer who understands well what he is doing and why, how it is arranged “under the hood” and which architecture does not take off.
It's no secret that today the data scientist profession is becoming more and more popular, thousands of students around the world want to get a job in this industry, and many mature specialists from other fields are changing their specialization in favor of data science. The reason is simple - high salaries, the solution of analytical problems and the growing unmet demand for data analysts. All this can result in a large number of unqualified personnel who came to the trend area, without having sufficient knowledge of programming and statistics, while analysts who are really interested in building models will find it difficult to stand out among this mass.
Now, from the same point of view, we will look at data engineers, who have the opposite situation: at first glance, the duties of the data engineer look less interesting than those of the data scientist (which is naturally not the case), so hundreds of resumes do not fly to the mail to employers who are in search of a good data engineer, although the salaries of data engineers and data scientists are about the same level ( $ 90 and $ 91 thousand per year, respectively, in the United States ). People need to see the result of their work, and best of all - customer satisfaction and business satisfaction. The easiest way to enjoy your work is to learn about hundreds of new customers by building a model for creating personalized offers than from cleared data, so it is difficult for most to appreciate the importance of data engineers who are no less than a data scientist contribute to the final result.
Today, almost everywhere, the question often arises as to whether some professions will soon be replaced by artificial intelligence. With regard to data engineering, many have expressed the opinion that the process of collecting, processing and cleaning data is routine and can be easily automated, therefore the profession is unpromising. However, this opinion is incorrect, since the preparation of data for analysis is a real art, and the approach that worked with one dataset may not suit the other data set at all. The machines are not yet able to adapt themselves to the data, in the near future a data engineer will still be engaged in setting them up.
Moreover, data engineer’s responsibilities are even more complex than data preprocessing, the task of building stable pipelines that make data accessible to all users within the company. Only thanks to the data engineer, data scientists are provided with high-quality datasets in a convenient form and at the right time, this is the indispensability of the data engineer. How much it affects business processes and the company's success can be seen with the naked eye.
Professionals agree with this point of view: Senior Software Engineer at Agoda, Artem Moskvin says: „Data engineer is the one who makes all that big data about which you heard possible. Work with data can be divided into 2 parts: engineering and research. However, in order to make the second possible, you need to work well on the first, “and according to Data Engineer in E-Contenta, Andrei Sutugin:“ In the world of data analysis, not everything is as rosy and beautiful as it may seem after solving a Titanic On the kaggle. In order to proceed directly to the analysis itself, it is necessary to do titanic work, but in order to “put on stream” the collection and transformation of data, more effort is required. Unfortunately, there are no “silver bullets” in the “big data” world, and an abundance of tools and frameworks can turn the head ”.
Many people who want to build a career in IT, after 1-2 courses of technical universities with furious courses of mathematical analysis and probability theory, give up, believing that without an advanced mathematical background, they will not be able to find a job, even though they write good code . In this regard, data engineering is a great opportunity to start a career in working with data for people who have only a basic understanding of machine learning, but are also interested in developing databases and their management. Thus, such work, of course, is more suitable for software engineers, architects, and database administrators.
According to Nikolay Markov, Senior Data Science Engineer at Aligned Research Group LLC: “Why engage in Data Engineering? I believe that this is a logical way in the field of data analysis for people who can program and have experience in the development industry. The fact is that people are extremely rarely deeply interested, and in fact, in the other - at the same time a serious knowledge of mathematics and deep computer science is almost never found in one person. Therefore, let's leave the mathematicians what they do best - research, models and graphics, and think about what you need to do to make a working product out of an analytical idea? ”
On November 13, Newprolab launches the Data Engineer program, where participants will create stable pipelines for processing data from collection to visualization for 6 weeks, learn and hone skills with the following tools: Divolte, Kafka, ELK, Spark, Luigi, Sqoop, Druid, ClickHouse, Superset, Storm, which will combine into one large and stable pipeline. Learn more about the Data Engineer program.
Soon, no company can do without Data Engineer
Let's look at the typical data scientist working day:
It turns out that about 80% of his time data scientist spends on data collection, their preprocessing and cleaning - processes that are not directly related to his main duty: search for insights and patterns in the data. Of course, data preparation requires the highest level of skill, but this is not data science, this is not why thousands of people today are striving to get into this industry.
')
That is why companies should free data scientists from the least pleasant part of their work and delegate data preprocessing to data to the engineer, whose presence in the data science team, first, will allow the data scientist to do what they truly love - to build models, which in turn will prevent their potential withdrawal from the company and attract the most talented. And secondly, the efficiency of data scientists will increase, since they will spend many times more time searching for valuable insights, which will naturally benefit business.
Also, do not forget about the principle of garbage in - garbage out: if the models are fed with poor-quality data, it is pointless to expect an adequate result from them. Therefore, in order to maximize the effectiveness of the data science department of a company, it is necessary to hire data engineers who, unlike data scientists, specialize in organizing the process of collecting, cleaning and preprocessing data.
That's what Big Pilot thinks about this in Mail.ru Group, Anton Pilipenko: “At the moment, most companies have learned to store a large amount of data and build different models on their basis. However, often, the issues of efficient storage and processing of accumulated data are not given sufficient attention. As a result, constantly here and there questions arise about sizing, scaling applications, streaming and near-realtime processing. As experience shows, the division of specialists into Data Science and Data Engineer did not appear from scratch. Data Engineer is, first of all, an engineer who understands well what he is doing and why, how it is arranged “under the hood” and which architecture does not take off.
Data Engineer is easier to attract the attention of the employer.
It's no secret that today the data scientist profession is becoming more and more popular, thousands of students around the world want to get a job in this industry, and many mature specialists from other fields are changing their specialization in favor of data science. The reason is simple - high salaries, the solution of analytical problems and the growing unmet demand for data analysts. All this can result in a large number of unqualified personnel who came to the trend area, without having sufficient knowledge of programming and statistics, while analysts who are really interested in building models will find it difficult to stand out among this mass.
Now, from the same point of view, we will look at data engineers, who have the opposite situation: at first glance, the duties of the data engineer look less interesting than those of the data scientist (which is naturally not the case), so hundreds of resumes do not fly to the mail to employers who are in search of a good data engineer, although the salaries of data engineers and data scientists are about the same level ( $ 90 and $ 91 thousand per year, respectively, in the United States ). People need to see the result of their work, and best of all - customer satisfaction and business satisfaction. The easiest way to enjoy your work is to learn about hundreds of new customers by building a model for creating personalized offers than from cleared data, so it is difficult for most to appreciate the importance of data engineers who are no less than a data scientist contribute to the final result.
Data Engineers are almost indispensable in the company.
Today, almost everywhere, the question often arises as to whether some professions will soon be replaced by artificial intelligence. With regard to data engineering, many have expressed the opinion that the process of collecting, processing and cleaning data is routine and can be easily automated, therefore the profession is unpromising. However, this opinion is incorrect, since the preparation of data for analysis is a real art, and the approach that worked with one dataset may not suit the other data set at all. The machines are not yet able to adapt themselves to the data, in the near future a data engineer will still be engaged in setting them up.
Moreover, data engineer’s responsibilities are even more complex than data preprocessing, the task of building stable pipelines that make data accessible to all users within the company. Only thanks to the data engineer, data scientists are provided with high-quality datasets in a convenient form and at the right time, this is the indispensability of the data engineer. How much it affects business processes and the company's success can be seen with the naked eye.
Professionals agree with this point of view: Senior Software Engineer at Agoda, Artem Moskvin says: „Data engineer is the one who makes all that big data about which you heard possible. Work with data can be divided into 2 parts: engineering and research. However, in order to make the second possible, you need to work well on the first, “and according to Data Engineer in E-Contenta, Andrei Sutugin:“ In the world of data analysis, not everything is as rosy and beautiful as it may seem after solving a Titanic On the kaggle. In order to proceed directly to the analysis itself, it is necessary to do titanic work, but in order to “put on stream” the collection and transformation of data, more effort is required. Unfortunately, there are no “silver bullets” in the “big data” world, and an abundance of tools and frameworks can turn the head ”.
Data Engineering does not require in-depth knowledge of statistics and probability theory.
Many people who want to build a career in IT, after 1-2 courses of technical universities with furious courses of mathematical analysis and probability theory, give up, believing that without an advanced mathematical background, they will not be able to find a job, even though they write good code . In this regard, data engineering is a great opportunity to start a career in working with data for people who have only a basic understanding of machine learning, but are also interested in developing databases and their management. Thus, such work, of course, is more suitable for software engineers, architects, and database administrators.
According to Nikolay Markov, Senior Data Science Engineer at Aligned Research Group LLC: “Why engage in Data Engineering? I believe that this is a logical way in the field of data analysis for people who can program and have experience in the development industry. The fact is that people are extremely rarely deeply interested, and in fact, in the other - at the same time a serious knowledge of mathematics and deep computer science is almost never found in one person. Therefore, let's leave the mathematicians what they do best - research, models and graphics, and think about what you need to do to make a working product out of an analytical idea? ”
On November 13, Newprolab launches the Data Engineer program, where participants will create stable pipelines for processing data from collection to visualization for 6 weeks, learn and hone skills with the following tools: Divolte, Kafka, ELK, Spark, Luigi, Sqoop, Druid, ClickHouse, Superset, Storm, which will combine into one large and stable pipeline. Learn more about the Data Engineer program.
Source: https://habr.com/ru/post/337938/
All Articles