Artem Astafurov: "We started with a soldering iron, but stopped on a cloud"

The leader of IoT DataArt practice tells how the Internet of Things has changed over the past three years, about new engineering challenges and personnel difficulties of the market, about where the DeviceHive platform is heading and what the Big Data Academy is.

- IoT a few years ago and today - two completely different stories?
')
“We started this journey about four years ago, and then the IoT direction began with sensors connected to the Internet and smart homes. Of course, now much of what seemed important and original at the time, looks rather naive. But on the whole, it is the work done from the very beginning to the present day that gives us an experience on which we can rely. Today, the Internet of Things is the second big Hip after Big Data. We can say that as soon as everyone finished talking about Big Data in the Enterprise, they immediately began to say how large corporations will apply IoT in practice.

However, essentially one thing is a logical continuation of the other: in IoT we deal with a large amount of data and events that need to be analyzed in real time and then draw conclusions from them. We are dealing with what used to be called simply analytics, and now they are called predictive analytics - we need, based on past and present events, to predict how the situation will change in the future. And the roots of many of the most important and interesting engineering problems of our time really grow from the Internet of Things, because along with finance, online advertising and social networks, it is things that generate the most events and data. And just receiving, transmitting, analyzing and storing are of the greatest interest for DataArt, and for enterprises, and, probably, for the entire IT community.

Initially, there was a hypothesis that we would help connect devices to the Internet and provide tools for interacting with these devices — this is how DeviceHive was born. In practice, it turned out that some of the most interesting problems arise around the large amounts of data that these devices generate. So our tasks went into the area of ​​creating distributed systems that can pump this data through themselves and make it possible to build various analytics on their basis in order to respond differently to complex events.

- Things are capable of generating powerful data streams, but what kind of volumes are we talking about?

- We are talking about terabytes and even petabytes of information. We are talking about data that has long been located on one server and, frankly, within the framework of the tasks that we solve, the very concept of a server no longer exists. We operate with the concepts of cloud infrastructure, in which, as the task becomes more complex and the amount of information increases, the necessary resources are simply added. We do not even know where our specific task is being performed, since we never look at these machines separately. We use them in dozens, and sometimes hundreds, and for us it’s just the computing power that pumps our data through.

- Will the necessary capacity in the future unite on a limited number of platforms? And will this increase competition between platforms?

- This is true. Now the main players are known: Amazon, Microsoft, Google and IBM. And we can discuss for a long time how the industry will look in five years, but it is worth admitting that we simply do not know. For example, many people think that IBM is somewhat behind the times. But let's remember what happened with IBM and personal computers in the 80s. Apple then opened this door, and in the middle of the decade lost this niche - it was occupied by IBM and Microsoft. So no one can predict how things will go with the same Amazon, which today, of course, dominates. Enterprise them - all corporations want to move to the infrastructure of Amazon, getting rid of their own. But we still have to see which card IBM will play.

- And Microsoft?

- You definitely shouldn't forget about him - lately he has looked very strongly against the general background. I wonder what they will do next after buying LinkedIn. Because, as we understand, it is not only about infrastructure for business applications and analytics, but also about data - about where this data is and what it means for business. A large amount of information from LinkedIn to Microsoft Azure can make accessing it a very attractive deal for many who want to use the cloud.

- Are the local platforms sharpened for specific tasks exactly a thing of the past?

- You can just look at how the evolution of operating systems. At first there were a lot of local options, then the main players appeared, covering almost the entire industry. They occupied their niches - some in personal computers, others in the fast-growing server systems of the Internet boom times. With cloud technologies, we can get a similar picture - large clusters of problems solved with the help of this or that infrastructure and this or that cloud offer are already outlined.
As I said before, most large companies want at Amazon, more conservative want at IBM and expect to be offered. However, after this offer is made, it is not known how all those who aspired to Amazon will behave. Many are afraid to be completely dependent on Amazon and build their systems so that they can be transferred from cloud to cloud - this is another non-trivial task that we help clients solve.

- Do the tasks facing the practice within the company change with the industry?

- In general, the history of IoT brought us to a very interesting place, which we did not expect to be in: our company was simply open to the world, which is why it brought us here. Now we are talking about parallel computing, working with large amounts of data, machine learning, cloud systems and the design of systems for the cloud. All this basically changed our view of things. Of course, serious preparation is required here, and in this regard, we have turned our open source DeviceHive platform in the same direction.

Three years ago we made DeviceHive, having a certain picture in mind: we saw it as a kind of web server to which devices would connect and send messages that we could read using web applications. Then we realized that for the real scale of the IoT tasks, the old design does not quite fit - we had to look at the architecture in a different way. We learned the lessons and began to develop DeviceHive, this was attended by several generations of teams. As it developed, DeviceHive also became a learning platform. New professionals who are currently gaining experience working with DeviceHive will then help our customers build scalable distributed systems.

- What tasks does IoT practice in DataArt now solve for clients?

- In most of our projects, we act not only as developers, but also as consultants. When a client asks us for a resume of people who own all the modern cloud technologies, we try to turn the conversation in the direction of specific problems that companies need to solve. We propose to look at the real situation together and decide how we can help, based on the total experience of the practice. Since IoT is a very hot market, where there is an acute shortage of staff, it usually responds to such a proposal very well.

- Are industry experts really badly lacking?

“We are preparing personnel with the help of DeviceHive, new engineers have the opportunity to participate in the development of the platform and write it down in the resume. Since the area itself is very new, it is difficult for us to build expertise only on projects that we do for clients. After all, to declare experience, you need to have it, and for this you need clients who are able to give tasks, solving which, we will gain this experience. Standard solutions in this area have just begun to emerge, and so far we have to train a lot ourselves. On the one hand, we understand how we can make DeviceHive better, on the other - engineers, using our own project as an example, are learning how to solve problems.

Another interesting structure emerged at the DeviceHive site - the Big Data Academy, which brings together people who care about the subject of big data and distributed systems and cloud computing. There are courses, topics for discussion, tasks, a forum where you can ask for advice or share interesting information. On this platform, there are groups that go, for example, to participate in machine learning competitions.

- Is it true that the requirements for IoT engineers are particularly high?

- This is true, but the growth in requirements is a general trend of the entire industry, associated with the development of production tools. I would even call it the industrial revolution within the industrial revolution, when more traditional, classical approaches are replaced by automated or more innovative ones. About the same happens with us. Many problems can still be solved with the help of less advanced technology, but they have their own limitations, a certain cost of support, some potential problems, which so far may not be so important in a particular project, but later more and more often manifest themselves. In this case, you need to turn around, look for new personnel and help your specialists with all the forces to master the technologies, to communicate more with those who have already received the necessary experience. For this, we came up with the Big Data Academy, where any curious person can go. There are no introductory tests - everyone can see if this topic is interesting to him.

But in general, the requirements for personnel are changing, which allows us to interest engineers in the labor market, to whom earlier our proposals might not seem so interesting. After all, one cannot say that these personnel are not in principle - just some time ago we did not have such a volume of tasks of sufficient complexity. Now the range of tasks capable of truly captivating engineering-minded specialists has seriously expanded.

- Ie. IoT - direction for these geeks?

- In a manner. This makes certain adjustments to how we provide service to the client. We see fewer interviews and resumes, and more and more often we immediately help by doing pilot projects and turning them into long-term relationships. And I must say that many who work in our group really live and breathe it all.

“In just three years, DeviceHive has experienced a fundamental transformation. Is this related to the speed of development of the entire IoT direction? Will it continue like this?

- Yes of course. Till the end of what Agile practices are, we were able just in IoT projects, including DeviceHive. Having built a plan for the year, after a lapse of time, you can only laugh at him and in general at what you thought about your work. Naturally, you need to make basic restrictions in order not to try to solve all the problems at the same time. But to have a structure completely open to the future is the “flexible methodology of development”. Some things, further work on which seemed important when creating DeviceHive, today seem to be weakly irrelevant: some technologies simply disappeared as a standard practice in the industry and even ceased to be supported. The team is changing - after all, we are writing a platform inside a service company, which is not easy even from an organizational point of view. But due to the willingness to change, we are moving DeviceHive with the times. And this is a fantastic feeling.

- Security issues remain key to the IoT industry? Or from inside the possible risks do not look so scary?

- In the end, we are all humans. And if we look at Maslow's pyramid, we’ll definitely find safety somewhere at its base. Some now like to add wi-fi to the very bottom, but still no one denies that security is a fundamental need of each of us. We are constantly, and increasingly, confronted with objects related to the Internet of things. These are not “smart houses” and wearable devices, but practically all the technologies surrounding us: applications on phones, sending statistics of their use, telemetry from public transport, with the help of which they track maintenance periods, data taken from airplanes in the air, events occurring on advertising sites, etc. And I would not say that in terms of security, all this looks very rosy. We will live, but, probably, everything could be thought out and better.

Now we are just preparing another release of DeviceHive, which will be released in October - there a whole bunch of changes will concern security. In principle, we are primarily focused on safety and productivity, which are very closely linked with each other. This is a dynamic system in which, after tweaking one, you can lose a lot in the other - a platform that does not cause any safety issues will turn out to be weakly productive.

And if we are talking about hundreds of thousands of devices, the cost of infrastructure for such a system will simply be prohibitive. Thus, the main issue is the question of optimization: we look at what threats are real, how to prevent them, how it will affect the infrastructure and the cost of maintenance, and as a result, whether its deployment is real. Some projects related to medicine are not unfolding en masse precisely for security reasons. Probably, we would all like to wear devices that tell doctors when we need to be examined. But there are a number of limitations - and the question of the possibility of creating such a device is only half the problem. The second half is to transfer all this information safely. At the same time, engineers dealt with the first part more successfully, because regulators are also watching the second part. But technologies are evolving, and I think that soon enough we will see, among other things, a new generation of mass-market devices capable of collecting and transmitting health information.

- To what extent do IT in general understand what engineers at IoT are doing?

- I think that of those who personally do not overlap with us, quite a few represent IoT engineers as crazy people with soldering irons. Three years ago, this was partly true, since we started with devices and connecting them to the cloud. But we finished with cloud and data - we plan to stay here. In this area, there is a greater amount of synergy with what other practices do, what large clients need - and here there is money in the form in which it is convenient for the company to earn it.