Remote (dev) tools do it yourself - interview with Roman Dvornov (Avito)

Roman Dvornov ( twitter ) is the head of front-end development in Avito, the author of basis.js, CSSO Maintener, CSSTree, Component Inspector and more. Roman is working on a new project designed to simplify the development of tools for remote monitoring and debugging of web applications.

Let's find out what problems Roman's project will help to solve, and what he eventually came to.

- Tell me what a rempl is and why is it needed at all?
')
- rempl - a solution that allows you to get controlled remote access to the application's runtime. The solution is composite and consists of several diverse components.
We begin by creating a provider (observer), an entity that monitors an application (or a page, processes in a browser, etc.) and publishes some data. Next we create an interface (consumer) that visualizes this data or does something else with it. At the same time, the interface is some UI that can be launched in an arbitrary WebView and the data from the provider get into it in a “magic” way. The provider and the consumer can also exchange commands.

When we talk about an arbitrary WebView, we mean another tab, a tab of another browser, browser Developer Tools, plug-ins in editors and IDE, both on the same machine as the provider, and on any other.

Rempl allows you to simply create your own tool, that is, the provider, and some interface, provides all the transport (we create the provider and the consumer, and they connect and begin to communicate with each other), and also provides a number of hosts like a web interface that can be opened in any modern browser, plug-ins for browser Developer Tools (for now only Chrome, but there will be Firefox), various editors (for now Atom and VS Code) and so on. The plans also have an SDK that will simplify typical operations, such as working with data streams and working with DOM, for example, to highlight some element on the page.
To get an idea of ​​how this works, you can watch the video - this video shows the tools for basis.js, on which the approach was run-in. Actually, when working on these tools, it became clear that the approach is quite universal. Namely, most of it, which concerns transport and hosts, does not depend on the specifics of the framework or application. We can assume that the implementation of tools in basis.js became the prototype for rempl - a universal solution for creating such tools. When everything worked with the tools of basis.js, I applied for HolyJS and started to cut rempl :)

Provider and consumer

- What is the provider? Does this entity only provide an API for interacting with the tool, and should the tool itself collect the data?

- The provider and the customer are the roles. Now there is some problem with terminology, but what comes out close to the publish-subscribe pattern and the producer-consumer problem.



The provider is the part that generates and publishes data, and the consumer uses them. Rempl allows you to get an API for publishing on one side (the provider side) and receive data on the other (consumer). How the data is collected and in what form to publish, the developer decides himself when creating the provider. In the same way as what to do with this data on the consumer side. There are no limitations in methods and technologies, in this case rempl only facilitates the transfer of data from one side to the other.

- Under what protocol is the communication between the provider and the consumer? Do you need an intermediary (server) for this interaction? Will it work in IE?

- It all depends on where the provider and the consumer is hosted. If the provider is hosted in the browser or node.js process, and the consumer is in a different browser tab (or in another browser), then communication occurs via socket.io. In this case, yes, an intermediate is used as a server. The server is not only a link, but also provides a web interface with a list of all connected providers, the ability to select the desired and organized launch of the user interface of the selected provider in the sandbox (sandbox). The latter is essentially the host for the consumer.

Since socket.io is used for communication, either WebSocket or long polling will be used as folbek. That is, in IE, theoretically, in terms of transport, everything should work.
But socket.io is not always used. For example, you cannot communicate with browser Developer Tools via web sockets. In this case, communication is provided through DOM events. Moreover, in order to transfer something to your bookmark, you need to connect 4 nodes together: provider <-> content script <-> background page <-> plugin (there is a diagram at the beginning of the article ). This is not a trivial task, but with rempl you do not need to think about it.

- When you talk about data collection by the provider, do you mean monitoring the client part of the application?

- Monitoring is only one of the possible directions as it can be used. Yes, the initial idea was precisely in the area of ​​the client side of applications and sites. But in the course of discussions with colleagues about the use of rempl, already during the preparation of the report, the understanding suddenly came that the provider can live not only in the browser, but also in the node-process. Thus, all the same infrastructure has become available for node-applications. So you can get information about internal processes in a convenient and intuitive interface.

For example, there is such a project as a deshborb for Webpack , which draws in the console information about what is happening in a more visual form than the usual output Webpack'a. A colleague made a clone of this plugin , the main difference of which is that the information is displayed using web technologies and can be performed in an arbitrary WebView. This provides much more visualization capabilities and a richer UX. For example, you can fasten ready-made solutions for analyzing bundles, such as source-map-explorer and others. At the same time, it is not necessary for the analysis of the bundle to be on the disk, because the provider has access to the Webpack runtime, and there is a lot of room for imagination.

Development tools are still missing

- That is, we should expect the emergence of new tools that will be better than the old ones?

- Time will tell, but I really hope so :) One of the main goals, besides providing a ready-made solution, is to lower the entry threshold for developers to make their own tools. I believe that a lot of tools do not happen, and your productivity depends very much on what you use in development. Despite the seemingly abundance of development tools, they are still not enough. This is due to the fact that tools are often done by other developers, solving their problems. But each has its own needs, tasks, perhaps a unique technology stack or its own framework. In this case, the solution will be the development of its tools. But much will have to be done, especially if there is a goal to launch the tool outside the page in which the application is running. It takes a lot of time and nerves.

Moreover, it can not be called useful work - it is a kind of overhead projector, which separates you from the development of the tool itself. So, rempl allows you to avoid unnecessary work and focus directly on the tool, which is much simpler and more interesting.

There is another important point. Different hosts for the tool interface can provide additional features. Especially the hosting tool in the editor - because this is the door to the world of new opportunities for web development. Now there are many tools that help to write code and analyze it. There are tools that show something about what is happening inside the application (in its runtime). And these two categories of tools live separately. But what if we unite these two worlds? So, having a loaded tool interface in your editor / IDE that has remote access to runtime - you can get new features. Just imagine, we have information about the code and runtime in one place! And with this you can definitely do something. For example, editing JavaScript, CSS or templates, you can highlight on the page that affects the edited file. Or in the templates to suggest what kind of binding is available. Or show what Webpack resolves require, with the ability to go to the desired file. Or what the current file is, and so on.

In fact, I myself do not yet know all the possible cases, but I can say for sure that there are a lot of opportunities and new ideas are constantly appearing. It is very interesting what others will think up :)
I saw the border between code and runtime in editors for a long time, and I realized that it can be overcome. But in order to somehow dispose of it in their tools, we had to do a lot. To rempl anyway, I have been going for more than five years, creating intermediate solutions. Many refactorings and work in the direction of "make the mind" led to a dead end and did not have a happy ending - patience and time were running out. I am glad that a complete solution is emerging now, although there is still a lot of work to be done.

This is an example of the fact that the solution of routine engineering problems removes useful work. Now I am only coming to realize many of my ideas. The path was thorny, and I definitely will not advise anyone to repeat it. I consider it my small victory that many will not have to, it will be enough to take the rempl and do what they have long wanted for their project or stack.

Developing such a tool is hard labor.

- How will rempl affect existing tools?

- This question is difficult to answer, as it is difficult to predict the reaction of the developers of the tools. Whether they want to transfer their tools to a common platform or continue to cut their own. One thing I can say, the basis.js tools accurately migrate to rempl and begin to use new features, which should have a positive impact on their functionality. Component Inspector will also be transferred to rempl. And I think that in our company, soon there will be new tools on the basis of rempl, solving urgent problems.

Speaking about the existing tools, I would like to talk about several problems. And although the problems are different, oddly enough, one follows from the other.

First, the problem of transport and hosts only seems simple. There are a lot of pitfalls in it and, I repeat, this is not the simplest and even more interesting task. If you look at the developers of the tools (usually they are tools for frameworks in the form of plug-ins for browser-based Developer Tools), then they all walk in the same minefield. That is, each developer re-implements, from scratch, everything that is necessary, namely transports, hosts, etc. At the same time losing time, solving the same problems as the rest. With rempl, tool developers will be able to remove their solution for different platforms (hosts) and take care only of the main functionality of their tool. For example, in react-devtools it may be possible to delete the shells folder - you can study how complex the code is there.

Secondly, the complexity of the development and development of the tool. For example, tools for frameworks are often limited to the plugin for browser Developer Tools. The development of such a tool is hard labor. After all, to see the changes, you need to rebuild the plugin, reload it in the browser settings (in Chrome, these are Extensions), reload the page where the plugin is used, close the Developer Tools browser, reopen them and select the tab of your plugin. And this needs to be done every time you make a change. We ourselves went through it, very quickly get bored and start to enrage.

It is much easier to run the same interface as in the plugin, in a separate tab of the browser and so to develop. And when everything is ready, just run this interface in the Developer Tools. Everything becomes much simpler: in order to see the changes, it is enough to reload the page, and if the stack on which the interface is created supports live update or live reload, then this is usually not necessary. But in order to organize this, you need to implement another type of transport and launch the interface in an environment other than Developer Tools. And here is someone who gets out of the situation, but, in any case, the process cannot be called pleasant and fascinating. Therefore, such tools are developing slowly, or even their development stops. Rempl allows you to run your tool (its interface) in the host that is most suitable for the task you are solving, and you do not need to do something special for this.

Third, versioning. When you make a tool, for example, for a framework that develops, you need to be prepared for something in the framework to change and the tool will stop working. Actually, this is what happens, the framework changes - the tool changes after it. But the tool cannot work only with the latest version of the framework: not everyone migrates quickly, and some projects can be supported at all, and updating libraries can be very expensive and irrational.

To remain completely without tools in such cases is also bad. Therefore, the tools “learn” to work with different versions of the framework, which eventually turns their code into porridge and makes it very difficult to add new functionality. We also did this with the tools of basis.js and this is a big pain, especially for middle-aged frameworks. Rempl solves this problem with its interface distribution model. Namely: a host (for example, a plugin for browser Developer Tools) is just a sandbox in which an arbitrary script can be executed. The host does not know anything about the interface. When a provider (tool) is selected from the page with which to work, the host sends a request “give me your interface” to the provider, and in response the provider gives the script (bundle that has everything you need to build the interface). The host receives and executes this script in the sandbox and you see the tool interface. Thus, for each version of the framework, you connect your own version of the provider (tool) on the page. As a result, depending on the version of the framework in the Developer Tools, you will see one or another version of the tool, the versions may differ in functionality and are tied to a specific version of the framework. Support of the tool in this case is greatly simplified and you do not need different browser plug-ins in order to work with different versions of the framework.

Summing up, I can say that existing tools can be great to win by going to a solution like rempl, since it is taken to solve a fairly wide range of problems. In addition, it can also simplify the lives of developers who use development tools, since they can not install separate plug-ins for different frameworks and libraries.

- I am very surprised that there is nothing ready in this area.

- The few are engaged in the topic, as it is not simple. A custom solution, though not easy to make, is much simpler than the universal one. Well, you need to cook in the subject for a considerable time so that it finally suffices and see ways to solve the problem;)

“Thank you for your time.” Tell me when and where you can see the result?

- Everything related to the project is available on GitHub in the organization rempl . True, now there is a “working mess” and not everything is ready yet, but everything should be much better for speaking at HolyJS. In any case, you can like and subscribe to updates today;)

During the speech itself on HolyJS, December 11, I will try to tell you more and more structured, with code examples, diagrams and live demo. This will be the first public talk about the project. But I think this is far from the last - there are still many wonderful discoveries ahead of which can be heard at future conferences.

---

Anyone who is interested in this topic, we offer to visit the report of the Roman at the JavaScript conference HolyJS . You can also hear another 19 reports on problems, solutions and the future of the JavaScript world: we will talk about the front-end, the backend, and even the desktop.