Script-server. WebUI to run your scripts remotely
Hello. In this article I would like to tell you about your home project. In short: Script server is a web server to provide users with access to your scripts through a web interface. The server and scripts are run locally, and are parameterized and displayed remotely.
At the new place of work, my first tasks were very routine and monotonous. Schematically, they looked like this:
There were more steps and they were more complicated, but after a couple of such iterations, laziness triumphed and shell scripts were written for all repetitive tasks that were run with several parameters and sometimes requested additional input data in the process. At the same time, the workflow began to look something like this:
')
All these points were obviously asking for one to collapse so that the project manager would specify the parameters and run the script himself. But giving him scripts was very problematic for a number of reasons - the scripts were adapted to my environment, they could fail, they were friendly enough for me, but not for non-technical people, etc.
It must be said that by a certain moment such tasks had already ceased to be assigned to me and the created scripts had a very illusive future without the necessary adaptation.
Therefore, I decided to create a tool that would allow remotely launching my scripts to other people, without the need to create the necessary environment and in a more convenient form to customize their launch.
The server administrator (i) creates configuration files for each script, which describes the purpose of the script, the launch path and the required parameters. These configuration files are used by the server to provide data about the scripts to the user, as well as to run them. Information on the user page is transmitted by Ajax requests.
The user fills in the parameters and runs the script on the server, where its execution is passed to a special handler. The handler accepts input in asynchronous mode and provides output, and also monitors the script execution process.
The web server serves as a layer between this handler and the page, and it communicates through web sockets.
At that time I was learning Python and decided to use it for development, as a good practice. Python 3 was chosen for the project, without support for version 2, since there is no need for this, and I did not want to waste time.
At the beginning of development for the server, I used Flask, but could not do ( read to understand ) the asynchronous connection and disconnection tracking of clients, so I quickly transferred everything to Tornado.
With regards to web development: there is no web developer out of me, so here I also decided to practice a little in syntax and basics. In this regard, almost no frameworks and libraries were used, with the exception of materializecss for more or less decent design.
A web page is a “one-page application” with a minimum of HTML and content creation in JS by Ajax requests.
About a month later I had a server that could somehow be used and I shared it with fellow developers, who also had to perform these routine tasks (which, unfortunately, almost ended in this period).
During operation, both bugs and obviously missing elements were discovered, on which I gradually worked and improved. At the same time, minor improvements were added, such as parameter types, logging of all launches, etc.
An interesting point: for some small tasks (related to working in the console and with scripts), I also stopped using the console, and switched to this UI. Those. I have got several configuration scripts that I use more often than other people.
After some time of local testing (by me and other developers), the tool was finally provided to the project manager, who began to use it little by little. And it should be noted that he is very pleased, because it saves including his time. The process of "implementation" and user training took about 5 minutes.
Due to the peculiarities of the processes and limitations, the Script server is mainly used for the test environment. But some of its parts have already been used for Production (in the part where it is safe).
Below is a list of the problems and my personal discoveries that I remembered the most.
Transfer of output from the script execution process to the page in asynchronous mode. This was due to a lack of experience in both Flask and the web: how best to organize such a transfer. As a result, I implemented data transfer in the form of data (type and data) on the websockets. Also in events you can send other commands at runtime, but for now this is not used.
Transfer user input to executable script. I tried to make a request for input only while waiting for one in the script, but could not find a way. Therefore, during the execution of the script, users can enter as much information as they want (the usual terminal behaves the same way). Data is sent via the same web socket.
Track user disconnection. It was precisely because of this problem that Flask refused: at that moment I was doing it on SSE (almost), not on websockets. However, later I did transfer the input / output transfer of the script to the web sockets, so maybe Flask would do. In Tornado, you can just subscribe to the closure of the website.
Read script execution output. There was a problem in order to read only the current available data and send it to the client. It is impossible to read in rows, because for example, “read -p input_prompt” does not transfer to the next line. You can read by symbols, but you should not send so many requests to the client. Reading the buffer gives the string truncated in obscure places (and also in the case of UTF-8, this generates the wrong characters). The current solution is replete with a set of compromises and crutches, but in general it is a read-by-byte reading that disables read locks.
Output scripts in (non) terminal mode. Honestly for me it was a discovery: the output of scripts can be different if you run them in a terminal or out. For example, the same ...
... shows input_prompt only in terminal mode. It is not very good to expect anything from users without showing them exactly what. I had to deal with pty to deceive the scripts being run. For this reason, I now have two types of wrappers for running scripts: with and without terminal support. The first is enabled by default, the second can be enabled using the configuration (left it just in case the terminal mode fails).
Autoscroll script output. On the one hand, this is solved quite simply, on the other hand, there were problems with setting up automatic shutdown of this (if the user scrolls manually or selects text), as well as with a shadow, user input, etc. Nothing extraordinary, but I had to spend a few hours.
Setting the panel with the output of the script so that it takes up all the available space, but it never went beyond the window. Thus, the page always fits in the window and the only scroll is added inside the output panel. For me it was especially important, because I absolutely do not like scrolls in scrolls.
Page performance with a very large amount of script output. Here with me the following code played a cruel joke:
The fact is that getting innerText is in itself a rather resource-intensive operation that tries to present the incoming html as a string, even if it is just text. If you call this operation a couple of hundred times and for a thousand characters, the effect is not noticeable, but in my case there was much more data and the browser just hung for a few minutes from such a load. I decided to change the code to the following:
Hanging processes i. no one does anything, but the executable script hangs in the processes. The reasons were different, for example: the user closed the page, and the server does not handle this. Or not a closed file descriptor. Or unclosed child processes. And that's not counting other obvious bugs. But eliminated by google, study and correction.
She's not at all. This tool is designed for local launch and work in the local network with trusted users. Since scripts are often also written for themselves, they are also not particularly protected. Accordingly, before starting such a server, you need to be 100% confident in the firewall and users.
From the point of view of data hiding, there is also nothing in the Script server at the current moment. It is not recommended to include passwords as parameters or into the configuration, since they will be logged and stored in a pure form.
For debugging in the project, I started test configurations that can be used for testing and demonstration. In this section, all provided screenshots will be based on them.
A relatively standard bash script with parameters, user input during operation and printing: Write file .
A simple python script that displays a wall of text, breaking it into paragraphs and querying user input: Destroy world . This configuration is used to debug the user interface when displaying parameters. Contains all possible types of parameters: Parameterized
On the left is the script selection area, on the right is information on the current open script:
In this chaos, you can see a family of different types of parameters: mandatory, lists, numbers, booleans. Screenshot created based on Parameterized configuration
Top button to start and stop. Since the script is running, only the last one is active. Below is the output panel of the script, at the very bottom of the input field. The output panel is compressed in height for a demo purpose, in a normal window it is much higher.
At the moment, the development is not conducted, because in its current form, the Script server fully covers the tasks assigned to it. I also see no need to optimize anything in the current interface. Thus, we are both waiting for new challenges and problems.
In particular, I would be very happy if this tool was also useful to someone from the community and I would have a need for new improvements.
»Link to the project repository: github.com/bugy/script-server
List of used libraries / frameworks / thanks:
Thanks to all those who took the time to read this article. I would appreciate valuable comments and criticism (in terms of approach, application or implementation features).
Prehistory
At the new place of work, my first tasks were very routine and monotonous. Schematically, they looked like this:
- Create file my_file.txt
- Register there new configuration
- Apply to server
- Check out
There were more steps and they were more complicated, but after a couple of such iterations, laziness triumphed and shell scripts were written for all repetitive tasks that were run with several parameters and sometimes requested additional input data in the process. At the same time, the workflow began to look something like this:
')
- ask the project manager the necessary parameters
- run the script
- report readiness
All these points were obviously asking for one to collapse so that the project manager would specify the parameters and run the script himself. But giving him scripts was very problematic for a number of reasons - the scripts were adapted to my environment, they could fail, they were friendly enough for me, but not for non-technical people, etc.
It must be said that by a certain moment such tasks had already ceased to be assigned to me and the created scripts had a very illusive future without the necessary adaptation.
Therefore, I decided to create a tool that would allow remotely launching my scripts to other people, without the need to create the necessary environment and in a more convenient form to customize their launch.
On the way of creation
Scheme of work
The server administrator (i) creates configuration files for each script, which describes the purpose of the script, the launch path and the required parameters. These configuration files are used by the server to provide data about the scripts to the user, as well as to run them. Information on the user page is transmitted by Ajax requests.
The user fills in the parameters and runs the script on the server, where its execution is passed to a special handler. The handler accepts input in asynchronous mode and provides output, and also monitors the script execution process.
The web server serves as a layer between this handler and the page, and it communicates through web sockets.
Selection of tools
At that time I was learning Python and decided to use it for development, as a good practice. Python 3 was chosen for the project, without support for version 2, since there is no need for this, and I did not want to waste time.
At the beginning of development for the server, I used Flask, but could not do ( read to understand ) the asynchronous connection and disconnection tracking of clients, so I quickly transferred everything to Tornado.
With regards to web development: there is no web developer out of me, so here I also decided to practice a little in syntax and basics. In this regard, almost no frameworks and libraries were used, with the exception of materializecss for more or less decent design.
A web page is a “one-page application” with a minimum of HTML and content creation in JS by Ajax requests.
First operation and improvements
About a month later I had a server that could somehow be used and I shared it with fellow developers, who also had to perform these routine tasks (which, unfortunately, almost ended in this period).
During operation, both bugs and obviously missing elements were discovered, on which I gradually worked and improved. At the same time, minor improvements were added, such as parameter types, logging of all launches, etc.
An interesting point: for some small tasks (related to working in the console and with scripts), I also stopped using the console, and switched to this UI. Those. I have got several configuration scripts that I use more often than other people.
Real user operation
After some time of local testing (by me and other developers), the tool was finally provided to the project manager, who began to use it little by little. And it should be noted that he is very pleased, because it saves including his time. The process of "implementation" and user training took about 5 minutes.
Due to the peculiarities of the processes and limitations, the Script server is mainly used for the test environment. But some of its parts have already been used for Production (in the part where it is safe).
Main stones
Below is a list of the problems and my personal discoveries that I remembered the most.
Transfer of output from the script execution process to the page in asynchronous mode. This was due to a lack of experience in both Flask and the web: how best to organize such a transfer. As a result, I implemented data transfer in the form of data (type and data) on the websockets. Also in events you can send other commands at runtime, but for now this is not used.
Transfer user input to executable script. I tried to make a request for input only while waiting for one in the script, but could not find a way. Therefore, during the execution of the script, users can enter as much information as they want (the usual terminal behaves the same way). Data is sent via the same web socket.
Track user disconnection. It was precisely because of this problem that Flask refused: at that moment I was doing it on SSE (almost), not on websockets. However, later I did transfer the input / output transfer of the script to the web sockets, so maybe Flask would do. In Tornado, you can just subscribe to the closure of the website.
Read script execution output. There was a problem in order to read only the current available data and send it to the client. It is impossible to read in rows, because for example, “read -p input_prompt” does not transfer to the next line. You can read by symbols, but you should not send so many requests to the client. Reading the buffer gives the string truncated in obscure places (and also in the case of UTF-8, this generates the wrong characters). The current solution is replete with a set of compromises and crutches, but in general it is a read-by-byte reading that disables read locks.
Output scripts in (non) terminal mode. Honestly for me it was a discovery: the output of scripts can be different if you run them in a terminal or out. For example, the same ...
read -p input_prompt ... shows input_prompt only in terminal mode. It is not very good to expect anything from users without showing them exactly what. I had to deal with pty to deceive the scripts being run. For this reason, I now have two types of wrappers for running scripts: with and without terminal support. The first is enabled by default, the second can be enabled using the configuration (left it just in case the terminal mode fails).
Autoscroll script output. On the one hand, this is solved quite simply, on the other hand, there were problems with setting up automatic shutdown of this (if the user scrolls manually or selects text), as well as with a shadow, user input, etc. Nothing extraordinary, but I had to spend a few hours.
Setting the panel with the output of the script so that it takes up all the available space, but it never went beyond the window. Thus, the page always fits in the window and the only scroll is added inside the output panel. For me it was especially important, because I absolutely do not like scrolls in scrolls.
Page performance with a very large amount of script output. Here with me the following code played a cruel joke:
htmlElement.innerText += "text"; The fact is that getting innerText is in itself a rather resource-intensive operation that tries to present the incoming html as a string, even if it is just text. If you call this operation a couple of hundred times and for a thousand characters, the effect is not noticeable, but in my case there was much more data and the browser just hung for a few minutes from such a load. I decided to change the code to the following:
var textNode = document.createTextNode("text"); htmlElement.appendChild(textNode); Hanging processes i. no one does anything, but the executable script hangs in the processes. The reasons were different, for example: the user closed the page, and the server does not handle this. Or not a closed file descriptor. Or unclosed child processes. And that's not counting other obvious bugs. But eliminated by google, study and correction.
Security
She's not at all. This tool is designed for local launch and work in the local network with trusted users. Since scripts are often also written for themselves, they are also not particularly protected. Accordingly, before starting such a server, you need to be 100% confident in the firewall and users.
From the point of view of data hiding, there is also nothing in the Script server at the current moment. It is not recommended to include passwords as parameters or into the configuration, since they will be logged and stored in a pure form.
Examples and screenshots
For debugging in the project, I started test configurations that can be used for testing and demonstration. In this section, all provided screenshots will be based on them.
Script Configuration
A relatively standard bash script with parameters, user input during operation and printing: Write file .
A simple python script that displays a wall of text, breaking it into paragraphs and querying user input: Destroy world . This configuration is used to debug the user interface when displaying parameters. Contains all possible types of parameters: Parameterized
Screenshots
Full Screen
On the left is the script selection area, on the right is information on the current open script:
- Title
- Description
- Parameter list
- Start / Stop Buttons
- Script output
Parameter Bar
In this chaos, you can see a family of different types of parameters: mandatory, lists, numbers, booleans. Screenshot created based on Parameterized configuration
Input / output panel
Top button to start and stop. Since the script is running, only the last one is active. Below is the output panel of the script, at the very bottom of the input field. The output panel is compressed in height for a demo purpose, in a normal window it is much higher.
Further steps
At the moment, the development is not conducted, because in its current form, the Script server fully covers the tasks assigned to it. I also see no need to optimize anything in the current interface. Thus, we are both waiting for new challenges and problems.
In particular, I would be very happy if this tool was also useful to someone from the community and I would have a need for new improvements.
»Link to the project repository: github.com/bugy/script-server
List of used libraries / frameworks / thanks:
- materializecss - materializecss.com
- tornado - www.tornadoweb.org
- Special thanks to my former colleague for the support and help in solving some web problems, especially concerning the layout.
Thanks to all those who took the time to read this article. I would appreciate valuable comments and criticism (in terms of approach, application or implementation features).
Source: https://habr.com/ru/post/309518/
All Articles