Redirecting data from a COM port to the web. Revision
Recently I published an article “Redirecting Data from a COM Port to the Web” , in which I described a prototype of a system translating strings from a serial port of a computer to a web browser. In that article, I indicated the directions in which the prototype should be finalized in order to bring it closer to the production stage:
- no web page design
- at each time point, only one web client will receive data
- A very limited set of browsers through which you can access. For example, it does not work in Internet Explorer 8, nor in the browser from Android 2.3.5
- python installation required
After a while I decided not to leave it in this form and modify it. Under the cut the result of refinement and a description of how I eliminated all the listed disadvantages.
')
Immediately show the final result:
This video shows that the lines from the COM port are displayed simultaneously in three browsers at the same time: in Firefox and IE 8 on the same computer to which the Arduino is connected, and on the smartphone.
The Arduino itself transmits the string “Temperature: XXXXXX”, and the string with the date-time and line number is one of the parts of the backend.
Now, in order, how to eliminate shortcomings. This time I will try not to be very tedious and not to describe every line of code, but to show only the main points. If you have questions, ask in the comments.
In a previous article I wrote that I understand the word “no way” in creating a web front end, so creating a normal web page was the most difficult for me. Fortunately, I almost immediately found the w3schools.com site, where I discovered Bootstrap and found some good Ajax and jQuery tutorials. For experienced front-end developers, the textbooks presented on this site will probably only cause a smile, but for such newcomers as me, this is exactly what is needed.
The most pleasant thing about these textbooks is that they are very small and very substantive. Without smearing porridge on a plate. In principle, one evening to study is enough to start doing something.
It turned out that using Bootstrap to create a more or less acceptable web page design is not so difficult. Of course, he will not make Artemy Lebedev out of you, but you can program the user interface very quickly.
The only thing that I couldn’t understand was how to use it to make the necessary division of a page into two parts: a large one, in which text is displayed in the middle in the vertical plane, and a small one, which is “pressed” to the bottom edge of the browser window all the time. But here came the article "Vertical Centering in CSS" . As a result, the following web page was obtained:
Practically all the rest of the page code turned out after reading Bootstrap and JavaScript tutorials (section JS HTML DOM). The exception is the jQuery code that updates the data on the page. But more on that later.
Weak browser support for the previous prototype was due to the chosen technology for delivering information updates: Server-Sent Events. Therefore, this time I decided to use the old time-tested technology Ajax. Using Ajax leads to an increase in web traffic, but, in my opinion, it should work in the maximum number of browsers.
Another disadvantage of Ajax is the fact that, if no special measures are taken, there may be gaps in the lines that are transmitted through the COM port: if the lines to the serial port are received very quickly, and Ajax requests come less often, all lines between requests will not be visible to the client. But for the task of displaying, for example, the current temperature - this is not at all scary.
I could probably use the WebSockets technology, but as far as I understood, in IE it is supported only from version 10, and I have IE 8, so I didn’t even work out this direction.
Just a few days ago in some of the articles on Habré or HykTimes I came across the mention of the SockJS library, which seems to be able to bypass the absence of WebSockets, but, first, it requires special. support on the server side, and, secondly, Ajax was working for me by this moment, so it was left without my attention.
So, Ajax. Once upon a time I tried to learn this technology. But all the paper books that I came across were too tedious and I very quickly gave up. But at the already mentioned w3schools.com tutorial turned out to be very good. As a result, the following code came out pretty quickly:
which was called at the end of the page loading:
In this code, you probably need to pay attention to two points. First, on the line
This is where the web server is accessed for the next line from the COM port. Additive
needed to ensure that Internet Explorer does not cache the answers. Otherwise, he will send only one Ajax request, will receive a response and will no longer contact the server. On the Internet, I have seen solutions for caching problems by sending special HTTP headers from the server side
but for some reason it didn't work for me.
An alternative is to use POST instead of GET. They say that IE does not cache such requests, but I did not check.
By the way, jQuery, uses the exact same way of dealing with caching - it also adds a random string to the URL. It only looks different.
The second point is a sequence of lines.
and
Initially, they were in a different order. And in Internet Explorer, this led to strange problems - he was surviving all the available memory. And only after that updated data. What and finished his work.
Only on the site XmlHttpRequest.ru I found that when reusing an XMLHttpRequest object, it is recommended to first call the open () method and only after that change the onreadystatechange property.
Having received so many problems with IE out of the blue, I decided that I should use the library, in which such nuances are most likely already taken into account. Since for Bootstrap, jQuery was already loaded on the page.
I decided that it was worth not to use the Ajax support built into this library. Therefore, the data update code has been converted to:
In the previous article I already wrote about a possible way to solve this problem: splitting the backend into two parts, using ZMQ for their connection + multiplayer http-server as the second part. As such a server, I chose Flask. I did not make any comparisons of alternatives, he just caught me first. It turned out that it can be run in the mode of parallel processing of http-requests and this turned out to be sufficient. To start this mode, it is enough to pass a parameter to it.
threaded = True (see stackoverflow.com/questions/14672753/handling-multiple-requests-in-flask ):
In addition to multi-threading, Flask provides another very convenient routing mechanism, i.e. compliance of individual procedures with certain http requests. So for creating simple web applications, Flask is a very nice thing.
But most of all I would like to tell you about the ZMQ library - to demonstrate its amazing capabilities.
Here is the code for the demo ZMQ server:
which simulates reading from a COM port. In fact, it generates a new line every second, “publishes” it (that is, sends it to all subscribing clients) and prints it for debugging.
And water client code:
which opens a ZMQ socket, subscribes to all possible messages and connects to the server. After that, in an infinite loop, it waits for new data and displays it.
But a demonstration of their collaboration:
What I want to draw attention to. First, you can start the client before starting the server. And it does not affect its performance. Secondly, even a server crash will not cause clients to crash - as soon as the server restarts, clients will again receive messages from it. Isn't that fantastic? And this is in the absence of any special instructions in the source code of the client and server. How much code would you have to write yourself to implement this functionality when using conventional TCP / IP sockets?
And this is just a little bit of what the ZMQ library can do. Once again, I highly recommend that you look at this library.
I already, it seems, mentioned that python is poorly adapted for creating standalone applications. It is an interpreted language for which normal work requires an interpreter program. Unfortunately, there is no compiler for it that can generate the native binary code. I know two programs that allow you to create a kind of self-contained application from a script: py2exe and pyInstaller. I myself often use the second one, so I decided to use it in this project.
pyInstaller takes the Python script, analyzes all the files on which it depends (determines all imported modules and dynamic link libraries) and then either collects them all into a separate folder, or packs them into some kind of analog self-extracting archive. The code of the interpreter itself is also added there, as well as a small starting part, which, at the start, prepares the execution environment and starts the interpretation of the target script.
Since neither the creators of python, nor the creators of libraries under it did not think about such a possible use, it all works somewhat “through the stump-deck”. The main problems are to prepare a list of all necessary libraries and modules. Because many authors of libraries use implicit import, with which there is no explicit indication of which module will be imported into a script or another module. Accordingly, often after automatic analysis of the pyInstaller script, you need to manually add the names of additional modules and / or paths in the configuration file (with the .spec extension) where to find them. Together with pyInstaller there is a large set of ready-made functions for such libraries, but life does not stand still. In particular, this turned out to be the case with the current version of ZMQ - the developers of pyZMQ (ZMQ binding to python) have changed the mechanism for importing auxiliary libraries, as a result of which the application built by pyInstaller does not start. The people have already figured out this case and have prepared a corresponding patch for pyInstaller. The patch is working, but I haven’t yet got into the official release, so I had to patch pyInstaller with my hands. See the patch code at github.com/pyinstaller/pyinstaller/pull/110/files
But even this patch was not enough if the standalone application should be just one executable file, and not a whole folder. I had to manually adjust the .spec file so that the libsodium.pyd library from the ZMQ distribution kit got to the right place when unpacking the .exe file.
The second problem was the multiprocessing module. I divided the backend into two parts, which should work in parallel. It seemed to me wrong if I had to run two separate programs to run the backend. And to use multithreading (multithreading) it seemed to me wrong because of the presence of GIL (Global Interpretation Lock). It would seem that it is easier to write a simple script that, when started, will simply generate two new processes: one for the ZMQ server that reads data from the COM port, the second for the HTTP server that responds to web client requests. And indeed, when working in normal mode (i.e., when “manually” running the interpreter), the following script works fine:
But after processing pyInstaller, the resulting .exe did not work normally. Several ZMQ servers and Flask applications were launched with the appropriate messages that the “socket is already busy” and some other oddities. It turned out that when launching a Flask application, it needs to pass the debug = False parameter:
and for the multiprocessing module, you need to call the special function freeze_support (), which is needed in that mode (froozen), which is created when the script is interpreted in the case of standalone applications created by pyInstaller.
In general, the result of this point is this: you can create a standalone application from the Python script, but this is not easy.
All source code can be downloaded from github: github.com/alguryanow/serial2web-2
PS Another Bootstrap Tutorial : www.tutorialrepublic.com/twitter-bootstrap-2.3.2-tutorial
- no web page design
- at each time point, only one web client will receive data
- A very limited set of browsers through which you can access. For example, it does not work in Internet Explorer 8, nor in the browser from Android 2.3.5
- python installation required
After a while I decided not to leave it in this form and modify it. Under the cut the result of refinement and a description of how I eliminated all the listed disadvantages.
')
Immediately show the final result:
This video shows that the lines from the COM port are displayed simultaneously in three browsers at the same time: in Firefox and IE 8 on the same computer to which the Arduino is connected, and on the smartphone.
The Arduino itself transmits the string “Temperature: XXXXXX”, and the string with the date-time and line number is one of the parts of the backend.
Now, in order, how to eliminate shortcomings. This time I will try not to be very tedious and not to describe every line of code, but to show only the main points. If you have questions, ask in the comments.
Bad web page design
In a previous article I wrote that I understand the word “no way” in creating a web front end, so creating a normal web page was the most difficult for me. Fortunately, I almost immediately found the w3schools.com site, where I discovered Bootstrap and found some good Ajax and jQuery tutorials. For experienced front-end developers, the textbooks presented on this site will probably only cause a smile, but for such newcomers as me, this is exactly what is needed.
The most pleasant thing about these textbooks is that they are very small and very substantive. Without smearing porridge on a plate. In principle, one evening to study is enough to start doing something.
It turned out that using Bootstrap to create a more or less acceptable web page design is not so difficult. Of course, he will not make Artemy Lebedev out of you, but you can program the user interface very quickly.
The only thing that I couldn’t understand was how to use it to make the necessary division of a page into two parts: a large one, in which text is displayed in the middle in the vertical plane, and a small one, which is “pressed” to the bottom edge of the browser window all the time. But here came the article "Vertical Centering in CSS" . As a result, the following web page was obtained:
<!-- Vertical aligment of text from "Vertical Centering in CSS" at http://www.jakpsatweb.cz/css/css-vertical-center-solution.html --> <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"> <html> <head> <style type="text/css"> html, body { height: 100%; margin: 0px; } </style> </head> <body> <div style="display: table; height: 90%; width: 100%; overflow: hidden;"> <div style=" display: table-cell; vertical-align: middle;"> <div style="text-align: center"> data place<br> </div> </div> </div> <div style="display: table; height: 10%; width: 100%; overflow: hidden;"> <div style=" display: table-cell; vertical-align: middle;"> <div style="text-align: right;"> buttons </div> </div> </div> </body> </html> Practically all the rest of the page code turned out after reading Bootstrap and JavaScript tutorials (section JS HTML DOM). The exception is the jQuery code that updates the data on the page. But more on that later.
Limited number of supported browsers
Weak browser support for the previous prototype was due to the chosen technology for delivering information updates: Server-Sent Events. Therefore, this time I decided to use the old time-tested technology Ajax. Using Ajax leads to an increase in web traffic, but, in my opinion, it should work in the maximum number of browsers.
Another disadvantage of Ajax is the fact that, if no special measures are taken, there may be gaps in the lines that are transmitted through the COM port: if the lines to the serial port are received very quickly, and Ajax requests come less often, all lines between requests will not be visible to the client. But for the task of displaying, for example, the current temperature - this is not at all scary.
I could probably use the WebSockets technology, but as far as I understood, in IE it is supported only from version 10, and I have IE 8, so I didn’t even work out this direction.
Just a few days ago in some of the articles on Habré or HykTimes I came across the mention of the SockJS library, which seems to be able to bypass the absence of WebSockets, but, first, it requires special. support on the server side, and, secondly, Ajax was working for me by this moment, so it was left without my attention.
So, Ajax. Once upon a time I tried to learn this technology. But all the paper books that I came across were too tedious and I very quickly gave up. But at the already mentioned w3schools.com tutorial turned out to be very good. As a result, the following code came out pretty quickly:
function get_data() { var xmlhttp; xmlhttp=new XMLHttpRequest(); xmlhttp.open("GET","/get_serial?r=" + Math.random(),true); xmlhttp.onreadystatechange=function() { if (xmlhttp.readyState==4 && xmlhttp.status==200) { document.getElementById("data").innerHTML=xmlhttp.responseText; get_data(); } } xmlhttp.send(); } which was called at the end of the page loading:
<body onload="get_data()"> In this code, you probably need to pay attention to two points. First, on the line
xmlhttp.open("GET","/get_serial?r=" + Math.random(),true); This is where the web server is accessed for the next line from the COM port. Additive
r=" + Math.random() needed to ensure that Internet Explorer does not cache the answers. Otherwise, he will send only one Ajax request, will receive a response and will no longer contact the server. On the Internet, I have seen solutions for caching problems by sending special HTTP headers from the server side
response.headers['Last-Modified'] = datetime.now() response.headers['Cache-Control'] = 'no-store, no-cache, must-revalidate, post-check=0, pre-check=0, max-age=0' response.headers['Pragma'] = 'no-cache' response.headers['Expires'] = '-1' but for some reason it didn't work for me.
An alternative is to use POST instead of GET. They say that IE does not cache such requests, but I did not check.
By the way, jQuery, uses the exact same way of dealing with caching - it also adds a random string to the URL. It only looks different.
The second point is a sequence of lines.
xmlhttp.open("GET","/get_serial?r=" + Math.random(),true); and
xmlhttp.onreadystatechange=function() ... Initially, they were in a different order. And in Internet Explorer, this led to strange problems - he was surviving all the available memory. And only after that updated data. What and finished his work.
Only on the site XmlHttpRequest.ru I found that when reusing an XMLHttpRequest object, it is recommended to first call the open () method and only after that change the onreadystatechange property.
Having received so many problems with IE out of the blue, I decided that I should use the library, in which such nuances are most likely already taken into account. Since for Bootstrap, jQuery was already loaded on the page.
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.3/jquery.min.js"></script> I decided that it was worth not to use the Ajax support built into this library. Therefore, the data update code has been converted to:
function on_load(responseTxt, statusTxt, xhr){ if( statusTxt == "success" ){ $("#data").load("/get_serial?r=" + Math.random(), on_load ) } else { $("#data").text( "Error: " + xhr.status + ": " + xhr.statusText ); } } $(document).ready(function(){ on_load("", "success", 3); }); One customer at a time
In the previous article I already wrote about a possible way to solve this problem: splitting the backend into two parts, using ZMQ for their connection + multiplayer http-server as the second part. As such a server, I chose Flask. I did not make any comparisons of alternatives, he just caught me first. It turned out that it can be run in the mode of parallel processing of http-requests and this turned out to be sufficient. To start this mode, it is enough to pass a parameter to it.
threaded = True (see stackoverflow.com/questions/14672753/handling-multiple-requests-in-flask ):
app.run(host='0.0.0.0', debug=False, threaded=True) In addition to multi-threading, Flask provides another very convenient routing mechanism, i.e. compliance of individual procedures with certain http requests. So for creating simple web applications, Flask is a very nice thing.
But most of all I would like to tell you about the ZMQ library - to demonstrate its amazing capabilities.
Here is the code for the demo ZMQ server:
import zmq import time context = zmq.Context.instance() pub_sock = context.socket(zmq.PUB) pub_sock.bind( 'tcp://*:12345' ) count = 0 def get_full_line_from_serial(): global count count += 1 return time.strftime( '%Y.%m.%d %H:%M:%S' ) + ': string N %d' % count while True: line = get_full_line_from_serial() print line pub_sock.send( line ) time.sleep(1) which simulates reading from a COM port. In fact, it generates a new line every second, “publishes” it (that is, sends it to all subscribing clients) and prints it for debugging.
And water client code:
import zmq context = zmq.Context.instance() zmq_sub_sock = context.socket(zmq.SUB) zmq_sub_sock.setsockopt(zmq.SUBSCRIBE, '') zmq_sub_sock.connect( 'tcp://localhost:12345' ) poller = zmq.Poller() poller.register( zmq_sub_sock, zmq.POLLIN ) while True: socks = dict(poller.poll(timeout=5000)) if zmq_sub_sock in socks: print zmq_sub_sock.recv() else: print 'No data within 5 sec' which opens a ZMQ socket, subscribes to all possible messages and connects to the server. After that, in an infinite loop, it waits for new data and displays it.
But a demonstration of their collaboration:
What I want to draw attention to. First, you can start the client before starting the server. And it does not affect its performance. Secondly, even a server crash will not cause clients to crash - as soon as the server restarts, clients will again receive messages from it. Isn't that fantastic? And this is in the absence of any special instructions in the source code of the client and server. How much code would you have to write yourself to implement this functionality when using conventional TCP / IP sockets?
And this is just a little bit of what the ZMQ library can do. Once again, I highly recommend that you look at this library.
Standalone app
I already, it seems, mentioned that python is poorly adapted for creating standalone applications. It is an interpreted language for which normal work requires an interpreter program. Unfortunately, there is no compiler for it that can generate the native binary code. I know two programs that allow you to create a kind of self-contained application from a script: py2exe and pyInstaller. I myself often use the second one, so I decided to use it in this project.
pyInstaller takes the Python script, analyzes all the files on which it depends (determines all imported modules and dynamic link libraries) and then either collects them all into a separate folder, or packs them into some kind of analog self-extracting archive. The code of the interpreter itself is also added there, as well as a small starting part, which, at the start, prepares the execution environment and starts the interpretation of the target script.
Since neither the creators of python, nor the creators of libraries under it did not think about such a possible use, it all works somewhat “through the stump-deck”. The main problems are to prepare a list of all necessary libraries and modules. Because many authors of libraries use implicit import, with which there is no explicit indication of which module will be imported into a script or another module. Accordingly, often after automatic analysis of the pyInstaller script, you need to manually add the names of additional modules and / or paths in the configuration file (with the .spec extension) where to find them. Together with pyInstaller there is a large set of ready-made functions for such libraries, but life does not stand still. In particular, this turned out to be the case with the current version of ZMQ - the developers of pyZMQ (ZMQ binding to python) have changed the mechanism for importing auxiliary libraries, as a result of which the application built by pyInstaller does not start. The people have already figured out this case and have prepared a corresponding patch for pyInstaller. The patch is working, but I haven’t yet got into the official release, so I had to patch pyInstaller with my hands. See the patch code at github.com/pyinstaller/pyinstaller/pull/110/files
But even this patch was not enough if the standalone application should be just one executable file, and not a whole folder. I had to manually adjust the .spec file so that the libsodium.pyd library from the ZMQ distribution kit got to the right place when unpacking the .exe file.
The second problem was the multiprocessing module. I divided the backend into two parts, which should work in parallel. It seemed to me wrong if I had to run two separate programs to run the backend. And to use multithreading (multithreading) it seemed to me wrong because of the presence of GIL (Global Interpretation Lock). It would seem that it is easier to write a simple script that, when started, will simply generate two new processes: one for the ZMQ server that reads data from the COM port, the second for the HTTP server that responds to web client requests. And indeed, when working in normal mode (i.e., when “manually” running the interpreter), the following script works fine:
# -*- coding: utf-8 -*- from multiprocessing import Process import serial_to_zmq import zmq2web_using_flask import time def main(): args = get_command_line_params() p1 = Process( target=serial_to_zmq.work, args=(args.serial_port_name, args.serial_port_speed, args.zmq_pub_addr) ) p1.start() p2 = Process(target=zmq2web_using_flask.work, args=(args.zmq_sub_addr,)) p2.start() print 'Press Ctrl+C to stop...', while True: time.sleep(10) def get_command_line_params(): ... if __name__ == '__main__': main() But after processing pyInstaller, the resulting .exe did not work normally. Several ZMQ servers and Flask applications were launched with the appropriate messages that the “socket is already busy” and some other oddities. It turned out that when launching a Flask application, it needs to pass the debug = False parameter:
app.run(host='0.0.0.0', debug=False, threaded=True) and for the multiprocessing module, you need to call the special function freeze_support (), which is needed in that mode (froozen), which is created when the script is interpreted in the case of standalone applications created by pyInstaller.
In general, the result of this point is this: you can create a standalone application from the Python script, but this is not easy.
All source code can be downloaded from github: github.com/alguryanow/serial2web-2
PS Another Bootstrap Tutorial : www.tutorialrepublic.com/twitter-bootstrap-2.3.2-tutorial
Source: https://habr.com/ru/post/266317/
All Articles