Redirecting data from the COM port to the Web
Recently on Habré there was an article "Displaying data from Serial in Chrome Application" about how beautifully to present the data sent by Arduin-koy to Serial. In my opinion, the guys offered a very beautiful solution, which on the one hand looks quite simple, and on the other allows you to get a great result with a minimum of effort.
In the comments to the article, it was regretted that such a solution would not work under Firefox, and the idea was expressed that "you can still write a simple web server with html output based on this thing." This idea “hooked” me, I didn’t give out a quick search in google, and I decided to implement the idea myself. And that's what came of it.
A warning! In no case can the proposed solution be regarded as complete. Unlike Amperka’s Serial Projector, this is a concept, a demonstration of a possible approach, a working prototype and nothing more.
Some time ago I did a project in which I used accelerometers built into the Android-smartphone to control the servers connected to the Arduino. Then for these purposes I used the Scripting Layer for Android (SL4A) and RemoteSensors projects. It turns out that the standard library python includes the BaseHTTPServer package, with which it is possible to raise a web service in python - this is a task for a couple of lines of code.
')
There were no sensors for the Arduino at hand, so I used the internal thermometer built into the Arduino Uno as the source of the displayed information. As far as I understand, it is not very accurate and is not intended to measure the ambient temperature at all, but it will completely come down for prototyping.
After a brief googling, this sketch for arduinka appeared:
This sketch opens the COM port, adjusts it to the speed of 115200 baud, and then every second writes the current value of the built-in thermometer to it. (Do not ask me in what units the temperature is given - for the described task it is not important). Since the value does not change very actively, a line number is displayed before the temperature for better visibility of data changes.
To check that the web server will only send out whole lines, not parts of them, as they are read from the COM port,
was replaced by
those. the generated string is not output to the serial port entirely, but character by character, with pauses of 200 ms.
To begin with, a very simple prototype of a web server was written (it is analyzed below in parts):
Let's sort the script in parts.
Since this is a prototype, all the basic operation parameters (the name of the COM port, its speed, and the number of the TCP port on which the web server will work) are indicated directly in the source text:
Of course, you can organize the reading of these parameters from the command line. For example, using the argparse module, this is done very quickly, simply and flexibly.
In this case, Windows users need to find out the name of the COM port to which Arduin is connected in Device Manager. I had it 'COM6'. Users of other OSes should use the tools of their OS. I have absolutely no experience with MacOS and in Linux I didn’t work with COM ports either, but most likely it will be something like "/ dev / ttySn".
Next comes the definition of a global variable to which an instance of the Serial class that is responsible for working with the COM port in python will be associated:
In line
a web server is created that will listen for requests on the specified WEB_SERVER_PORT port. And an instance of the Handler class, described below, will process these requests.
The following lines are a small “hack” that allows you to display the IP address on which the running web server actually runs:
As I understand it, there is no other way to find out this IP. And how, without this knowledge, we will access our server from a browser?
Therefore, you have to open a socket and connect to the Google site in order to extract information about your own IP address from the attributes of this socket.
Just below the opening of the COM port and the actual launch of the web server:
Then follows the description of the class that is responsible for processing the requests received by the launched web server:
This is a descendant of the class built into the BaseHTTPServer module, in which it is enough to override only the do_GET method
Since this is still a prototype, the server will be “happy” to any request - whatever URL it is requested from, it will give the client all the data read from the COM port. Therefore, in Handler.do_GET, it immediately responds with a success code and the necessary headers:
after which an infinite loop starts, in which an attempt is made to read a whole line from the COM port and, if this attempt was successful, transfer it to the web client:
In the project, which was taken as a basis, this infinite cycle was wrapped in a try… except block, with which it was supposed to carefully handle the disconnection of the connection. Perhaps, in Android (the base project was developed for it), this works fine, but it didn’t work for me under Windows XP - when I disconnected, there was some other exception that I didn’t learn to intercept. But, fortunately, this did not prevent the web server from operating normally and accepting the following requests.
The function of receiving a whole string from a COM port works on the same principle as the creators of the Serial Projector:
The result is:
It can be seen that the lines read from the COM port appear in the browser. I do not understand anything in the web front end: JavaScript, Ajax, CSS and DOM are for me a dark forest. But it seems to me that for programmers creating web interfaces this should be quite enough to convert this output into the same beautiful picture that Amperka’s Serial Projector produces. In my opinion, the task is to create a javascript script that accesses the web server, reads the stream from it and displays the last read line in the right place on the web page.
Just in case, I decided to play it safe and tried to make the first approach on my own. Not very deep search in Google prompted that generally for such purposes, at least, used WebSockets or Server-Sent Events earlier. I found, it seemed to me, a good tutorial on using Server-Sent Events and decided to use this technology.
Note! It seems that this is not the best solution, because this technology did not work either in Internet Explorer 8, or in the browser built into Android 2.3.5. But it worked at least in Firefox 39.0, so I did not “dig” further.
After reading this textbook, as well as another one in Russian , I took the simpl.info/eventsource project as a basis .
From the point of view of the Python script, the changes for Server-Sent Events are quite minor:
Everything else could probably remain unchanged, but ...
First, I created an index.html file with this content:
The most interesting in it is a string
which forms a place to output the next line from the COM port, and a javascript script
which is actually engaged in reading the stream from the web server and outputting the read information to the specified location.
I assumed to open this file in a browser, for example, from a disk or from some other web server, but this did not work: when opening a page from disk, the javascript script once turned to the running Python web server and immediately disconnected. I did not understand why this was happening, and suggested that this might be some kind of browser protection against various attacks. He probably does not like the fact that the page itself is open from one source, and the script reads data from another source.
Therefore, it was decided to change the Python web server so that it gives this html page. Then it would happen that both the page and the stream are read from the same source. I do not know whether my assumption about security turned out to be true, or something else, but with this implementation everything worked as it should.
Of course, you only need to change the Handler class request handler:
In this variant, it is assumed that the web server will respond only to two requests: '/index.html' (giving the page's html-code) and '/ get_serial' (giving up an endless stream of lines read from the COM port). It will respond to all other requests with the code 404.
Since index.html is given to the Python web server, it can be slightly changed by specifying a relative address instead of the absolute address of the stream of lines from the COM port:
the string
replaced by
The result was this:
On this I decided to stop. It seems to me that it is beautiful to design a page - it should already be quite simple. But I do not own either HTML or CSS, so let someone else do it. I saw my task in showing that making a web service that provides data from a COM port is not at all difficult.
All sources can be taken on githabe .
I repeat: the presented code is not a complete solution that can be “let in production”. This is only a prototype that shows a fundamental approach to solving a problem.
What else can you work on:
In conclusion: the question may arise: “is it not easier to solve this problem through some ready-made cloud?”. Why not publish the data read from the COM port in the cloud, and on the clients simply access the corresponding service in the cloud? Probably, such a decision also has the right to exist, but before applying such a decision, you must answer the following questions:
In the comments to the article, it was regretted that such a solution would not work under Firefox, and the idea was expressed that "you can still write a simple web server with html output based on this thing." This idea “hooked” me, I didn’t give out a quick search in google, and I decided to implement the idea myself. And that's what came of it.
A warning! In no case can the proposed solution be regarded as complete. Unlike Amperka’s Serial Projector, this is a concept, a demonstration of a possible approach, a working prototype and nothing more.
Some time ago I did a project in which I used accelerometers built into the Android-smartphone to control the servers connected to the Arduino. Then for these purposes I used the Scripting Layer for Android (SL4A) and RemoteSensors projects. It turns out that the standard library python includes the BaseHTTPServer package, with which it is possible to raise a web service in python - this is a task for a couple of lines of code.
')
There were no sensors for the Arduino at hand, so I used the internal thermometer built into the Arduino Uno as the source of the displayed information. As far as I understand, it is not very accurate and is not intended to measure the ambient temperature at all, but it will completely come down for prototyping.
After a brief googling, this sketch for arduinka appeared:
// source: https://code.google.com/p/tinkerit/wiki/SecretThermometer long readTemp() { long result; // Read temperature sensor against 1.1V reference ADMUX = _BV(REFS1) | _BV(REFS0) | _BV(MUX3); delay(2); // Wait for Vref to settle ADCSRA |= _BV(ADSC); // Convert while (bit_is_set(ADCSRA,ADSC)); result = ADCL; result |= ADCH<<8; result = (result - 125) * 1075; return result; } void setup() { Serial.begin(115200); } int count = 0; void loop() { String s = String(count++, DEC) + ": " + String( readTemp(), DEC ); Serial.println(s) delay(1000); } This sketch opens the COM port, adjusts it to the speed of 115200 baud, and then every second writes the current value of the built-in thermometer to it. (Do not ask me in what units the temperature is given - for the described task it is not important). Since the value does not change very actively, a line number is displayed before the temperature for better visibility of data changes.
To check that the web server will only send out whole lines, not parts of them, as they are read from the COM port,
Serial.println(s) was replaced by
for(int i=0; i < s.length(); i++ ){ Serial.print( s.charAt(i) ); delay( 200 ); } Serial.println(""); those. the generated string is not output to the serial port entirely, but character by character, with pauses of 200 ms.
To begin with, a very simple prototype of a web server was written (it is analyzed below in parts):
# -*- coding: utf-8 -*- #-- based on: https://raw.githubusercontent.com/Jonty/RemoteSensors/master/remoteSensors.py SERIAL_PORT_NAME = 'COM6' SERIAL_PORT_SPEED = 115200 WEB_SERVER_PORT = 8000 import time, BaseHTTPServer, urlparse import serial ser = None def main(): global ser httpd = BaseHTTPServer.HTTPServer(("", WEB_SERVER_PORT), Handler) #-- workaround for getting IP address at which serving import socket s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) s.connect(('google.co.uk', 80)) sData = s.getsockname() print "Serving at '%s:%s'" % (sData[0], WEB_SERVER_PORT) ser = serial.Serial(SERIAL_PORT_NAME, SERIAL_PORT_SPEED, timeout=0) httpd.serve_forever() class Handler(BaseHTTPServer.BaseHTTPRequestHandler): # Disable logging DNS lookups def address_string(self): return str(self.client_address[0]) def do_GET(self): self.send_response(200) self.send_header("Content-type", "application/x-javascript; charset=utf-8") self.end_headers() try: while True: new_serial_line = get_full_line_from_serial() if new_serial_line is not None: self.wfile.write(new_serial_line) self.wfile.write("\n") self.wfile.flush() except socket.error, e: print "Client disconnected.\n" captured = '' def get_full_line_from_serial(): """ returns full line from serial or None Uses global variables 'ser' and 'captured' """ global captured part = ser.readline() if part: captured += part parts = captured.split('\n', 1); if len(parts) == 2: captured = parts[1] return parts[0] return None if __name__ == '__main__': main() Let's sort the script in parts.
Since this is a prototype, all the basic operation parameters (the name of the COM port, its speed, and the number of the TCP port on which the web server will work) are indicated directly in the source text:
SERIAL_PORT_NAME = 'COM6' SERIAL_PORT_SPEED = 115200 WEB_SERVER_PORT = 8000 Of course, you can organize the reading of these parameters from the command line. For example, using the argparse module, this is done very quickly, simply and flexibly.
In this case, Windows users need to find out the name of the COM port to which Arduin is connected in Device Manager. I had it 'COM6'. Users of other OSes should use the tools of their OS. I have absolutely no experience with MacOS and in Linux I didn’t work with COM ports either, but most likely it will be something like "/ dev / ttySn".
Next comes the definition of a global variable to which an instance of the Serial class that is responsible for working with the COM port in python will be associated:
ser = None In line
httpd = BaseHTTPServer.HTTPServer(("", WEB_SERVER_PORT), Handler) a web server is created that will listen for requests on the specified WEB_SERVER_PORT port. And an instance of the Handler class, described below, will process these requests.
The following lines are a small “hack” that allows you to display the IP address on which the running web server actually runs:
#-- workaround for getting IP address at which serving import socket s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) s.connect(('google.co.uk', 80)) sData = s.getsockname() print "Serving at '%s:%s'" % (sData[0], WEB_SERVER_PORT) As I understand it, there is no other way to find out this IP. And how, without this knowledge, we will access our server from a browser?
Therefore, you have to open a socket and connect to the Google site in order to extract information about your own IP address from the attributes of this socket.
Just below the opening of the COM port and the actual launch of the web server:
ser = serial.Serial(SERIAL_PORT_NAME, SERIAL_PORT_SPEED, timeout=0) httpd.serve_forever() Then follows the description of the class that is responsible for processing the requests received by the launched web server:
class Handler(BaseHTTPServer.BaseHTTPRequestHandler): This is a descendant of the class built into the BaseHTTPServer module, in which it is enough to override only the do_GET method
Since this is still a prototype, the server will be “happy” to any request - whatever URL it is requested from, it will give the client all the data read from the COM port. Therefore, in Handler.do_GET, it immediately responds with a success code and the necessary headers:
self.send_response(200) self.send_header("Content-type", "application/x-javascript; charset=utf-8") self.end_headers() after which an infinite loop starts, in which an attempt is made to read a whole line from the COM port and, if this attempt was successful, transfer it to the web client:
while True: new_serial_line = get_full_line_from_serial() if new_serial_line is not None: self.wfile.write(new_serial_line) self.wfile.write("\n") self.wfile.flush() In the project, which was taken as a basis, this infinite cycle was wrapped in a try… except block, with which it was supposed to carefully handle the disconnection of the connection. Perhaps, in Android (the base project was developed for it), this works fine, but it didn’t work for me under Windows XP - when I disconnected, there was some other exception that I didn’t learn to intercept. But, fortunately, this did not prevent the web server from operating normally and accepting the following requests.
The function of receiving a whole string from a COM port works on the same principle as the creators of the Serial Projector:
- there is some global buffer in which everything that is read from the COM port is stored
- each time the function is accessed, it tries to read something from the COM port.
- if she succeeds, then
- it adds just read to the specified global buffer
- tries to divide the global buffer into at most two parts with a line terminator
- if it succeeds in this, then it returns the first part to the calling procedure, and uses the second part as the new value of the global buffer
- if there is no new data in the COM port or the end of line character is not found, the function returns None:
captured = '' def get_full_line_from_serial(): """ returns full line from serial or None Uses global variables 'ser' and 'captured' """ global captured part = ser.readline() if part: captured += part parts = captured.split('\n', 1); if len(parts) == 2: captured = parts[1] return parts[0] return None The result is:
It can be seen that the lines read from the COM port appear in the browser. I do not understand anything in the web front end: JavaScript, Ajax, CSS and DOM are for me a dark forest. But it seems to me that for programmers creating web interfaces this should be quite enough to convert this output into the same beautiful picture that Amperka’s Serial Projector produces. In my opinion, the task is to create a javascript script that accesses the web server, reads the stream from it and displays the last read line in the right place on the web page.
Just in case, I decided to play it safe and tried to make the first approach on my own. Not very deep search in Google prompted that generally for such purposes, at least, used WebSockets or Server-Sent Events earlier. I found, it seemed to me, a good tutorial on using Server-Sent Events and decided to use this technology.
Note! It seems that this is not the best solution, because this technology did not work either in Internet Explorer 8, or in the browser built into Android 2.3.5. But it worked at least in Firefox 39.0, so I did not “dig” further.
After reading this textbook, as well as another one in Russian , I took the simpl.info/eventsource project as a basis .
From the point of view of the Python script, the changes for Server-Sent Events are quite minor:
- It is necessary to replace the type of data given to the client:
lineself.send_header("Content-type", "application/x-javascript; charset=utf-8")
replaced byself.send_header("Content-type", "text/event-stream") - as well as before the line read from the COM port, insert the “data:” prefix and add another newline character:
linesself.wfile.write(new_serial_line) self.wfile.write("\n")
replaced byself.wfile.write('data: ' + new_serial_line) self.wfile.write("\n\n")
Everything else could probably remain unchanged, but ...
First, I created an index.html file with this content:
<!DOCTYPE html> <html> <head> <meta charset="utf-8" /> </head> <body> <h1></h1> <p id="data"></p> <script> var dataDiv = document.querySelector('#data'); var source = new EventSource('http://192.168.1.207:8000/') source.onmessage = function(e) { dataDiv.innerHTML = e.data; }; </script> </body> </html> The most interesting in it is a string
<p id="data"></p> which forms a place to output the next line from the COM port, and a javascript script
<script> var dataDiv = document.querySelector('#data'); var source = new EventSource('http://192.168.1.207:8000/') source.onmessage = function(e) { dataDiv.innerHTML = e.data; }; </script> which is actually engaged in reading the stream from the web server and outputting the read information to the specified location.
I assumed to open this file in a browser, for example, from a disk or from some other web server, but this did not work: when opening a page from disk, the javascript script once turned to the running Python web server and immediately disconnected. I did not understand why this was happening, and suggested that this might be some kind of browser protection against various attacks. He probably does not like the fact that the page itself is open from one source, and the script reads data from another source.
Therefore, it was decided to change the Python web server so that it gives this html page. Then it would happen that both the page and the stream are read from the same source. I do not know whether my assumption about security turned out to be true, or something else, but with this implementation everything worked as it should.
Of course, you only need to change the Handler class request handler:
class Handler(BaseHTTPServer.BaseHTTPRequestHandler): # Disable logging DNS lookups def address_string(self): return str(self.client_address[0]) def do_GET(self): if self.path == '/' or self.path == '/index.html': self.process_index() elif self.path == '/get_serial': self.process_get_serial() else: self.process_unknown() def process_index(self): self.send_response(200) self.send_header("Content-type", "text/html; charset=utf-8") self.end_headers() self.wfile.write(open('index.html').read()) self.wfile.write("\n\n") self.wfile.flush() def process_get_serial(self): self.send_response(200) self.send_header("Content-type", "text/event-stream") self.end_headers() try: while True: new_serial_line = get_full_line_from_serial() if new_serial_line is not None: self.wfile.write('data: ' + new_serial_line) self.wfile.write("\n\n") self.wfile.flush() except socket.error, e: print "Client disconnected.\n" def process_unknown(self): self.send_response(404) In this variant, it is assumed that the web server will respond only to two requests: '/index.html' (giving the page's html-code) and '/ get_serial' (giving up an endless stream of lines read from the COM port). It will respond to all other requests with the code 404.
Since index.html is given to the Python web server, it can be slightly changed by specifying a relative address instead of the absolute address of the stream of lines from the COM port:
the string
var source = new EventSource('http://192.168.1.207:8000/') replaced by
var source = new EventSource('/get_serial') The result was this:
On this I decided to stop. It seems to me that it is beautiful to design a page - it should already be quite simple. But I do not own either HTML or CSS, so let someone else do it. I saw my task in showing that making a web service that provides data from a COM port is not at all difficult.
All sources can be taken on githabe .
I repeat: the presented code is not a complete solution that can be “let in production”. This is only a prototype that shows a fundamental approach to solving a problem.
What else can you work on:
- First, reading the data from the COM port in the Python script is very clumsy - in fact, there is a constant polling “is there anything fresh?”. This approach, of course, loads the processor and one core on my computer is 100% busy.
As a solution, you can use a blocking read with timeout. For this, it is enough to specify a nonzero value (in seconds) when opening the COM port as a timeout, for example:ser = serial.Serial(SERIAL_PORT_NAME, SERIAL_PORT_SPEED, timeout=0.03)
In addition, in the description of the pySerial module there are three examples of creating a bridge: "TCP / IP - serial bridge", "Single-port TCP / IP - serial bridge (RFC 2217)" and "Multi-port TCP / IP - serial bridge (RFC 2217) "- you can see how such problems are solved by professionals. - secondly, only one client can receive data. Until the page is closed on the first client, you cannot connect to this server and receive values on the second computer. On the one hand, this is probably correct: the COM port is one, and there are several consumers - to whom of them should they give the read line? If you think that the answer to this question should be “everything”, then here are my thoughts on this. It seems to me that the issue cannot be solved only by using an “honest” multi-threaded web server (for example, some Tornado or Flask), which can simultaneously serve requests from several web clients. Because you cannot open a COM port from each thread and read from it - in this case, the data from the COM port will leave only one thread / process. Therefore, in my opinion, it is necessary to split the server part into two parts:
- A zmq server that works with a COM port, reads strings from it and sends them through a PUB socket to all interested consumers.
- Python web server instead of connecting to a COM port connects to a zmq server and receives data from it.
If you are not familiar with the ZMQ library (ZeroMQ), then instead of it you can use normal TCP / IP or UDP sockets, but I would strongly recommend to get acquainted with ZMQ, because this library greatly facilitates the solution of such problems. It seems to me that with the help of ZMQ, the solution will be packed with a maximum of 20 lines. (I can not help but write: even if you do not plan to solve the described task, but your work is connected with multi-thread / multi-process programming with data exchange between threads / processes, look at this library - perhaps this is what you have been so long ago dreamed) - data flow is still unidirectional - from the COM port to the web browser. You cannot send data to the Arduino from the browser yet. It seems to me that this task is also not very difficult and, unlike the previous one, it can only be solved
- using a multi-threaded server
- refining the Handler.do_GET method so that it perceives GET requests with parameters and values of which of them sent to the COM port
In my opinion, if you want to write a full-fledged analogue of a serial port monitor based on web technologies built into the Arduino IDE, it is not that difficult. For myself, I see the difficulty only in creating a normal front end. - It is not possible to set the COM port name and its operation parameters through the browser. On the one hand, this seems logical: how does a user on the other side of our planet know from which COM port and at what speed the arduin is connected? But it knows exactly Python web server running on the same computer. But if, after all, it is desirable to give the user the opportunity to change the name of the COM port or the parameters of his work, then again this is easily solved by refining the Handler.do_GET method
- To start the server you need to install python. This is generally not difficult, but if for some reason you don’t want to do it or don’t feel like it, pyInstaller can come to the rescue. With its help, the Python script can be compiled into one executable file (in the case of Windows, into .exe), which can be simply copied to the computer to which arduinka is connected.
Perhaps the best solution would be to use the Go language in this case. As far as I know, in it the task of creating a file for “distribution” is solved better.
In conclusion: the question may arise: “is it not easier to solve this problem through some ready-made cloud?”. Why not publish the data read from the COM port in the cloud, and on the clients simply access the corresponding service in the cloud? Probably, such a decision also has the right to exist, but before applying such a decision, you must answer the following questions:
- Are there ready web services that allow you to publish data with the speed / frequency I need? Are there any free ones among them or are you ready to pay the appropriate money?
- are you ready for the fact that in case of a cloud falling or a connection to it, you will be left without data
- Does it bother you that in order to transfer data from one room to another, they will cross the ocean or half a continent twice?
Source: https://habr.com/ru/post/264663/
All Articles