Data collection from recording devices
Greetings to all the inhabitants of Habr! I would like to share my idea, which I partially managed to realize. But I cannot appreciate the significance of such a project, and I would like to hear your opinion and your constructive criticism on this matter.
Having come to my native plant as an engineer in the ASODU group, one of the tasks for me was to follow the mechanism for collecting data from various types of recording devices. I note that the factory has a good “zoo” of this kind of equipment. As it is known, for recording devices, specialized software is always provided, which allows for its configuration and polling. But far from all types of devices there is software, with which you can get data from the device and put them in a common environment for further processing and archiving. So this problem was solved even before me, by writing a single program that would poll all the devices that are available and upload the collected data into a single database. But the problem was that when a new type of device appeared, it was constantly necessary to recompile this program, besides it was tightly tied to a specific DBMS. Having neither a configurator, nor any tester at hand, all this led the process of accompaniment into continuous flour. And then there was the idea to implement a certain system that would simplify the work on supporting the data collection mechanism as much as possible. For such a system I put forward the following requirements:
To implement my venture, I used the following tools:
Compilers: gcc-3.4.2, gcc-4.6.1 and tinyc-0.9.25
Graphic library: wxWidgets-1.8.10 + wxFormBuilder-3.2
Database: SQLite-3.7.6.2
')
In the implementation, I briefly describe only the main part of the program - data collection. The rest of the software package, in my opinion, is not as interesting in its implementation as this one.
The most important task for me was to store data in the system. The buffer implemented by me took the following form:
Each polling thread has access to its own branch of the buffer, from which it takes all the initial (initializing) settings, writes its own state and records the received data. The data upload stream has access to both the survey branch (with read-only access) and its own export branch. Access to objects marked in red in the figure is synchronized using critical sections.
The core of the program is shown in the figure below. It works as follows. When the configuration file is read by the loader, a final buffer is formed (see figure above), a list of downloaded survey plugins, and a list of data upload plugins. Both the generated buffer and the lists of plug-ins are transferred to the kernel. The kernel, in turn, initializes a stream for each com-port, and at the same time, it sends a buffer branch and a list of device plug-ins to it. Once the polling threads are all created, the kernel begins to initialize along the same pattern and data export threads. But, besides the above, it transmits to everyone a constant link to the survey thread. Thus, export streams can receive at any time all the information, both on the progress of the survey process and the final results of the survey.
Since there are two types of polling programs (as a service and as a custom graphics application), the kernel is placed in a dynamic library that both applications use.
All threads generated by the kernel, among other things, receive a constant reference to the kernel itself. Thus, each of the threads can monitor the state of the kernel (RUN, STOP). In the event that the kernel goes into STOP mode, then all threads begin to automatically shut down. When switching to RUN mode, the kernel re-creates the threads described above.
In order to interrogate the device, it is necessary to have two functions: the function that forms the final packet sent to the device, and the function that processes the received response from the device. Thus, the plug-in interface consists of two functions of forming and processing a package and another function that returns information about it. This information is needed both by the user and the program, due to the fact that its content contains information about the length of the packet being formed and sent. As a result, we have the following functions:
I think it is unnecessary to describe the structures with which these functions operate, because my goal is to describe the general principle of the system operation. But to whom it became interesting, can glance in the documentation .
The export plug-in interface consists of four functions:
Structures that operate on these functions, I also will not describe for the reason indicated above.
At the moment, the complex includes the following software:
The complex is focused on the Windows OS, although there is a strict binding to WinAPI only in the class that works with the COM port. And of course the device polling service. Everything else is based on the classes and functions of the wxWidgets library.
Suppose that there are two devices of the type "rmt-59 " and "ecograph-t" . Each of them is connected to a separate port to the interface converter "RS-485 - Ethernet". On the computer that performs the survey, there is a driver that converts "Ethernet - RS232". Thus, we have two com ports (for example, com-10 and com-11), on which one device is located. For both devices, suppose address 1. Both devices are configured for a data transfer rate of 19200 bps.
First you need to make sure that the available plugins are suitable for working with these devices. To do this, run the TestRequest program and try to interrogate these devices.
After that, you should create a poll configuration. Run the Editor and set up a poll.
If you created a new database, then you need to register the path to it in the Reader.ini file. To test the survey run the program ReaderGUI
Now, you need to take care of the data export. I did not create specialized plug-ins. For the test of the export operation, one test plugin is included in the package, which exports data to a text file. To begin with, you should test its operation using the TestExport program.
Now that we have verified that the plugin is working correctly, we can add it to the survey configuration.
All configuration and testing is complete. Now you can install and start the service. Service management is carried out using the ReaderSvcCtrl program.
Of course, I could write much more, but in order not to bore the reader, I will not do that. Here is a link to my project. I will be happy to answer all your questions in the comments.
Many of the shortcomings of the system became clear already during operation These disadvantages include the following:
Brief background
Having come to my native plant as an engineer in the ASODU group, one of the tasks for me was to follow the mechanism for collecting data from various types of recording devices. I note that the factory has a good “zoo” of this kind of equipment. As it is known, for recording devices, specialized software is always provided, which allows for its configuration and polling. But far from all types of devices there is software, with which you can get data from the device and put them in a common environment for further processing and archiving. So this problem was solved even before me, by writing a single program that would poll all the devices that are available and upload the collected data into a single database. But the problem was that when a new type of device appeared, it was constantly necessary to recompile this program, besides it was tightly tied to a specific DBMS. Having neither a configurator, nor any tester at hand, all this led the process of accompaniment into continuous flour. And then there was the idea to implement a certain system that would simplify the work on supporting the data collection mechanism as much as possible. For such a system I put forward the following requirements:
- Interrogate any type / type of device. This should be achieved by expanding the program through modules.
- Unload data as you like, anytime, anywhere. Such an approach should also be achieved at the expense of the modules.
- Availability of a tool that allows you to set up the entire survey process as easily as possible.
- Availability of a tool that allows you to test and debug both survey modules and data upload modules.
- The task of the survey should work as a service, but the possibility of visual observation of the progress of the survey and data upload should also be present.
Tools
To implement my venture, I used the following tools:
Compilers: gcc-3.4.2, gcc-4.6.1 and tinyc-0.9.25
Graphic library: wxWidgets-1.8.10 + wxFormBuilder-3.2
Database: SQLite-3.7.6.2
')
Implementation
In the implementation, I briefly describe only the main part of the program - data collection. The rest of the software package, in my opinion, is not as interesting in its implementation as this one.
Clipboard
The most important task for me was to store data in the system. The buffer implemented by me took the following form:
Each polling thread has access to its own branch of the buffer, from which it takes all the initial (initializing) settings, writes its own state and records the received data. The data upload stream has access to both the survey branch (with read-only access) and its own export branch. Access to objects marked in red in the figure is synchronized using critical sections.
Core
The core of the program is shown in the figure below. It works as follows. When the configuration file is read by the loader, a final buffer is formed (see figure above), a list of downloaded survey plugins, and a list of data upload plugins. Both the generated buffer and the lists of plug-ins are transferred to the kernel. The kernel, in turn, initializes a stream for each com-port, and at the same time, it sends a buffer branch and a list of device plug-ins to it. Once the polling threads are all created, the kernel begins to initialize along the same pattern and data export threads. But, besides the above, it transmits to everyone a constant link to the survey thread. Thus, export streams can receive at any time all the information, both on the progress of the survey process and the final results of the survey.
Since there are two types of polling programs (as a service and as a custom graphics application), the kernel is placed in a dynamic library that both applications use.
All threads generated by the kernel, among other things, receive a constant reference to the kernel itself. Thus, each of the threads can monitor the state of the kernel (RUN, STOP). In the event that the kernel goes into STOP mode, then all threads begin to automatically shut down. When switching to RUN mode, the kernel re-creates the threads described above.
Interface of device polling plugin
In order to interrogate the device, it is necessary to have two functions: the function that forms the final packet sent to the device, and the function that processes the received response from the device. Thus, the plug-in interface consists of two functions of forming and processing a package and another function that returns information about it. This information is needed both by the user and the program, due to the fact that its content contains information about the length of the packet being formed and sent. As a result, we have the following functions:
- GetInfo - information about the plugin;
- GetPackage - package formation;
- GetData - processing the received packet.
I think it is unnecessary to describe the structures with which these functions operate, because my goal is to describe the general principle of the system operation. But to whom it became interesting, can glance in the documentation .
Export plugin interface
The export plug-in interface consists of four functions:
- About - information about the plugin;
- Begin - the function is executed once after the kernel enters RUN mode. It is designed to make any connection to the data store. If the function returns an error, the thread writes an error to the buffer and terminates its work.
- Export - directly export data;
- End - the function is executed once after the kernel goes into STOP mode. It is executed only if the Begin function did not return any error. The function is designed to disconnect from the data storage.
Structures that operate on these functions, I also will not describe for the reason indicated above.
Result
At the moment, the complex includes the following software:
- Editor - survey configuration editor;
- ReaderGUI - polling program in the form of a user application;
- ReaderSvc - polling program as a Windows service;
- ReaderSvcCtrl - polling service management;
- TestExport - testing export plugins;
- TestRequest - testing poll plugins.
The complex is focused on the Windows OS, although there is a strict binding to WinAPI only in the class that works with the COM port. And of course the device polling service. Everything else is based on the classes and functions of the wxWidgets library.
Work example
Suppose that there are two devices of the type "rmt-59 " and "ecograph-t" . Each of them is connected to a separate port to the interface converter "RS-485 - Ethernet". On the computer that performs the survey, there is a driver that converts "Ethernet - RS232". Thus, we have two com ports (for example, com-10 and com-11), on which one device is located. For both devices, suppose address 1. Both devices are configured for a data transfer rate of 19200 bps.
First you need to make sure that the available plugins are suitable for working with these devices. To do this, run the TestRequest program and try to interrogate these devices.
After that, you should create a poll configuration. Run the Editor and set up a poll.
If you created a new database, then you need to register the path to it in the Reader.ini file. To test the survey run the program ReaderGUI
Now, you need to take care of the data export. I did not create specialized plug-ins. For the test of the export operation, one test plugin is included in the package, which exports data to a text file. To begin with, you should test its operation using the TestExport program.
Now that we have verified that the plugin is working correctly, we can add it to the survey configuration.
All configuration and testing is complete. Now you can install and start the service. Service management is carried out using the ReaderSvcCtrl program.
Afterword
Of course, I could write much more, but in order not to bore the reader, I will not do that. Here is a link to my project. I will be happy to answer all your questions in the comments.
Many of the shortcomings of the system became clear already during operation These disadvantages include the following:
- No type of polled channel. The necessity of the type is due to the fact that many devices, for different types of values (analog channel, math channel, integral value), require a specially formed query.
- Lack of multiplier concept. Those. Some devices, when polled, transmit the value as an integer. And the position of the comma in the number of such devices is polled separately. And having such a multiplier, the user could change the position with a comma. For example, if for a given channel a separator between an integer and a fractional value is placed after the first number, then a factor of 0.1 would allow the number to appear in the proper form. And having received the value 123, the system, multiplying this number by the corresponding multiplier, would produce the result 12.3.
Source: https://habr.com/ru/post/142652/
All Articles