Hardware solution hang GSM-modems

1. Problem
2. Solution to the problem
3. How is the device used
4. Result

Problem

Our organization has 140 unattended units (complexes), which are installed in remote locations from Anapa to Pevek, mainly at departmental sites. The complex itself consists of a compact Win32 computer and a GNSS receiver with an antenna on the roof. We need to receive data from them every 15 minutes. Some stations have access to a departmental network, but often have to use the mobile Internet . Here is the complex:


Ordinary USB modems (so-called “whistles”) have proven themselves very bad: they hang too often, and the reception quality of a weak signal leaves much to be desired. We began to use more expensive industrial-type USB-modems with a remote room antenna. But they periodically hung up, which led to the inaccessibility of the complex. We had to call at the meteorological station, where our complexes are installed, for a long time to explain to older observers what kind of cable you need to pull out and insert back. Not everyone agreed to perform such a complex procedure.

Solution to the problem

There are GSM routers on the market with a watchdog built-in inside, but most often they work with end equipment over Ethernet, which complicates setup and costs several times more. Stable modems at a reasonable price do not support 3G.

We decided to go the other way. At first, we set up an automatic daily reboot of computers, but this did not always help: power was supplied to the USB modem even when the computer was turned off, and it was impossible to reconfigure in the BIOS. In addition, a reboot interrupts data collection.
')
So the idea arose to create a USB relay that will hard reset the power of the GSM modem in case of its hangup. We received a suitable product to order and began to open the "plus" using the relay by a team from a computer. In general, the solution worked, but when the modem was turned on, in half the cases, the computer went into reboot. This was due to the fact that the modem had two large input capacitors that caused a current surge when turned on, which the USB port was not designed for, and our small computer felt this. To avoid unwanted reboots, we ordered a special release of a USB relay with a two-step load switch : first, the modem is turned on through a resistor, and then directly. Here is the finished device with an extension for convenience:



The design has two USB plugs: one to control the relay, and one to transit to the controlled device (modem). The only USB-slot is designed to connect the device, it is in it that the plus command is opened by the control command.

How is the device used

Next, we wrote a control program that determines when to reboot the modem, and sends a command to reboot it. It works according to the following algorithm:

1. 3 times per hour the program is launched from the Windows Task Scheduler.
2. The program checks the availability of our server.
3. If 4 pings in a row were unsuccessful, then the program sends a command to turn off / on the modem via a relay. In this case, a delay is made before switching on so that the built-in capacitors have time to discharge.
4. Ping results and actions taken are logged (turn off / turn on the modem).
5. At the first opportunity, logs are sent to our server, where all events are recorded in a database for subsequent centralized analysis of all causes of station failures.

Here is an example log:

Date Time	Number of servers available	Number of unsuccessful pings in a row	Primary server availability	Command execution
2017-01-24 20:37:00	0/2	6	no
2017-01-24 20:57:00	0/2	7	no
2017-01-24 21:17:00	0/2	eight	no	resetmodem
2017-01-24 21:37:00	0/2	9	no
2017-01-24 21:57:00	0/2	ten	no
2017-01-24 22:17:00	0/2	eleven	no
2017-01-24 22:37:00	0/2	12	no	resetmodem
2017-01-24 22:57:00	0/2	13	no
2017-01-24 23:17:00	1/2	0	ok
2017-01-24 23:37:00	1/2	0	ok
2017-01-24 23:57:00	1/2	0	ok

From this log, it is clear that two modem reboots were made, after the second of them our server became ultimately available.

A typical situation: the money on the SIM card ran out, the modem does not work, the station is not available. The account was replenished, but the modem itself cannot start the mobile Internet until it reboots the USB relay (it may be necessary to re-register in the GSM network via cold start). If everything is okay with the money, the network does not disappear, the modem will work without problems for about 36 hours, and then, most likely, it will freeze again and will require a reboot. Previously, to reboot the modem, we had to call other time zones, disturb people, explain ... Observers at remote locations received parcels with our fancy device, overpowered the instructions for connecting three USB plugs, and we didn’t ask them to reboot the modems anymore, now it happens on click USB relay and solved by the computer itself.

The program itself is written by us in JScript (it is such a Javascript-like language built into Windows) and installed on the station remotely either via Remote Desktop or through its own centralized station management system using PowerShell Remoting . It must be said that using PowerShell it is easier to work with a large number of stations: I wrote instructions once, and installation is done automatically at all stations. In addition, you do not need to wait for the graphical interface to load (as is the case with Remote Desktop).

At some critical stations with unstable wired connection, a GSM modem is used as a backup channel. And it was desirable to turn it on only when the wired channel is not working. Here, again, the USB relay comes to the rescue: as soon as the pings stopped passing through the main (wired) channel, the modem is turned on, Windows automatically switches to a new connection with another default gateway. Then, once a day, the modem switches off via a relay so that the main connection can be checked. If it still does not work, the modem is turned back on.

It should be noted that managing default gateways is usually a whole story: there is only one “default gateway” in IP laws, and routing table manipulations lead to problems. The fact is that when a mobile connection is disconnected, the numbers of network adapters, to which manual entries in the routing table are attached, can change. There are two ways out: either “iron” disconnection of the second channel through the relay, or fine and accurate setting of the gateway metrics with checking the logic of the metrics in all scenarios.

All events are written to the database and displayed in the interface of the accounting table of the central server stations:



Result

I briefly summarize how the resulting system works:

1. The computer collects data from the measuring device of the station.
2. USB-modem transmits data over the cellular network.
3. USB relay controls modem power.
4. The program turns off and turns on the power of the modem (through a relay) in case of a hang or when it is necessary to switch the communication channel.
5. The relay turns on the modem smoothly to avoid current surges.

The modems that we use cost about 6,000 rubles each, and 2,300 rubles more. USB relay. In sum, it is cheaper than using modems with a built-in watchdog timer, which have several disadvantages:

1. Watchdog modems use a separate power supply unit, the connector from which would have to be soldered to our 12-volt uninterruptible power supply.
2. It is not possible to simply turn off or turn on the modem to automatically switch between the wired communication channel and GSM.
3. There are cases when the modem is not frozen, but the SIM card is not registered on the network. Whether the watchdog will help here is unknown. As well as no one guarantees that he will not hang. Our solution acts on the end result: no connection - reset the power of the modem.

Using the same USB relay to control the modem has the following advantages:

1. Modems can be used already available, including inexpensive, often hanging.
2. When a wired connection is working, the modem can simply be turned off, in order not to consume traffic, but to switch it on only if there are problems with the wired connection. And automatically.
3. Easy installation: you only need to connect three USB plugs and install the program. It is difficult to make a mistake in the physical connection.

You can see and hear how the relay works, turning off and on the modem, in this video:


As a result, the problem of GSM-modem hangup is solved by iron (in the direct and figurative sense) and does not bother us anymore. But at the same time there were additional opportunities for managing communication channels.

The problem of remote unattended points arose because of the specifics of working with distributed systems: we are monitoring space weather and maintain a network of measuring stations, you can read about it here .

If you have any questions, please contact: vort@ipg.geospace.ru

PS I express my gratitude to my colleague Alexey Vasilyev for participating in writing the article.