How we built the data center alarm system

It so happened that in the Embox project team I have the most experience in the field of automated control systems: at the previous place of work I developed industrial controllers. Therefore, it is not surprising that when the task arose to make the automatic control system for the LEDs in the data center, I was asked to work out the project architecture. It was originally planned to purchase ready-made remote controllers for I / O port control, but after a more thorough study of the requirements, it became clear that the option of developing a custom controller was preferable for the customer. Actually you see it in the photo.
For those who are interested to know about what rake we stepped on, what exploded chips look like, how to connect the ground correctly on a DC / DC converter, and, of course, why we used our project, I ask for cat. Careful, a lot of pictures!

Customer’s wish is law

Initially, the task seemed very standard, you need to manage + 24V diodes according to messages coming from the customer's SCADA system. Of course, SCADA receives messages about the status of any sensors, and not about the need to light or extinguish a particular diode, but this is solved by developing a kind of PROXY that translates emergency-type messages in the N cabinet to the corresponding command sent via the Modbus protocol. on a specific controller.

Accordingly, the architecture proposed by me was very simple and proceeded from my experience in the field of automated control systems. Namely, take electric boards, such as those in which there are meters, fill them with controllers with a maximum set of discrete outputs, for example, MOXA ioLogik E1211, supply power sources, and, in principle, everything. That is, the picture was as follows: the shield, as it should be in the ACS, is mounted on the wall, one Ethernet network wire and 220 power supply come into it, and a bunch of wire pairs go to the LEDs. There were, of course, questions about such a large number of diodes, but they were easily solved by increasing the number of cabinets.

But with further refinement of details, nuances began to emerge that began to influence such a simple and understandable scheme of implementation. The first condition that shattered our confidence was that the equipment should be installed only in telecommunication cabinets. On the one hand, nobody canceled the DIN rail, but on the other hand, you see, the telecommunications counter will look somehow miserable if I / O controllers and power supply units are hung there. In addition, the installation and maintenance of such a design is a fairly non-trivial task. The obvious solution was to use the rack case, and already in it hide the entire inaccessible “user” kitchen. In order to make it easier to scale the circuit and increase the number of outputs, I suggested using controllers that do not work via Ethernet, but via RS485 (for example, ioLogik R1212), and put a Modbus-RTU to Modbus / TCP converter (MGate MB3170-T) at the input .
')
But the wish of the customer to have a friendly user web-interface strongly confused our plans. I began to propose not so beautiful solutions, for example - to put proxy in each case and so on. These ideas were not very successful, but I continued to defend the decision to do everything on ready-made elements, until it became clear that the delivery time of ready-made controllers in sufficient quantity did not fit into the order time.

Own bike should be comfortable

Began to think about other solutions. The guys from the team said:

“What is there to do at all? We just need to control the controller's legs, let the voltage and current be quite large, but there are keys like ULN2803A. We have a couple of STM32F4Discovery debugging boards on the table right now, and a similar task is perfectly solved on them. In addition, if you use these debugs, you can very easily implement a web interface, since the web server is already running there. ”

I objected that not everything is so simple that we, for example, do not have support for the ModBus protocol, which is the standard for this kind of devices. To which one of the members of our 0xdde team the next day did support by moving the libmodbus library.

In general, I gave up, and it was decided to make the controller using the existing STM32F4Discovery, circuit board, MGTF wires, keys and some kind of mother .

In order to look beautiful, we decided instead of the terminals to put the connectors on the three contacts W7166-03PSGB00, because, as I said, the control is carried out by pairs of diodes, and they have a common ground. Unfortunately, we could not find any connectors for three contacts with a normal delivery time and put the W7166-04PSGB00 connectors on four contacts. Initially, we wanted to have 80 such connectors located on one controller, but here we were again confronted with the fact that 80 such connectors barely fit onto the cover of the U1 rack rack. In addition, we got that the 24V power supply should be at least 20A. Therefore, we decided to limit ourselves to 40 connectors (pairs of diodes) and put everything into the U2 case.

The next question that arose was that our STM32F4Discovery did not have 80 free leads for controlling 40 pairs of diodes. Help had to call for shift registers 74HCT164N.

Requirement analysis is important

Problems with the procurement component, we omit. But about our oversight when reading the description of the chip keys, I will tell. If this is not interesting, then at least instructive.


As I said, the diodes are paired and have a common wire - ground. The diodes are controlled by the positive voltage on the control contacts. So, when the diodes arrived, we turned on one diode, successfully controlled it and calmed down. But when it came to checking on a real pair, it turned out that these keys control the breaking of the ground (common wire), that is, the switching circuit should be the same as in the picture.

In general, we urgently had to change the type of keys, first on the UDN2981A-T, and then on the TD62783APG, since the former were considered obsolete, and besides they cost quite a lot, they also turned out to be difficult to buy.

When, finally, the scheme for one cascade was created, it looked like this.



The result of an oversight on the documentation was a couple of lost days and the presence of a couple of hundreds of microcircuits that had already arrived by that time.

Mounting is not that simple.

After we finally earned one cascade under the control of the STM32F4, we proceeded to the next stage, namely to create a full-fledged prototype of the device.

The idea was as follows.
We take the circuit board, arrange the chips and connectors on it. We solder all this, we connect according to a rather simple scheme (will be next), add a couple of connectors, one for power and one for control signals from STM32F4Discovery, then power up, connect the loop with Discovery, and only the body remains. In general, everything seemed simple and easy. However, the reality was not so rosy.

The first problem we encountered was a surface mount. The fact is that there is a big difference between soldering dozens of wires and hundreds of wires, and the latter turned out to be a terrible occupation.

In general, they were soldered for a long time, but this is not the worst. When, finally, they soldered, rang and gave food, an explosion occurred. Turned off, rang again, turned out to be short between the ground (5B) and + 24V. We simply drove two tires, and they touched the same metallized hole, and we didn’t ring the power supply between us. In general, they themselves are to blame, but it was very scary.

We taped the place of the short one with the tape (yes, yes, it did not do without the tape):



And then they made another mistake, namely, they didn’t remove the exploded microcircuit from the circuit. At least, we think that the reason for the next explosion was exactly that.

And this is what the chips cut from the board after such a short look like.

Do not work on Saturday

We have ensured that, after energizing the circuit, nothing exploded, did not smoke, and did not get very hot. Having been delighted, they hooked up the Discovery train and even tried to control the diodes, but to no avail. It was necessary to look for a problem. First of all, we checked the supply voltage (+ 5V and + 24V). They were correct. Then the voltages on the output contacts of the shift registers — they, too, were correct. And they could even be controlled. But, oddly enough, nothing changed at the outputs of the keys; they were always open. Long check of the connection scheme was unsuccessful.

Measuring the voltage once again, we were very surprised to see that at the output contact of the shift register is + 5V, and at the input of the key something strange, although they are simply connected by wire. A little later, it turned out that we measure the voltage at these points from different lands. We measured the potential difference between the 5V and 24V lands and saw 5V. It turned out that we simply incorrectly connected the TRACO TEN 12-2411 DC / DC converter chip. I don’t even know how it occurred to us, because it’s obvious that if two voltages come on one and the same microcircuit, then they should have a common ground, and we didn’t connect the earth and got two untied voltages on the circuit. In my defense, I can only say that we are programmers, although they are systemic, and even an obvious fact about a common point, we were suggested by familiar engineers.

In general, we understood our mistake and decided to connect the lands, but since it was already about ten o'clock on Saturday evening, we decided to make it a thin wiring, while holding it with hands, simply shorting these points on the board. I don’t know what happened next, or the hand trembled at the person who held the wire, or after another accident another key flew out, but this time I saw the microcircuit burning, literally burning, and I used to think that neither the ceramic body nor the metal contacts can burn. In general, even minus the chip of the key and the shift register, which we bit, remembering about the previous experience. And on this beautiful note, we went home, because further attempts to do something in a hurry seemed out of place to us. After all, we did not have enough keys even for one device. Well at least they should have come on Monday.

Simplicity will save the world

At the end of Saturday's battles, we still got a working prototype of the device, and it looked like this:



Resistors (300 Ohms) in the photo imitate the load. Before the demonstration to the customer, we did test runs for several hours. Heat resistors, by the way, up to 120 degrees.

I will not talk about working with the body and the final assembly, just bring pictures of the entrails, and that in order for the story to be complete:





We have demonstrated the work to the customer, the functionality has arranged it, the appearance too. The only thing he asked to change was to put in place of our connectors something that was adopted in telecommunications: RJ45 or RJ11, since this is much more familiar, and therefore easier to operate and maintain. This time, after consulting with the engineers about whether the necessary currents would withstand such connectors, whether there was enough space on the front panel, and so on, we decided to install RJ12. They were in stock, well, and constructively take up less space than RJ45. In addition, given the previous experience, we decided that soldering on a breadboard is not an option, and ordered the design of a printed circuit board. And since it turned out to be very simple, bilateral, even without metallization, we were let out of it in less than a week.

The upgrades carried out greatly simplified our lives and made the board itself more beautiful.

What does it have to do with Embox?

The strongest argument, as you understand, for using Embox was that we know it well, to say the least.

But still I want to note the technical characteristics that greatly simplified the creation of the desired device. The main thing, of course, is having a full-fledged web server with cgi support. As a result, you can control our device using a browser, the page looks as close as possible to the real front panel.



Of course, you can argue that there is an easier way to get a web server on the device, namely, to use Linux. I do not argue this way, and it will certainly be more functional and will allow you to use a bunch of frameworks and programming languages ​​for development. But, first, we are sincerely convinced that managing a GPIO from under Linux is not the right approach. And secondly, he simply will not get on such a hardware platform without special tricks .

Well, if you use FreeRTOS with the lwIP stack, which comes bundled, then the implementation of this functionality is much more complicated, and, of course, libmodbus is so easy to move.

PS When developing this device, no LED in the datacenter suffered