Greetings to you, dear habrovchane!

Today I wanted to start a story about one interesting product I presented in Tibbo . This product can be useful to a wide range of IT-professionals, including system administrators, information security specialists, and, finally, simple developers who have no-no, and have to program communication with devices and other low-level I / O.

The conversation will be about the programmable terminal / sniffer of IO Ninja (hereinafter I will omit the word “emulator” and speak simply “terminal”). I suspect that the definition of “terminal / sniffer” itself may look quite unusual, if not to say strange. Therefore, we start with the history of the IO Ninja.



')

Motivation and history

The very first product of our company was the EM100 embedded module (embedded module) for converting a Serial RS232 serial interface into Ethernet 10BASE-T (now its equivalent is the EM1206 module).

On this device, protocols of interaction between the firmware (on modules) and virtual port drivers, as well as configuration utilities (on PC) were tested and tested.

As you understand, in our imperfect world, no program works immediately after writing, and we have many months to debug communication protocols. In our particular case of debugging a Serial-to-Ethernet converter, this resulted in the need to use the following utilities in parallel:

• Serial terminal;
• TCP terminal;
• UDP terminal;
• Serial monitor;
• Ethernet sniffer.

In addition, to debug our protocols, we also needed:

• Listen and accept incoming TCP connections;
• Send broadcast (broadcast) UDP packets;
• Receive responses from multiple nodes;
• Send and be able to display binary data in hexadecimal form.

There were no difficulties with monitors for Ethernet and Serial - even free products like Wireshark (which was also called Ethereal at that time) completely satisfied our requests. But with the terminals, we ran into problems.

Like monitors, there are a lot of terminals. But there is a nuance: most of them are very lame in terms of sending and displaying binary data. If this opportunity is provided, then in an extremely inconvenient form - a full hexadecimal viewer of incoming packets and an outgoing editor (with the possibility of selecting, copying, pasting, etc.) was not found in any of the terminals we tested. In addition, the situation is also bad with regard to the reception of TCP connections and a completely total darkness - with support for working with broadcast UDP packets.

It was after an unsuccessful search that the idea was born of creating a universal utility for debugging communication protocols that could do all of the above and would do it well. Yes - even dream so dream! - would be programmable and would allow us to easily add analyzers of our own protocols and test scripts (such as, for example, wait for commands and send answers to them dynamically generated according to certain rules).

Dreams of programmability, of course, for a long time remained just dreams. But the utility for sending and displaying binary data, while supporting TCP reception of connections and UDP broadcasting, was spliced ​​out quite quickly. She was called ingenuously - SockTerm - and looked like this:



In addition to supporting the socket operating modes that are not available in standard terminals, this utility demonstrated the extreme convenience of its approach to logging, namely, hexadecimal “sheets” with pasting new packages to the previous ones.

Work on the utility continued in its free time from the main projects and tasks for a couple of years. The logging module was gradually refined, expanded and eventually reorganized as a full-fledged engine (its capabilities will be described in more detail below). This engine was quite suitable for use in both terminals and sniffers - which was enthusiastically brought to life. The result was the appearance of the first version of IO Ninja.



This utility allowed us to work with all the transports we are interested in (Serial, TCP, UDP) both in the sniffer mode for passive listening and in the terminal mode for the directional transmission and reception of data. Multiple modes of operation were implemented as plug-ins (written, like IO Ninja itself, in C ++), which in theory allowed us to easily add support for new modes, and eventually publish the SDK and allow customers to do it themselves. However, in practice, the latter has not been implemented - those dreams of programmability did not give us peace of mind, and plug-ins in C ++ - in this case, is not the best architecture.

What we needed instead was to make a ninja scripted. But bad luck - in languages ​​without pointers and address arithmetic, working with binary data (and this is an integral part of any protocol analyzer) turns into a nightmare like “pull bytes on the N index, move 8 bits to the left, add up with the byte on the N + 1 index etc. " And scripting languages ​​with pointers in nature does not exist. More precisely, it did not exist before - we corrected this annoying misunderstanding by creating the Jancy language.

This article does not set itself the goal of telling about all the possibilities of the Jancy language (although, if respected habrovans show interest, the corresponding articles will definitely appear). Now we are talking only about the fact that in our hands - after a long series of half-dead prototypes (and then a prototype of a completely viable and used when creating the second, scripted version of IO Ninja - its screenshot you see below) was a language with safe pointers and address arithmetic.



At the same time, this language is designed so binary compatible with C / C ++ that a direct call between Jancy and C ++ is possible, and this leads, firstly, to high efficiency of interaction between the script and the host C ++ application, and secondly, to ease creating C / C ++ libraries for use from Jancy scripts. In addition to ABI (binary) compatibility, Jancy also has a high degree of compatibility at the source code level, which makes it possible to copy and use (often without any modifications at all) C code taken from publicly available sources.

In 2014, work was completed on the creation and release of the third version of IO Ninja, based on the Jancy language. About her and will be discussed further.

If you highlight the most significant features of the new IO Ninja, then it will be:

• Programmability;
• Logging engine;
• Package Builder;
• Universal redirector;
• Combining all kinds of utilities for debugging communications in one product.

The fact that a ninja can be turned into using programmability will be discussed in later parts of the article; in this same let's see what is available out of the box.

Logging engine

The logging engine since the first non-programmable version of IO Ninja offered something that was not available in other terminals and sniffers. This engine is highlighted primarily by the approach to displaying log data.



Instead of a pair of “main list” and “recording details” windows (as in most monitors and log viewers), the IO Ninja engine offers a log of “endless sheets” with alternating text messages and binary data — something like an unboundedly growing heterogeneous HTML page. By the way, it was in connection with the unlimited growth of the log that the development of its own engine was required - otherwise it would be possible to fasten something like Gecko or WebKit.

At the same time, despite the actual unlimited size of the log (actually, it is limited only by the free space on the disk), the IO Ninja engine provides a “smooth” scrolling, while without directly depending on the amount of memory consumed from the size of the log.

One of the reasons for avoiding standard terminals was the lack of convenient means for working with binary data, so it is not surprising that the IO Ninja logging engine is literally “sharpened” for this task.

Binary data can be displayed in hexadecimal or textual form, with customizable line widths and tabs, and in any of the UTF encodings (including variations with reverse byte ordering). Packages of the same type (for example, incoming) are optionally glued together. Gluing can be configured to "always", "never" and "if the interval does not exceed a certain threshold." Gluing places are highlighted (again, optional) color - checkered.

The IO Ninja log also supports collapsing / unfolding of records, which may be necessary to switch between short and full descriptions and is used, for example, in a network sniffer:

Package builder

Speaking about the support of working with binary data, it is impossible not to mention the IO Ninja tools for preparing outgoing packets.

For the simplest cases, IO Ninja has a text editor package. This editor understands all the well-known escape sequences (\ n \ e \ x01 \ u2661, etc.), therefore, introducing a couple of special characters is not a problem. And if you want to make a package, consisting entirely of special characters? Welcome to the IO Ninja Package Builder!

The package builder is one of the strengths of IO Ninja available out of the box. In addition to the modern hex editor with support for all UTF encodings, editing of nibbles (nibbles), restrictions on the size of the package above and below, convenient highlighting of the selection, as well as all other features that can be expected from a good editor (insert, overwrite, copy, paste, undo, redo) - in IO Ninja there is a tool for creating packages from a template.

Remember, we talked about the high degree of compatibility between Jancy and C? Preparing binary packages is one of the places where the benefits of this compatibility are reaped.

Phase 1 : find the source code for the implementation of the C protocol we need in publicly available sources;

Phase 2 : copy descriptions of packet structures in IO Ninja and, perhaps, a bit of retouching (after all, Jancy and C are different languages);

Phase 3 : ???

Phase 4 : PROFIT! You can fill in the package fields from the property grid editor (property grid)!

Instead of conclusion

This concludes the introductory review, and in the next part of the article we will demonstrate some real life scenarios from our company, in which IO Ninja helped solve certain practical problems better than their counterparts.

In the meantime, you can download and try the latest version of IO Ninja for Windows or Linux at tibbo.com/ioninja/downloads . There is also a companion IDE based on NetBeans available for download (however, more about this another time).

The following parts of the article

Part 2: http://habrahabr.ru/company/tibbo/blog/255221/
Part 3: http://habrahabr.ru/company/tibbo/blog/256803/