As promised, we are releasing our linux terminal emulator emulation library under the LGPL, which we use to show virtual machine consoles in the cloud. It is called, respectively, pyte (PYthon Terminal Emulator).

According to our own estimates, the “textual” functionality of the console_codes is close to 100% (from 80 to 90%, as suggested by opportunistic pessimists among developers).

Not implemented: font download codes for the VGA adapter character generator, VESA power saving control, sound alarm (codes for controlling the frequency and duration of the sound), palette control, own charsets; In general, all that is irrelevant to the text.
')
But all the other complex functions are implemented, such as blocking screen regions for recording, scrolling, managing line feed modes, correct processing of attributes for various types of text deletion, etc. - everything that existing applications need, such as nano, adom (in the picture is a fragment of ESC-codes and the resulting image just from ADOM), vim, emacs, mc, aptitude, dialog, yast2, etc. for full rendering.

The library is written in python and sharpened for the convenience of manipulating the screen, abstracting from the graphic representation of the image, which allows it to be used in code that performs further transformations (for example, screen transfers to JS or serialization to the database).

The second important feature of the library is the support of diffs (differences) between the previous and the current screen state. This is necessary for economical image transfer via limited speed channels (read, Internet) and rendering on devices with limited performance (read, javascript).

History of creation

In the beginning there was a remarkable vt102 library in its naivety. She somehow draws a part of vt102 esc-codes, without even thinking about unicode, subtle nuances of scrolling the screen regions, performance and stability of work. As the author wrote about it, it was intended for reading in the game nethack , and no more.

We started with her. For some time we corrected errors and improved, but then we found that, firstly, vt102 does not support some buttons (PgUp, for example), and secondly ... As I wrote earlier, the main advantage of linux terminal is that it expects linux by default.

I had to `from scratch 'to implement all the functionality of the linux console (with the above-mentioned restrictions on the features of the serial port compared to VGA adapters). Thus, the library was smoothly renamed from vt102 to pyte (Python Terminal Emulator).

Despite the fact that the library is now written from scratch, we decided to also publish it under the LGPL, mainly out of respect for the author vt102, whose code we even tactfully deleted, but some of the ideas still used, for example, the concept of the dispatcher.

Internal organization

The library consists of the screen and stream classes. Screen contains a copy of the screen (characters and attributes), Stream is a finite state machine responsible for processing ESC and (in particular) CSI sequences. Stream is fed a sequence of characters, moreover, we have a situation where the program using the library does not know anything about “characters” and operates on a stream of bytes with a custom encoding (incremental decoder, so that there are no strange squares due to a gap in the middle of UTF- 8 characters). In principle, we work not only with UTF-8, but also with other encodings. We do not use this functionality in the cloud now, but for the sake of purity of conscience we have implemented it.

After eating the entire sequence of characters that form the CSI sequence (this is a special class of ESC sequences starting with ESC codes, followed by parameters and command, for example, ESC [1; 31m generates a red bold color tone). Each output generates an event (character output, scrolling, etc.) for all subscribers, among which is the library itself that processes these events. Of course, external code may also be among those who wish, for example, in this way we will learn about the appearance of new characters and report them to the Java script in the control panel.

Objects inside Screen are stored as normal python objects. We had a version that uses faster binary arrays, but we stumbled upon some difficulties with converting between “convenient” for the library format and array, so the performance of the arrays was heavily blurred with constant conversions. By the way, the use of snake objects simplifies access to their giblets and, most importantly, serialization to the database (which is the killer feature of our console, which is experiencing the transfer of a virtual machine between servers without breaks and zeroing the screen content).

By the way, this library works fine under pythons from 2.6 to 3.2+ and even under PyPy .

Future plans

In the near future, the library will be supplemented with scrolling support (the same Shift-PgUp), which, by the way, we will add to our cloud.

In addition, there is an idea to do a little useless perfectionism and conform to standards better than the Linux console itself, that is, to pass vttest without a single error.

The thing that we are not planning yet (although we will take commits with great pleasure) is an expanded support for Unicode - support for the normalization of characters, symbols of different widths, etc. In principle, python has everything for that - but we have no need for such (until we set our sights on the Asian market).

If this library is useful to someone, we will be grateful for reports of use, bug reports and complaints.

Links

easy_install pyte
github.com/selectel/pyte

Who say thank you?
bobry for the code
o_l and akme for agreeing to publish under the LGPL.