Minicomputer from a router with OpenWRT: we write the framebuffer driver
Good afternoon, dear habrovchane. Here we come to the most interesting and important part of my series of articles about turning a small router into a minicomputer - now we are going to develop a real framebuffer driver that will allow you to run different graphical applications on the router. So that the enthusiasm did not fade away, here is a video of one of these applications - I think most people will recognize this magnificent old quest:
In case you missed the previous parts, here are the links:
1 - Minicomputer from a router with OpenWRT: we are developing a USB video card
2 - Minicomputer from router with OpenWRT: write USB class-driver for Linux
So, get down to work.
Iron has not changed again (although we will definitely change something in the firmware in the next article), so we will start with an overview of what we have to do.
Despite the fact that our driver from the previous article already knows how to display graphics, it is very difficult to fully use it (for example, to display the console or launch a graphical application). The fact is that such applications require a well-defined, standardized interface, the so-called framebuffer. When registering a framebuffer in the system, we will need to fill in several specialized structures-descriptors, this time much more specific than the abstract “file operations” that we filled out last time. Operations will also be there, but there will be special callbacks, such as fb_imageblit (called when someone wants to transfer a block of bits with an image to a specific place on the screen), fb_copyarea (similar transfer, but a block of bits is taken not from the external buffer but from video memory), etc. In addition, there will be structures with a description of the screen resolution, the color bitness and how the color components are located in this “bit depth”.
The first thing to realize is that we have a somewhat unusual situation when compared with PC video cards (although it’s quite usual for embedded devices) - our device doesn’t have any video memory as such, which we could access - more precisely, it has memory the same GRAM, hidden in the depths of the display, but we only have access to it through the “window” of 16 bits wide. The memory on board is also not enough to store the entire frame buffer there.
Fortunately, Linux has a special approach for this - functions with the prefix " sys_ " are already written for us, for example, " sys_imageblit ", which implement the functionality required by the framebuffer, working with the system memory as a frame buffer.
That is, if the frame buffer is located in the video card and we have hardware support for such operations, we simply kick our piece of hardware in the callbacks, giving the command “execute block bit transfer” or “copy the video memory area”.
If we don’t have this, we allocate memory in the kernel that is equal to our frame buffer, and in callbacks we call these same functions with the prefix " sys_ " that perform the necessary operations in RAM.
Thus, you can get a fully working framebuffer, which will not interact with iron at all - such a driver already exists and it is called vfb, virtual framebuffer. Its source code is in drivers / video / vfb.c.
If we add a periodic data transfer to a real device to such a driver, we will get a real framebuffer driver. But before we do this, let's build a little of our system and practice on the virtual driver, vfb .
I was busy with this part for a long time, mainly because I first wrote my driver and then tried to understand why I only have a black screen - I sinned on my own errors in the code. Then I guessed to install the VFB driver instead, having read the contents of the memory of which I saw the same black screen. Then I finally realized that the matter is not in the driver, but in the fact that the kernel itself refuses to display information on it, after which the problem was resolved rather quickly. But first things first.
')
Now we will do all three points regarding the virtual framebuffer driver, and when we get the picture in memory, we will proceed to write our own. fbcon and the framebuffer driver can be either statically biased, or both as plug-ins, or one static one, the other dynamically - this will not cause problems, fbcon will catch the framebuffer as soon as it sees it.
However, when working with vfb, there is one subtlety - in order to activate it, you need to transfer the vfb_enable = 1 parameter to the module, or start the kernel with the " video = vfb " parameter. It will be easier to work with the module, so we will limit ourselves to it. fbcon is built into the core.
The last thing we need to do before rebuilding is to go to make menuconfig and include the fbset utility in the capabilities provided by busybox , which will allow us to set the parameters of our framebuffer. It is located in the menu Base System - Busybox - Linux SyStem Utilities
Now you can rebuild the kernel. In build_dir / target-mips_r2_uClibc-0.9.33.2 / linux-ar71xx_generic / linux-3.6.9 / dri
vers / video /
we take away that with the .ko extension
Contrary to expectations, it will not be there alone; turning on a driver that uses functions with the prefix " sys_ " activates the assembly of several modules in which these same functions lie. Interestingly, in principle, nothing prevents them from being statically added to the kernel, leaving the driver as a plug-in, but I could not do this from the menu, I had to write the corresponding patch to the Kconfig file. But we will do this later, but for now just rewrite the router with a new firmware and transfer all modules to it.
After we go through SSH to the router and go / etc. Open the inittab file and see there is something like this:
The last line just says that you need to run the software for the login (in this case, like all system binaries - part of the busybox) on ttyATH0, the serial port. It is indicated (askfirst) that to activate this console, you will first need to press enter.
Add one more line:
Let's see if the kernel parameters are correctly specified through
and reboot the router.
Now let's take turns insmodim everything except the vfb driver, and at the very end we write
After that, we should see words like these in the dmesg: Console: switching to color frame buffer device 53x21
Console sizes will vary depending on the font selected. Set the framebuffer options, more similar to the parameters of our display:
This will set the visible and virtual resolution to 320x240 (most often they are the same, but in principle, you can set the virtual resolution to more visible, getting the frame buffer to more output and use this for double buffering), and the color depth to 16 bits.
fbcon should respond to this by changing its permission and sending a message to dmesg, but if this didn’t happen, disconnect the console from the framebuffer and reconnect:
This is a useful pair of commands that will come in handy for us many times - without this, do not unload the framebuffer driver, he will be busy with the console.
It doesn’t hurt to connect a keyboard to the router - you can blindly enter the clear and dmesg commands to make sure that there is something on the virtual display.
After we get a "screenshot" with the command
And download it to the desktop. There we open via GIMP or any other software that can load raw graphic data in the RGB565 format - set the image size to 320x240, don't forget about the bit depth, and get a picture like this (the message about opening and closing / dev / fb0 gives my driver, t. I didn’t take a screenshot from a virtual framebuffer. A virtual one is silent about such cases):
Notice the beautiful, “hacker” green color of the console? In fact, this tells us about an error, more precisely, about one particular feature to be reckoned with. But we'll talk about that later. Before moving on to the main step — writing your own framebuffer driver — let's compare the available console fonts. For this, I prepared six photos, three for the console and for Midnight Commander with 4x4, 6x11 and 8x8 fonts. In my opinion, the most convenient is 6x11:
4x4
6x11
8x8
4x4
6x11
8x8
For a start - about the approach. An obvious and not very good solution would be to periodically update the entire screen - one could start a timer that would throw the entire contents of the framebuffer via USB with the commands already familiar to us from the previous article. However, there is a far more correct solution, the so-called deferred io.
Its essence is simple: we only need to specify the callback function, set the time interval and register this deferred io for our framebuffer. When registering, virtual memory will be configured so that accessing the frame buffer memory will cause an exception that puts our callback in the queue for processing at the interval we set. In this case, the write to memory will not be interrupted! And when the callback is called, a list of pages that have been changed will be transferred to it. Isn't it very convenient? Yuzerspeys can safely write to the video memory without thinking about anything and without interrupting, while our callback will periodically twitch with a list of memory pages that have been changed - we will only need to throw on the device, not the entire buffer.
Since the release of the framebuffer is not as simple as it seems at first glance (the USB device may already be absent, but the clients themselves will not let go of the framebuffer yet and it’s impossible to clean the memory at this moment) Todo and vow to promise to implement the correct cleaning and disabling the device a little bit later, but for now let's write everything else to finally see the fruit of our actions. Normal cleaning, along with finishing the video card firmware (which will raise the FPS at least twice), we will definitely consider in the next article.
Let's start with a simple one - since we will receive a list of pages to which we have been recorded, the easiest way is to save in advance the coordinates on the screen corresponding to the top of the page, as well as a pointer to the corresponding memory area. Therefore, we will create a structure with these fields, without forgetting to add an atomic flag indicating whether this page needs an update. This is necessary because internal framebuffer operations that are performed through functions with the " sys_ " prefix do not call our handler deferred io , so we will need to manually calculate which pages were accessed and mark them as updateable.
Here everything is transparent, the only thing that - keep the length, because The last page may be incomplete - we don’t need to send extra data to the display.
Let's declare several defines related to the size of the display - the number of pixels per page, the number of pages in the framebuffer, etc.
PAGE_SIZE is defined for us in the kernel source.
We declare the required callbacks of framebuffer operations and the callback handler of our deferred io .
Next comes a structure with unchangeable display information.
Everything is also simple - we set the string ID of our display, the type of pixels - others do not interest us, we have the most common bitmap, the color is truecolor, at least close to it and certainly not monochrome or direct color.
Next comes a more interesting structure with (potentially) variable information. But since we will not implement the possibility of changing the resolution, for us it will be as constant as in the previous structure.
Here we see the visible and virtual resolutions already familiar to fbset , the physical dimensions of the display in millimeters (you can set, in principle, any, I set the same as its resolution), the image bit depth, and - important structures - descriptors of how color components are in bytes. In them, the first value is the offset, the second is the length in bits and the third is the flag, the "significant bit to the right."
The last to declare is the deferred I / O structure, deferred io .
The choice of the period value, the .delay field, the task, for the most part, is empirical - if you choose too small a value, the update will occur less frequently than the hardware allows, if you choose too large - the queue of pending jobs will be clogged. Our display is currently more than slow, and this is determined entirely by USB, and not by output to the screen. In the implementation of the first article, full screen redrawing is possible with a frequency of no higher than 3.6 FPS. But do not despair - firstly, we will not always redraw the entire screen, and secondly, in the next article, I will show how to squeeze the most out of the hardware we have, so the FPS will jump to ~ 8 - taking into account the incomplete redrawing we get a completely usable device, as in the video at the beginning of the article. These 9 FPS, by the way, are the physical limit that we can get on Full Speed ​​USB when transferring raw video data (without compression). This becomes obvious if we recall the FS USB transfer rate limit - 12 Mbit / s. Our frame occupies 320 * 240 * 16 = 1 228 800 bits. Thus, the ideal spherical frame rate in vacuum will be no higher than 9.8 FPS. From here we will throw away our headers, losses in our driver, losses in the driver of the host controller, losses in the driver on STM, and we will get a real limit of 8-9 FPS, which is quite good to achieve. But we will do this in the next article, and now we remember that our frequency is about 3.5 FPS, and, according to my measurements, the period is approximately twice as long, that is, 6-7 Hz, turned out to be optimal. This is what we ask, using the predefined definition in the HZ kernel source. By the way, you should pay attention - despite the name, this macro does not determine the frequency of 1 Hz, but the corresponding period (in nuclear quanta), therefore, to obtain a frequency of 6 Hz, it should not be multiplied, but divided by six.
Finally, fill the structure with callbacks of the framebuffer operations.
We'll immediately write a callback from those that are " sys_ " for reading , we have nothing to do there. In all the rest we will need to still mark the corresponding pages for the update, so we indicate our own.
The structure-descriptor of our device has not changed much since the time of the previous article, we will describe it.
An array of our video memory page handles has been added to it. Generally, it is not recommended to statically place large amounts of data in the kernel, but the structure itself is small, and by the number there are only 38, therefore, in order not to fool around with extra pointers, we leave this array static, 760 bytes or so the core will master it.
The last field, pseudo_palette, is the space for the pseudo-palette that fbcon requires. It is filled in .fb_setcolreg callback , without which fbcon refuses to work. In all the firewood that I have seen, this callback looks like copy-paste from the example frame file from the kernel sources, so we will not reinvent the wheel either, especially since nobody seems to be using this fbcon . Let's start with it.
This, as I said, is the standard code for such a callback, including the macro CNVT_TOHW, which is used to get the values ​​of the color components. It is also dragged from the driver to the driver - it is not entirely clear why it will not be included in the main header file fb.h in the end .
The task of this callback is to fill in a 16-color pseudo-palette, to which the already mentioned console driver will apply.
Now we will declare a small function, to which we will transfer, in essence, a rectangle, in the region of which the video memory was affected. The function will calculate which pages of video memory are affected by this rectangle, will give them the “required update” flag and will schedule the execution of the same callback, which is called in the case of deferred io . After that, all callback operations will be reduced to calling the " sys_ " functions and the function we wrote, which we call touch, by analogy with the Linux command.
The code, I think, is quite understandable - we simply calculate which pages we touched on the basis of the resolution, we always round upwards, or rather, we just take with a margin, one page at both ends.
Now we will describe all the other callbacks - they become very simple:
Now we finally describe our callback from deferred io , which will send information to the display. It will largely coincide with the .write callback from the previous article. We will write to the display through the pages, not forgetting to attribute, as in the previous article, the required heading to them. Thanks to our videopage structure, the x, y coordinates and lengths are already calculated, so all you need to do is just stuff it into the buffer and drop it over USB.
Since we can get into this callback honestly (via deferrd io ), but we can get it on our own initiative (by planning to perform it in one of the callback operations, calling display_touch), we will simply run through all the pages transmitted to us, if any. mark them up for update.
If we got here not through delayed I / O, the list will simply be empty.
After that, we simply go through all the pages, atomically checking the need for an update and performing this update by synchronous sending via USB. In the next article, when we finish the driver to a normal state, we will replace the synchronous parcel with a more correct mechanism, called USB Request Block, or URB. It will allow the USB host to send a request to send data and immediately return to further processing. And the fact that the URB flew (or did not fly) to the recipient will tell us in the interrupt. This will allow us to squeeze a little more FPS from our system (without, however, exceeding the theoretical limit, which I mentioned above).
We have just a little bit left - since we decided to do badly and not clean up after ourselves, all that remains is to initialize everything in the Probe callback.
Here, we first allocate memory for our device descriptor, then for the structure of the framebuffer descriptor, and only then - for the framebuffer itself. Please note - select through vmalloc . The difference from kmalloc is that vmalloc can reconfigure the virtual memory page tables, and “collect in pieces” the buffer we requested. That is, for us it will look like a single block of memory, but physically it may consist of pages that are not even close together. kmalloc returns memory, which physically is a single unit. Since we are requesting a sufficiently large chunk, vmalloc will be a good practice.
Everything, we compile, insmodim, and if everything is done correctly we see the console on the display!
In this article, we finally took this important step - from a custom device driver that was not used by the system, we went into the framework's embrace, which allowed us to tell all applications and drivers that we have a real display. Yes, we did a little badly, not clearing the resources and not disconnecting the device, so pulling out the USB and reinstalling it on a working device is not recommended yet.
But we will definitely fix it in the next article. What do we have to do?
Below is another video with an old quest (no less loved by me than Gobliins), a few photos of a small graphical shell that you can easily run on your device and the console browser Elinks.
The first Kyrandia
Habr in the console browser (font VGA8x8)
The shell Gmenu2X
Built-in file browser
Settings
On this I have everything. Successful implementation and until the next article!
In case you missed the previous parts, here are the links:
1 - Minicomputer from a router with OpenWRT: we are developing a USB video card
2 - Minicomputer from router with OpenWRT: write USB class-driver for Linux
So, get down to work.
Introduction
Iron has not changed again (although we will definitely change something in the firmware in the next article), so we will start with an overview of what we have to do.
Despite the fact that our driver from the previous article already knows how to display graphics, it is very difficult to fully use it (for example, to display the console or launch a graphical application). The fact is that such applications require a well-defined, standardized interface, the so-called framebuffer. When registering a framebuffer in the system, we will need to fill in several specialized structures-descriptors, this time much more specific than the abstract “file operations” that we filled out last time. Operations will also be there, but there will be special callbacks, such as fb_imageblit (called when someone wants to transfer a block of bits with an image to a specific place on the screen), fb_copyarea (similar transfer, but a block of bits is taken not from the external buffer but from video memory), etc. In addition, there will be structures with a description of the screen resolution, the color bitness and how the color components are located in this “bit depth”.
The first thing to realize is that we have a somewhat unusual situation when compared with PC video cards (although it’s quite usual for embedded devices) - our device doesn’t have any video memory as such, which we could access - more precisely, it has memory the same GRAM, hidden in the depths of the display, but we only have access to it through the “window” of 16 bits wide. The memory on board is also not enough to store the entire frame buffer there.
Fortunately, Linux has a special approach for this - functions with the prefix " sys_ " are already written for us, for example, " sys_imageblit ", which implement the functionality required by the framebuffer, working with the system memory as a frame buffer.
That is, if the frame buffer is located in the video card and we have hardware support for such operations, we simply kick our piece of hardware in the callbacks, giving the command “execute block bit transfer” or “copy the video memory area”.
If we don’t have this, we allocate memory in the kernel that is equal to our frame buffer, and in callbacks we call these same functions with the prefix " sys_ " that perform the necessary operations in RAM.
Thus, you can get a fully working framebuffer, which will not interact with iron at all - such a driver already exists and it is called vfb, virtual framebuffer. Its source code is in drivers / video / vfb.c.
If we add a periodic data transfer to a real device to such a driver, we will get a real framebuffer driver. But before we do this, let's build a little of our system and practice on the virtual driver, vfb .
Enable graphics support in the kernel.
I was busy with this part for a long time, mainly because I first wrote my driver and then tried to understand why I only have a black screen - I sinned on my own errors in the code. Then I guessed to install the VFB driver instead, having read the contents of the memory of which I saw the same black screen. Then I finally realized that the matter is not in the driver, but in the fact that the kernel itself refuses to display information on it, after which the problem was resolved rather quickly. But first things first.
- In order for us to see the console output in the framebuffer's memory (well, on the screen if it is real and not virtual) two drivers are needed - this is, in fact, the driver of the framebuffer itself, which will create the / dev / fb [x] device and the console driver running on top of it is the fbcon driver
- In the core, respectively, support for framebuffers , support for virtual terminals (an abstraction combining an output device + input device, a display and a keyboard), support for displaying the system console on such terminals (yes, this can also be disabled, then the system console output only to physically existing character devices like com-ports), the fbcon driver itself , as well as some of the available fonts available to it .
- The very point that I missed at the beginning when I couldn’t understand why nothing is output - you need to tell the kernel that you need to output the contents of the system console to that / dev / tty [x] that fbcon captured !
The fact is that the fbcon driver tries to capture the first available / dev / tty [x] , for example, tty0 . But the kernel does not bring anything there, it is an abstraction that is not tied to anything, because neither the application allowing to log in to the system, nor the output of the system console are running on it.
In order to solve this problem, we must first tell the kernel that we want to see the system console on / dev / tty0 (however, this is optional, if suddenly someone does not want to see the boot process and system output, then this item can be omitted ), and secondly, inform Inita that there you need to run the software for the login
')
Now we will do all three points regarding the virtual framebuffer driver, and when we get the picture in memory, we will proceed to write our own. fbcon and the framebuffer driver can be either statically biased, or both as plug-ins, or one static one, the other dynamically - this will not cause problems, fbcon will catch the framebuffer as soon as it sees it.
However, when working with vfb, there is one subtlety - in order to activate it, you need to transfer the vfb_enable = 1 parameter to the module, or start the kernel with the " video = vfb " parameter. It will be easier to work with the module, so we will limit ourselves to it. fbcon is built into the core.
- Run make kernel_menuconfig and go to the item Device Drivers
- We turn on Graphics Support - Support for frame buffer devices , after which we get a list of the drivers themselves, in which we select the Virtual framebuffer .
- Go back up, go to Character devices and turn it on
Virtual terminal
Enable character translations in console
Support for console on virtual terminal
Support for binding and unbinding console drivers
Thanks to the last two options, we will be able to bring the system console to a virtual terminal, and then, if you wish, unpin it from the framebuffer driver, which will allow it to be unloaded from memory. - Go back to Graphics Support , go to the Console display driver support menu that has become available and enable Framebuffer Console support , then activate the Select compiled-in fonts item and select a font there - say, VGA 8x8 font
- We exit to the main menu and pay attention to the Kernel Hacking item - if you go in there, you can find, near the end of the list, the item containing the kernel boot options. In general, they are transferred to the kernel by a boot-loder, but you can add a parameter string using this item, or redefine it altogether. We will not redefine, but we will add - we will add, because The bootloader sends the console = ttyATH0 parameter to it, which means outputting the system console to the serial port. Unfortunately, we cannot do this right here - this parameter will be overridden when applying platform-specific patches, so we’ll go there. Do not touch anything here, save the config and exit.
- Go to where, as we remember, are stored platform-specific files and patches - target / linux / ar71xx / . Go to generic and open the file config-default . In it, we see a single line, the same parameter that was seen in the kernel configuration:
CONFIG_CMDLINE = "rootfstype = squashfs, jffs2 noinitrd"
append to the end console = tty0 and fbcon = font: <font name> , setting the font name to one of those selected in the kernel configuration. We get something like
CONFIG_CMDLINE = "rootfstype = squashfs, jffs2 noinitrd console = tty0 fbcon = font: ProFont6x11"
The last thing we need to do before rebuilding is to go to make menuconfig and include the fbset utility in the capabilities provided by busybox , which will allow us to set the parameters of our framebuffer. It is located in the menu Base System - Busybox - Linux SyStem Utilities
Now you can rebuild the kernel. In build_dir / target-mips_r2_uClibc-0.9.33.2 / linux-ar71xx_generic / linux-3.6.9 / dri
vers / video /
we take away that with the .ko extension
Contrary to expectations, it will not be there alone; turning on a driver that uses functions with the prefix " sys_ " activates the assembly of several modules in which these same functions lie. Interestingly, in principle, nothing prevents them from being statically added to the kernel, leaving the driver as a plug-in, but I could not do this from the menu, I had to write the corresponding patch to the Kconfig file. But we will do this later, but for now just rewrite the router with a new firmware and transfer all modules to it.
After we go through SSH to the router and go / etc. Open the inittab file and see there is something like this:
::sysinit:/etc/init.d/rcS S boot ::shutdown:/etc/init.d/rcS K shutdown ttyATH0::askfirst:/bin/ash --login The last line just says that you need to run the software for the login (in this case, like all system binaries - part of the busybox) on ttyATH0, the serial port. It is indicated (askfirst) that to activate this console, you will first need to press enter.
Add one more line:
tty0::respawn:/bin/ash --login Let's see if the kernel parameters are correctly specified through
cat /proc/cmdline and reboot the router.
Now let's take turns insmodim everything except the vfb driver, and at the very end we write
insmod vfb.ko vfb_enable=1 After that, we should see words like these in the dmesg: Console: switching to color frame buffer device 53x21
Console sizes will vary depending on the font selected. Set the framebuffer options, more similar to the parameters of our display:
fbset -g 320 240 320 240 16 This will set the visible and virtual resolution to 320x240 (most often they are the same, but in principle, you can set the virtual resolution to more visible, getting the frame buffer to more output and use this for double buffering), and the color depth to 16 bits.
fbcon should respond to this by changing its permission and sending a message to dmesg, but if this didn’t happen, disconnect the console from the framebuffer and reconnect:
echo "0" > /sys/class/vtconsole/vtcon1/bind echo "1" > /sys/class/vtconsole/vtcon1/bind This is a useful pair of commands that will come in handy for us many times - without this, do not unload the framebuffer driver, he will be busy with the console.
It doesn’t hurt to connect a keyboard to the router - you can blindly enter the clear and dmesg commands to make sure that there is something on the virtual display.
After we get a "screenshot" with the command
cat /dev/fb0 > scrn.raw And download it to the desktop. There we open via GIMP or any other software that can load raw graphic data in the RGB565 format - set the image size to 320x240, don't forget about the bit depth, and get a picture like this (the message about opening and closing / dev / fb0 gives my driver, t. I didn’t take a screenshot from a virtual framebuffer. A virtual one is silent about such cases):
Notice the beautiful, “hacker” green color of the console? In fact, this tells us about an error, more precisely, about one particular feature to be reckoned with. But we'll talk about that later. Before moving on to the main step — writing your own framebuffer driver — let's compare the available console fonts. For this, I prepared six photos, three for the console and for Midnight Commander with 4x4, 6x11 and 8x8 fonts. In my opinion, the most convenient is 6x11:
4x4
6x11
8x8
4x4
6x11
8x8
We write the framebuffer driver
For a start - about the approach. An obvious and not very good solution would be to periodically update the entire screen - one could start a timer that would throw the entire contents of the framebuffer via USB with the commands already familiar to us from the previous article. However, there is a far more correct solution, the so-called deferred io.
Its essence is simple: we only need to specify the callback function, set the time interval and register this deferred io for our framebuffer. When registering, virtual memory will be configured so that accessing the frame buffer memory will cause an exception that puts our callback in the queue for processing at the interval we set. In this case, the write to memory will not be interrupted! And when the callback is called, a list of pages that have been changed will be transferred to it. Isn't it very convenient? Yuzerspeys can safely write to the video memory without thinking about anything and without interrupting, while our callback will periodically twitch with a list of memory pages that have been changed - we will only need to throw on the device, not the entire buffer.
Since the release of the framebuffer is not as simple as it seems at first glance (the USB device may already be absent, but the clients themselves will not let go of the framebuffer yet and it’s impossible to clean the memory at this moment) Todo and vow to promise to implement the correct cleaning and disabling the device a little bit later, but for now let's write everything else to finally see the fruit of our actions. Normal cleaning, along with finishing the video card firmware (which will raise the FPS at least twice), we will definitely consider in the next article.
Let's start with a simple one - since we will receive a list of pages to which we have been recorded, the easiest way is to save in advance the coordinates on the screen corresponding to the top of the page, as well as a pointer to the corresponding memory area. Therefore, we will create a structure with these fields, without forgetting to add an atomic flag indicating whether this page needs an update. This is necessary because internal framebuffer operations that are performed through functions with the " sys_ " prefix do not call our handler deferred io , so we will need to manually calculate which pages were accessed and mark them as updateable.
Memory page structure
struct videopage { int x, y; unsigned char *mem; unsigned long length; atomic_t toUpdate; }; Here everything is transparent, the only thing that - keep the length, because The last page may be incomplete - we don’t need to send extra data to the display.
Let's declare several defines related to the size of the display - the number of pixels per page, the number of pages in the framebuffer, etc.
PAGE_SIZE is defined for us in the kernel source.
Auxiliary Defines
#define WIDTH 320 #define HEIGHT 240 #define BYTE_DEPTH 2 #define FB_SIZE WIDTH*HEIGHT*BYTE_DEPTH #define FP_PAGE_COUNT PAGE_ALIGN(FB_SIZE)/PAGE_SIZE #define PIXELS_IN_PAGE PAGE_SIZE/BYTE_DEPTH We declare the required callbacks of framebuffer operations and the callback handler of our deferred io .
Kolbeki
static void display_fillrect(struct fb_info *p, const struct fb_fillrect *rect); static void display_imageblit(struct fb_info *p, const struct fb_image *image); static void display_copyarea(struct fb_info *p, const struct fb_copyarea *area); static ssize_t display_write(struct fb_info *p, const char __user *buf, size_t count, loff_t *ppos); static int display_setcolreg(unsigned regno, unsigned red, unsigned green, unsigned blue, unsigned transp, struct fb_info *info); //--------------- static void display_update(struct fb_info *info, struct list_head *pagelist); Next comes a structure with unchangeable display information.
Fb_fix_screeninfo structure
static struct fb_fix_screeninfo fixed_info = { .id = "STM32LCD", .type = FB_TYPE_PACKED_PIXELS, .visual = FB_VISUAL_TRUECOLOR, .accel = FB_ACCEL_NONE, .line_length = WIDTH * BYTE_DEPTH, }; Everything is also simple - we set the string ID of our display, the type of pixels - others do not interest us, we have the most common bitmap, the color is truecolor, at least close to it and certainly not monochrome or direct color.
Next comes a more interesting structure with (potentially) variable information. But since we will not implement the possibility of changing the resolution, for us it will be as constant as in the previous structure.
Fb_var_screeninfo structure
static struct fb_var_screeninfo var_info = { .xres = WIDTH, .yres = HEIGHT, .xres_virtual = WIDTH, .yres_virtual = HEIGHT, .width = WIDTH, .height = HEIGHT, .bits_per_pixel = 16, .red = {11, 5, 0}, .green = {5, 6, 0}, .blue = {0, 5, 0}, .activate = FB_ACTIVATE_NOW, .vmode = FB_VMODE_NONINTERLACED, }; Here we see the visible and virtual resolutions already familiar to fbset , the physical dimensions of the display in millimeters (you can set, in principle, any, I set the same as its resolution), the image bit depth, and - important structures - descriptors of how color components are in bytes. In them, the first value is the offset, the second is the length in bits and the third is the flag, the "significant bit to the right."
The last to declare is the deferred I / O structure, deferred io .
Fb_deferred_io structure
static struct fb_deferred_io display_defio = { .delay = HZ/6, .deferred_io = &display_update, }; The choice of the period value, the .delay field, the task, for the most part, is empirical - if you choose too small a value, the update will occur less frequently than the hardware allows, if you choose too large - the queue of pending jobs will be clogged. Our display is currently more than slow, and this is determined entirely by USB, and not by output to the screen. In the implementation of the first article, full screen redrawing is possible with a frequency of no higher than 3.6 FPS. But do not despair - firstly, we will not always redraw the entire screen, and secondly, in the next article, I will show how to squeeze the most out of the hardware we have, so the FPS will jump to ~ 8 - taking into account the incomplete redrawing we get a completely usable device, as in the video at the beginning of the article. These 9 FPS, by the way, are the physical limit that we can get on Full Speed ​​USB when transferring raw video data (without compression). This becomes obvious if we recall the FS USB transfer rate limit - 12 Mbit / s. Our frame occupies 320 * 240 * 16 = 1 228 800 bits. Thus, the ideal spherical frame rate in vacuum will be no higher than 9.8 FPS. From here we will throw away our headers, losses in our driver, losses in the driver of the host controller, losses in the driver on STM, and we will get a real limit of 8-9 FPS, which is quite good to achieve. But we will do this in the next article, and now we remember that our frequency is about 3.5 FPS, and, according to my measurements, the period is approximately twice as long, that is, 6-7 Hz, turned out to be optimal. This is what we ask, using the predefined definition in the HZ kernel source. By the way, you should pay attention - despite the name, this macro does not determine the frequency of 1 Hz, but the corresponding period (in nuclear quanta), therefore, to obtain a frequency of 6 Hz, it should not be multiplied, but divided by six.
Finally, fill the structure with callbacks of the framebuffer operations.
Fb_ops structure
static struct fb_ops display_fbops = { .owner = THIS_MODULE, .fb_read = fb_sys_read, .fb_write = display_write, .fb_fillrect = display_fillrect, .fb_copyarea = display_copyarea, .fb_imageblit = display_imageblit, .fb_setcolreg = display_setcolreg, }; We'll immediately write a callback from those that are " sys_ " for reading , we have nothing to do there. In all the rest we will need to still mark the corresponding pages for the update, so we indicate our own.
The structure-descriptor of our device has not changed much since the time of the previous article, we will describe it.
Usblcd handle structure
struct usblcd { struct usb_device *udev; struct usb_interface *interface; struct device *gdev; struct fb_info *info; struct usb_endpoint_descriptor *bulk_out_ep; unsigned int bulk_out_packet_size; struct videopage videopages[FP_PAGE_COUNT]; unsigned long pseudo_palette[17]; }; An array of our video memory page handles has been added to it. Generally, it is not recommended to statically place large amounts of data in the kernel, but the structure itself is small, and by the number there are only 38, therefore, in order not to fool around with extra pointers, we leave this array static, 760 bytes or so the core will master it.
The last field, pseudo_palette, is the space for the pseudo-palette that fbcon requires. It is filled in .fb_setcolreg callback , without which fbcon refuses to work. In all the firewood that I have seen, this callback looks like copy-paste from the example frame file from the kernel sources, so we will not reinvent the wheel either, especially since nobody seems to be using this fbcon . Let's start with it.
Callback display_setcolreg
#define CNVT_TOHW(val,width) ((((val)<<(width))+0x7FFF-(val))>>16) static int display_setcolreg(unsigned regno, unsigned red, unsigned green, unsigned blue, unsigned transp, struct fb_info *info) { int ret = 1; if (info->var.grayscale) red = green = blue = (19595 * red + 38470 * green + 7471 * blue) >> 16; switch (info->fix.visual) { case FB_VISUAL_TRUECOLOR: if (regno < 16) { u32 *pal = info->pseudo_palette; u32 value; red = CNVT_TOHW(red, info->var.red.length); green = CNVT_TOHW(green, info->var.green.length); blue = CNVT_TOHW(blue, info->var.blue.length); transp = CNVT_TOHW(transp, info->var.transp.length); value = (red << info->var.red.offset) | (green << info->var.green.offset) | (blue << info->var.blue.offset) | (transp << info->var.transp.offset); pal[regno] = value; ret = 0; } break; case FB_VISUAL_STATIC_PSEUDOCOLOR: case FB_VISUAL_PSEUDOCOLOR: break; } return ret; } This, as I said, is the standard code for such a callback, including the macro CNVT_TOHW, which is used to get the values ​​of the color components. It is also dragged from the driver to the driver - it is not entirely clear why it will not be included in the main header file fb.h in the end .
The task of this callback is to fill in a 16-color pseudo-palette, to which the already mentioned console driver will apply.
Now we will declare a small function, to which we will transfer, in essence, a rectangle, in the region of which the video memory was affected. The function will calculate which pages of video memory are affected by this rectangle, will give them the “required update” flag and will schedule the execution of the same callback, which is called in the case of deferred io . After that, all callback operations will be reduced to calling the " sys_ " functions and the function we wrote, which we call touch, by analogy with the Linux command.
Display_touch function
static void display_touch(struct fb_info *info, int x, int y, int w, int h) { int firstPage; int lastPage; int i; struct usblcd *dev=info->par; firstPage=((y*WIDTH)+x)*BYTE_DEPTH/PAGE_SIZE-1; lastPage=(((y+h)*WIDTH)+x+w)*BYTE_DEPTH/PAGE_SIZE+1; if(firstPage<0) firstPage=0; if(lastPage>FP_PAGE_COUNT) lastPage=FP_PAGE_COUNT; for(i=firstPage;i<lastPage;i++) atomic_dec(&dev->videopages[i].toUpdate); schedule_delayed_work(&info->deferred_work, info->fbdefio->delay); } The code, I think, is quite understandable - we simply calculate which pages we touched on the basis of the resolution, we always round upwards, or rather, we just take with a margin, one page at both ends.
Now we will describe all the other callbacks - they become very simple:
All callback operations
static void display_fillrect(struct fb_info *p, const struct fb_fillrect *rect) { sys_fillrect(p, rect); display_touch(p, rect->dx, rect->dy, rect->width, rect->height); } static void display_imageblit(struct fb_info *p, const struct fb_image *image) { sys_imageblit(p, image); display_touch(p, image->dx, image->dy, image->width, image->height); } static void display_copyarea(struct fb_info *p, const struct fb_copyarea *area) { sys_copyarea(p, area); display_touch(p, area->dx, area->dy, area->width, area->height); } static ssize_t display_write(struct fb_info *p, const char __user *buf, size_t count, loff_t *ppos) { int retval; retval=fb_sys_write(p, buf, count, ppos); display_touch(p, 0, 0, p->var.xres, p->var.yres); return retval; } Now we finally describe our callback from deferred io , which will send information to the display. It will largely coincide with the .write callback from the previous article. We will write to the display through the pages, not forgetting to attribute, as in the previous article, the required heading to them. Thanks to our videopage structure, the x, y coordinates and lengths are already calculated, so all you need to do is just stuff it into the buffer and drop it over USB.
Callback delayed I / O
unsigned char videobuffer[PAGE_SIZE+8]; static void display_update(struct fb_info *info, struct list_head *pagelist) { struct usblcd* dev = info->par; int retval; struct page *page; int i; int usbSent=0; list_for_each_entry(page, pagelist, lru) { atomic_dec(&dev->videopages[page->index].toUpdate); } for (i=0; i<FP_PAGE_COUNT; i++) { if(atomic_inc_and_test(&dev->videopages[i].toUpdate)) atomic_dec(&dev->videopages[i].toUpdate); else { *(unsigned short*)(videobuffer)=cpu_to_le16(dev->videopages[i].x); *(unsigned short*)(videobuffer+2)=cpu_to_le16(dev->videopages[i].y); *(unsigned long*)(videobuffer+4)=cpu_to_le32(dev->videopages[i].length>>1); memcpy(videobuffer+8,dev->videopages[i].mem,dev->videopages[i].length); retval = usb_bulk_msg(dev->udev, usb_sndbulkpipe(dev->udev, 1),videobuffer, dev->videopages[i].length+8, &usbSent, HZ); if (retval) printk(KERN_INFO "usblcd: sending error!\n"); } } } Since we can get into this callback honestly (via deferrd io ), but we can get it on our own initiative (by planning to perform it in one of the callback operations, calling display_touch), we will simply run through all the pages transmitted to us, if any. mark them up for update.
If we got here not through delayed I / O, the list will simply be empty.
After that, we simply go through all the pages, atomically checking the need for an update and performing this update by synchronous sending via USB. In the next article, when we finish the driver to a normal state, we will replace the synchronous parcel with a more correct mechanism, called USB Request Block, or URB. It will allow the USB host to send a request to send data and immediately return to further processing. And the fact that the URB flew (or did not fly) to the recipient will tell us in the interrupt. This will allow us to squeeze a little more FPS from our system (without, however, exceeding the theoretical limit, which I mentioned above).
We have just a little bit left - since we decided to do badly and not clean up after ourselves, all that remains is to initialize everything in the Probe callback.
Callback Probe
static int LCDProbe(struct usb_interface *interface, const struct usb_device_id *id) { struct usblcd *dev; struct usb_host_interface *iface_desc; struct usb_endpoint_descriptor *endpoint; unsigned char *videomemory; int retval = -ENODEV; int i; dev_info(&interface->dev, "USB STM32-based LCD module connected"); dev = kzalloc(sizeof(*dev), GFP_KERNEL); if (!dev) { dev_err(&interface->dev, "Can not allocate memory for device descriptor\n"); retval = -ENOMEM; goto exit; } dev->udev=interface_to_usbdev(interface); dev->interface = interface; iface_desc = interface->cur_altsetting; for (i = 0; i < iface_desc->desc.bNumEndpoints; ++i) { endpoint = &iface_desc->endpoint[i].desc; if(usb_endpoint_is_bulk_out(endpoint)) { dev->bulk_out_ep=endpoint; dev->bulk_out_packet_size = le16_to_cpu(endpoint->wMaxPacketSize); break; } } if(!dev->bulk_out_ep) { dev_err(&interface->dev, "Can not find bulk-out endpoint!\n"); retval = -EIO; goto error_dev; } dev->gdev = &dev->udev->dev; dev->info = framebuffer_alloc(0, dev->gdev); dev->info->par = dev; dev->info->dev = dev->gdev; if (!dev->info) { dev_err(&interface->dev, "Can not allocate memory for fb_info structure\n"); retval = -ENOMEM; goto error_dev; } dev->info->fix = fixed_info; dev->info->var = var_info; dev->info->fix.smem_len=FP_PAGE_COUNT*PAGE_SIZE; dev_info(&interface->dev, "Allocating framebuffer: %d bytes [%lu pages]\n",dev->info->fix.smem_len,FP_PAGE_COUNT); videomemory=vmalloc(dev->info->fix.smem_len); if (!videomemory) { dev_err(&interface->dev, "Can not allocate memory for framebuffer\n"); retval = -ENOMEM; goto error_dev; } dev->info->fix.smem_start =(unsigned long)(videomemory); dev->info->fbops = &display_fbops; dev->info->flags = FBINFO_FLAG_DEFAULT|FBINFO_VIRTFB; dev->info->screen_base = videomemory; memset((void *)dev->info->fix.smem_start, 0, dev->info->fix.smem_len); for(i=0;i<FP_PAGE_COUNT;i++) { dev->videopages[i].mem=(void *)(dev->info->fix.smem_start+PAGE_SIZE*i); dev->videopages[i].length=PAGE_SIZE; atomic_set(&dev->videopages[i].toUpdate,-1); dev->videopages[i].y=(((unsigned long)(PAGE_SIZE*i)>>1)/WIDTH); dev->videopages[i].x=((unsigned long)(PAGE_SIZE*i)>>1)-dev->videopages[i].y*WIDTH; } dev->videopages[FP_PAGE_COUNT-1].length=FB_SIZE-(FP_PAGE_COUNT-1)*PAGE_SIZE; dev->info->pseudo_palette = &dev->pseudo_palette; dev->info->fbdefio=&display_defio; fb_deferred_io_init(dev->info); dev_info(&interface->dev, "info.fix.smem_start=%lu\ninfo.fix.smem_len=%d\ninfo.screen_size=%lu\n",dev->info->fix.smem_start,dev->info->fix.smem_len,dev->info->screen_size); usb_set_intfdata(interface, dev); retval = register_framebuffer(dev->info); if (retval < 0) { dev_err(dev->gdev,"Unable to register_frambuffer\n"); goto error_buff; } return 0; error_buff: vfree(videomemory); error_dev: kfree(dev); exit: return retval; } Here, we first allocate memory for our device descriptor, then for the structure of the framebuffer descriptor, and only then - for the framebuffer itself. Please note - select through vmalloc . The difference from kmalloc is that vmalloc can reconfigure the virtual memory page tables, and “collect in pieces” the buffer we requested. That is, for us it will look like a single block of memory, but physically it may consist of pages that are not even close together. kmalloc returns memory, which physically is a single unit. Since we are requesting a sufficiently large chunk, vmalloc will be a good practice.
Everything, we compile, insmodim, and if everything is done correctly we see the console on the display!
Conclusion
In this article, we finally took this important step - from a custom device driver that was not used by the system, we went into the framework's embrace, which allowed us to tell all applications and drivers that we have a real display. Yes, we did a little badly, not clearing the resources and not disconnecting the device, so pulling out the USB and reinstalling it on a working device is not recommended yet.
But we will definitely fix it in the next article. What do we have to do?
- , FPS .
- , - ( - , . , - ?)
- ,
- URB' Bulk-,
- , -. , , , Gobliins, , , .
Below is another video with an old quest (no less loved by me than Gobliins), a few photos of a small graphical shell that you can easily run on your device and the console browser Elinks.
The first Kyrandia
Habr in the console browser (font VGA8x8)
The shell Gmenu2X
Built-in file browser
Settings
On this I have everything. Successful implementation and until the next article!
Source: https://habr.com/ru/post/164635/
All Articles