We pump over NES Classic Mini - continuation
In February, I wrote an article about how Russian enthusiasts “hacked” the NES Classic Mini console , how we did its reverse engineering, and how I wrote the application so that everyone can easily download their games into it in just a few clicks.
Today I will write a continuation of this story, because in six months we have made much more substantial progress in this matter.
')
As usual, my story is under the cut both in the modern video form and in the form of a traditional article.
The first turning point came when a person under the pseudonym madmonkey (if someone does not remember, it was he who was the author of the hacking) was able to compile the emulator RetroArch for NES Mini. More precisely, it is not an emulator, but a multiplatform shell based on Libretro for launching other emulators, which are installed as modules. Thanks to this, now on the NES Mini you can now run not only any games for NES, regardless of mappers, but also games from other consoles: SNES, Sega Mega Drive, Gameboy, Gameboy Advance and even simple games with Nintendo 64. Even the emulation of the cartridge works. I developed it myself if I compiled the fceux emulator module with my additions. Open Source Magic.
Of course, the emergence of such opportunities was accompanied by a bunch of new problems and challenges. For example, I really wanted the emulation through RetroArch to run not from its own menu, but also from the NES Mini shell.
I have already said that for each game in the config simply indicates the executable file that you want to run, and the command line parameters. But besides this, the native NES Mini emulator interacts with the shell, namely, it allows you to exit the menu, save and change settings. I wanted all these features to be maintained when using RetroArch. So that the user could not even understand that a third-party emulator is used.
This required some study of how the shell is arranged in the NES Mini. However, everything turned out to be trite and simple. In fact, when you press a button during a game to display a menu, the emulation does not pause. The shell kills the emulator process, and the emulator saves the game to a temporary file when exiting and makes a screenshot. The shell picks up these temporary files and shows this as a suspended game that can be saved. In this case, the temporary files are simply copied to the permanent memory.
As for the settings, they are all simply passed in the command line parameters.
It turns out that the task is quite simple: you need to write a script to start RetroArch, which will parse the parameters from the NES Mini shell, start the emulation, kill the emulator by the signal from the shell, while keeping the game in the right place and making a screenshot.
However, it was not without crutches. If RetroArch is able to make automatic saves, then with the creation of a screenshot at exit, everything is not so simple. Yes, RetroArch has an option to create screenshots for saving. But it does not apply to automatic saves. I decided to get a screenshot using an alternative method. In Linux, the framebuffer, which contains the image from the screen, is accessible via the pseudo file " / dev / fb0 ". It is only necessary to somehow read the data from it and save it as a PNG file. Fortunately, I quickly found a ready-made utility for this, which is called “ fbgrab ”. To my great surprise, I was able to compile it under the NES Mini on my own, but the screenshots obtained with its help were strange. For some reason, in the NES Mini framebuffer, almost all useful pixels have an absolute transparency level. I had to refine this utility so that it ignores transparency:
And where can there be transparency in the screenshot? I hope someone will explain it to me in the comments.
Soon everything worked, although the aspect ratio of the screenshots was a bit wrong, because the shell is designed to receive a screenshot in a ratio of four to three, not sixteen to nine, but this was again decided by a simple refinement of the utility. Well, or not very simple. I did an automatic aspect ratio determination.
Alas, I could not find a full-fledged replacement for the retro-filter, which is in the original emulator, and which very realistically creates an imitation of a CRT-TV. I do not like these filters, but in this case it looks really amazing and very realistic, I have never seen anything like it anywhere else. Therefore, if possible, I still give priority to the original NES Mini emulator, but it does not start all the games.
Let me remind you that inside each ".nes" file with the game contains a header, it describes the iron that was inside the original cartridge. The emulator must emulate not only the console itself, but also this hardware. The original NES Mini emulator emulates only the most popular cartridge options, RetroArch, thanks to modules, can run almost everything. I added a code to the RetroArch startup script that reads the header of the NES file through the hexdump utility, parses it and automatically decides which emulator to run - the original one or RetroArch:
As a result, the user generally does not need to bathe about the compatibility of NES games. If the native NES Mini emulator does not support the game, RetroArch will start automatically. If necessary, you can forcibly select the emulator via a special command line parameter.
However, there were more serious tasks ahead. How to allow users to install emulator and modules as easily as possible? madmonkey proposed to develop a universal mod system that would be compatible with both his hakchi and my hakchi2, and at the same time would make it possible to install a variety of hacks on the NES Mini in the form of plug-ins: from banal skins to emulators and alternative shells.
It was a series of sleepless nights when madmonkey and I sat in IRC and discussed how best to do everything. At the same time, he wrote a very beautiful code, and I tested it, found errors and corrected them with the help of my terrible code. Then he replaced my curve code with his own, after which everything was repeated anew. This went on for a week, in the end the mod system was ready.
We agreed that modules should have the extension “.hmod”, while being in fact “.tar.gz” archives that contain the necessary files and scripts that are executed during installation and removal. At the same time, hakchi itself implemented a set of functions for copying and replacing files. Later, I wrote a detailed guide on how to create mods with examples, starting with a banal interface editing and even before putting a password on your NES Mini, you can read it on GitHub:
github.com/ClusterM/hakchi2/wiki/Modifications-and-modules-guide (in English).
All this allowed the community to participate in the creation and distribution of various modifications. So, skins and modules for the Famikom translation into English, and much more, soon began to appear on the network. By the way, madmonkey made the process of turning the NES Mini into a Famicom Mini very easy, as well as vice versa. I transferred to the modules a part of the hakchi2 functional like extended fonts and a controller driver. And of course, the very first mod presented to the public was the RetroArch emulator described above and plug-ins to it for emulating different consoles. We must pay tribute to the person with the nickname pcm720 , who took the responsibility to develop the mod project further, I even gave him full rights to GitHub. It's great when there are such people, because it's difficult to follow several projects at once, but I wanted to continue working on hakchi2.
RetroArch mod on GitHub: github.com/ClusterM/retroarch-clover
By the way, hakchi2 of course had to be finalized too. And if everything is clear with the installation and removal of modules, it was not entirely clear with the games. I really didn’t want to add support for games from platforms other than NES directly to hakchi2. Still, this is a non-standard chip added by a separate mod, which is not included. However, a huge community of users really wanted it, and even with icons, covers and other features. So I gave up. Made for each console a separate class with its own features. and the person under the name NeoRame drew a bunch of cool pictures for each console, for which he thanks a lot.
And then it was the most interesting. When people began to record hundreds of different games from different platforms, constantly adding or deleting something, they were faced with the problem that every time, even with the addition of just one game, it was necessary to download all the games again. After all, as I told in the last article, we have no feedback from the console. We just take a bunch of games, a script to install them and run this case on the NES Mini, assuming that the games already installed should be removed and the new ones installed. We have no way of knowing which games are already installed.
Similarly, without a UART cable, there is no way to get any files from the system. You can not even banal save their save from there. You can’t even see how your system was dubbed, that there is not enough space there, which is really bad.
In general, we began to think how to solve this problem. madmonkey, for example, offered to write the necessary information into separate areas of flash memory, and then read them from there through the FEL mode, but I considered this option as the most extreme.
Reflecting on this task, I gradually continued to study the device of the NES Mini system and noticed that the startup of the USB gadget called “clover” was in startup. Obviously, this is something developed by Nintendo itself. Under the GPL license, the source code of the module must be publicly available. That's right, I found it among the Linux kernel sources on the Nintendo website . Just one small file ...
Judging by the comments, its main goal is a request for high current consumption from USB. Logically, this is so that the console can be powered not only from the mobile charge, but also from the computer. Actually, therefore, the computer defines the NES Mini as a device. What else is there?
Suddenly, a USB input / output code. And the code to work with the pseudo file. I checked - this pseudo file really is on the NES Mini.
At this moment, I was torn apart by contradiction and misunderstanding. On the one hand, I see that this should work. In theory. On the other hand, I cannot believe that no one had thought of this before me. And if this is true, have we really used such crutches for so long? I quickly sketched a program for a computer that writes data to USB. And successfully read them from this pseudo file. As well as vice versa. It works.
For the next few days, I remembered my long-forgotten programming skills under Linux and wrote a daemon that runs on the NES Mini and allows me to directly access the console and execute commands via USB, redirecting I / O streams via USB. For the computer, the client wrote:
github.com/ClusterM/clovershell-client
With it, you can dump all the original firmware with one simple command:
Later, I completely adapted hakchi2 for a new data transfer method. As a result, to download games, you no longer need to clamp the reset, and the transfer occurs very, very quickly. Literally in a minute all memory will be filled. Previously, it took half an hour, and it was necessary to break the process into several visits, while large files could not be transferred at all. I don’t know why Nintendo left this functionality, but it really helped us.
Soon, based on all of this, hakchi2 introduced a game saving manager, the ability to take screenshots, and even an FTP server. Yes, I took a ready-made FTP server library, where the work with the file system was implemented as a separate abstraction, and simply wrote a class that redirects all file operations via USB. As a result, we connect via FTP to localhost and see the files that are inside the NES Mini. Needless to say, as far as this facilitated the work of the creators of mods. UART-cable, even I stopped using without much need.
The moral of this story is that - if no one has yet done what seems easy and obvious to you, do not stop. You are very likely to do this. There are millions of people in the world who can do amazing things, but don’t do it just because they don’t believe in themselves. They do not believe that others would not have done it if they could. And sometimes it's enough just to start.
In the future, there were still a lot of minor fixes and improvements, about which there is no point in telling. I'll tell you about the most interesting.
I was able to figure out why many third-party controllers designed for the Wii did not work with the NES Mini, despite the same connector and protocol. I did not have such, but often complained about the problem.
First, it turns out that Classic Controller has two data transfer formats. And they can choose. On the Internet, only one is described everywhere:
And in many third-party controllers, only he is implemented. The driver of the clovercon-controllers (I remind you that its sources are also publicly available) uses a different format, which has more capacity and accuracy of analog sticks. Very ironic, because the NES Mini does not initially use analogs.
Secondly, in the clovercon driver code on the NES Mini, there is protection from power interference, which is typical when powering is not from batteries. He checks that all unused data in the memory of the controller is zero, although many Chinese controllers, on the contrary, are clogged with FFs.
All this was decided by the driver. But still there were very, very Chinese controllers who did not want to work. And the problem was the hardware, apparently. To solve this riddle, immediately two people sent me their controllers for testing. Thanks for this to Vladimir Kondratenko and keithelmcity from the GBATemp website. The answer turned out to be simple - just pull up the SDA and SCL lines with two resistors to the power supply. This is required by the I²C protocol, which is used in these controllers. The Chinese saved on a couple of resistors. Alas, to fix this is not as easy as updating the program, but many users without any problems were able to do it themselves. Fortunately, even the one who holds the soldering iron for the first time can probably solder two resistors.
When I was already bored with this project, and the plans for further development were almost over, I finally shared a table with lines from the program for people, and almost two dozen people from different parts of the world translated it into their languages. You know, this is a very strange feeling when you work on a program for a long time, and then you see it in French, German, Spanish, Italian, Greek, and a bunch of different languages:
Perceived as something native, and someone else at the same time.
I still had some ideas and plans. For example, connect the SD card to the free contacts that are inside the NES Mini. But I really do not like for a long time to engage in any one project, and the people have already lost interest. So I’ll probably make further changes only after the release of the SNES Mini, if it is hacked.
hakchi2 on github: github.com/ClusterM/hakchi2
Today I will write a continuation of this story, because in six months we have made much more substantial progress in this matter.
')
As usual, my story is under the cut both in the modern video form and in the form of a traditional article.
Video
Article
More emulators
The first turning point came when a person under the pseudonym madmonkey (if someone does not remember, it was he who was the author of the hacking) was able to compile the emulator RetroArch for NES Mini. More precisely, it is not an emulator, but a multiplatform shell based on Libretro for launching other emulators, which are installed as modules. Thanks to this, now on the NES Mini you can now run not only any games for NES, regardless of mappers, but also games from other consoles: SNES, Sega Mega Drive, Gameboy, Gameboy Advance and even simple games with Nintendo 64. Even the emulation of the cartridge works. I developed it myself if I compiled the fceux emulator module with my additions. Open Source Magic.
Of course, the emergence of such opportunities was accompanied by a bunch of new problems and challenges. For example, I really wanted the emulation through RetroArch to run not from its own menu, but also from the NES Mini shell.
I have already said that for each game in the config simply indicates the executable file that you want to run, and the command line parameters. But besides this, the native NES Mini emulator interacts with the shell, namely, it allows you to exit the menu, save and change settings. I wanted all these features to be maintained when using RetroArch. So that the user could not even understand that a third-party emulator is used.
This required some study of how the shell is arranged in the NES Mini. However, everything turned out to be trite and simple. In fact, when you press a button during a game to display a menu, the emulation does not pause. The shell kills the emulator process, and the emulator saves the game to a temporary file when exiting and makes a screenshot. The shell picks up these temporary files and shows this as a suspended game that can be saved. In this case, the temporary files are simply copied to the permanent memory.
As for the settings, they are all simply passed in the command line parameters.
It turns out that the task is quite simple: you need to write a script to start RetroArch, which will parse the parameters from the NES Mini shell, start the emulation, kill the emulator by the signal from the shell, while keeping the game in the right place and making a screenshot.
However, it was not without crutches. If RetroArch is able to make automatic saves, then with the creation of a screenshot at exit, everything is not so simple. Yes, RetroArch has an option to create screenshots for saving. But it does not apply to automatic saves. I decided to get a screenshot using an alternative method. In Linux, the framebuffer, which contains the image from the screen, is accessible via the pseudo file " / dev / fb0 ". It is only necessary to somehow read the data from it and save it as a PNG file. Fortunately, I quickly found a ready-made utility for this, which is called “ fbgrab ”. To my great surprise, I was able to compile it under the NES Mini on my own, but the screenshots obtained with its help were strange. For some reason, in the NES Mini framebuffer, almost all useful pixels have an absolute transparency level. I had to refine this utility so that it ignores transparency:
/* ALPHA */ outbuffer[(i<<2)+Alpha] = /*srcAlpha >= 0 ? inbuffer[i*4+srcAlpha] :*/ 0; And where can there be transparency in the screenshot? I hope someone will explain it to me in the comments.
Soon everything worked, although the aspect ratio of the screenshots was a bit wrong, because the shell is designed to receive a screenshot in a ratio of four to three, not sixteen to nine, but this was again decided by a simple refinement of the utility. Well, or not very simple. I did an automatic aspect ratio determination.
Alas, I could not find a full-fledged replacement for the retro-filter, which is in the original emulator, and which very realistically creates an imitation of a CRT-TV. I do not like these filters, but in this case it looks really amazing and very realistic, I have never seen anything like it anywhere else. Therefore, if possible, I still give priority to the original NES Mini emulator, but it does not start all the games.
Let me remind you that inside each ".nes" file with the game contains a header, it describes the iron that was inside the original cartridge. The emulator must emulate not only the console itself, but also this hardware. The original NES Mini emulator emulates only the most popular cartridge options, RetroArch, thanks to modules, can run almost everything. I added a code to the RetroArch startup script that reads the header of the NES file through the hexdump utility, parses it and automatically decides which emulator to run - the original one or RetroArch:
supported_mappers="0 1 2 3 4 5 7 9 10 86 87 184" emulator=retroarch args=$@ filename=$1 extension=${filename##*.} if [ "$extension" == "nes" ]; then header=$(hexdump -v -n 8 -e '1/1 "%02X"' "$filename") mapper=$((0x${header:14:1}${header:12:1})) fourscreen=$(expr $((0x${header:13:1})) / 8) echo mapper: $mapper echo four-screen: $fourscreen for m in $supported_mappers; do [ "$m" == "$mapper" ] && emulator=kachikachi done [ "$fourscreen" == "1" ] && emulator=retroarch fi As a result, the user generally does not need to bathe about the compatibility of NES games. If the native NES Mini emulator does not support the game, RetroArch will start automatically. If necessary, you can forcibly select the emulator via a special command line parameter.
Mod system
However, there were more serious tasks ahead. How to allow users to install emulator and modules as easily as possible? madmonkey proposed to develop a universal mod system that would be compatible with both his hakchi and my hakchi2, and at the same time would make it possible to install a variety of hacks on the NES Mini in the form of plug-ins: from banal skins to emulators and alternative shells.
It was a series of sleepless nights when madmonkey and I sat in IRC and discussed how best to do everything. At the same time, he wrote a very beautiful code, and I tested it, found errors and corrected them with the help of my terrible code. Then he replaced my curve code with his own, after which everything was repeated anew. This went on for a week, in the end the mod system was ready.
We agreed that modules should have the extension “.hmod”, while being in fact “.tar.gz” archives that contain the necessary files and scripts that are executed during installation and removal. At the same time, hakchi itself implemented a set of functions for copying and replacing files. Later, I wrote a detailed guide on how to create mods with examples, starting with a banal interface editing and even before putting a password on your NES Mini, you can read it on GitHub:
github.com/ClusterM/hakchi2/wiki/Modifications-and-modules-guide (in English).
All this allowed the community to participate in the creation and distribution of various modifications. So, skins and modules for the Famikom translation into English, and much more, soon began to appear on the network. By the way, madmonkey made the process of turning the NES Mini into a Famicom Mini very easy, as well as vice versa. I transferred to the modules a part of the hakchi2 functional like extended fonts and a controller driver. And of course, the very first mod presented to the public was the RetroArch emulator described above and plug-ins to it for emulating different consoles. We must pay tribute to the person with the nickname pcm720 , who took the responsibility to develop the mod project further, I even gave him full rights to GitHub. It's great when there are such people, because it's difficult to follow several projects at once, but I wanted to continue working on hakchi2.
RetroArch mod on GitHub: github.com/ClusterM/retroarch-clover
By the way, hakchi2 of course had to be finalized too. And if everything is clear with the installation and removal of modules, it was not entirely clear with the games. I really didn’t want to add support for games from platforms other than NES directly to hakchi2. Still, this is a non-standard chip added by a separate mod, which is not included. However, a huge community of users really wanted it, and even with icons, covers and other features. So I gave up. Made for each console a separate class with its own features. and the person under the name NeoRame drew a bunch of cool pictures for each console, for which he thanks a lot.
It could be easier
And then it was the most interesting. When people began to record hundreds of different games from different platforms, constantly adding or deleting something, they were faced with the problem that every time, even with the addition of just one game, it was necessary to download all the games again. After all, as I told in the last article, we have no feedback from the console. We just take a bunch of games, a script to install them and run this case on the NES Mini, assuming that the games already installed should be removed and the new ones installed. We have no way of knowing which games are already installed.
Similarly, without a UART cable, there is no way to get any files from the system. You can not even banal save their save from there. You can’t even see how your system was dubbed, that there is not enough space there, which is really bad.
In general, we began to think how to solve this problem. madmonkey, for example, offered to write the necessary information into separate areas of flash memory, and then read them from there through the FEL mode, but I considered this option as the most extreme.
Reflecting on this task, I gradually continued to study the device of the NES Mini system and noticed that the startup of the USB gadget called “clover” was in startup. Obviously, this is something developed by Nintendo itself. Under the GPL license, the source code of the module must be publicly available. That's right, I found it among the Linux kernel sources on the Nintendo website . Just one small file ...
Judging by the comments, its main goal is a request for high current consumption from USB. Logically, this is so that the console can be powered not only from the mobile charge, but also from the computer. Actually, therefore, the computer defines the NES Mini as a device. What else is there?
Suddenly, a USB input / output code. And the code to work with the pseudo file. I checked - this pseudo file really is on the NES Mini.
At this moment, I was torn apart by contradiction and misunderstanding. On the one hand, I see that this should work. In theory. On the other hand, I cannot believe that no one had thought of this before me. And if this is true, have we really used such crutches for so long? I quickly sketched a program for a computer that writes data to USB. And successfully read them from this pseudo file. As well as vice versa. It works.
For the next few days, I remembered my long-forgotten programming skills under Linux and wrote a daemon that runs on the NES Mini and allows me to directly access the console and execute commands via USB, redirecting I / O streams via USB. For the computer, the client wrote:
github.com/ClusterM/clovershell-client
With it, you can dump all the original firmware with one simple command:
clovershell exec "dd if=/dev/mapper/root-crypt | gzip" > dump.img.gz Later, I completely adapted hakchi2 for a new data transfer method. As a result, to download games, you no longer need to clamp the reset, and the transfer occurs very, very quickly. Literally in a minute all memory will be filled. Previously, it took half an hour, and it was necessary to break the process into several visits, while large files could not be transferred at all. I don’t know why Nintendo left this functionality, but it really helped us.
Soon, based on all of this, hakchi2 introduced a game saving manager, the ability to take screenshots, and even an FTP server. Yes, I took a ready-made FTP server library, where the work with the file system was implemented as a separate abstraction, and simply wrote a class that redirects all file operations via USB. As a result, we connect via FTP to localhost and see the files that are inside the NES Mini. Needless to say, as far as this facilitated the work of the creators of mods. UART-cable, even I stopped using without much need.
The moral of this story is that - if no one has yet done what seems easy and obvious to you, do not stop. You are very likely to do this. There are millions of people in the world who can do amazing things, but don’t do it just because they don’t believe in themselves. They do not believe that others would not have done it if they could. And sometimes it's enough just to start.
In the future, there were still a lot of minor fixes and improvements, about which there is no point in telling. I'll tell you about the most interesting.
Improved support for third-party controllers
I was able to figure out why many third-party controllers designed for the Wii did not work with the NES Mini, despite the same connector and protocol. I did not have such, but often complained about the problem.
First, it turns out that Classic Controller has two data transfer formats. And they can choose. On the Internet, only one is described everywhere:
And in many third-party controllers, only he is implemented. The driver of the clovercon-controllers (I remind you that its sources are also publicly available) uses a different format, which has more capacity and accuracy of analog sticks. Very ironic, because the NES Mini does not initially use analogs.
Secondly, in the clovercon driver code on the NES Mini, there is protection from power interference, which is typical when powering is not from batteries. He checks that all unused data in the memory of the controller is zero, although many Chinese controllers, on the contrary, are clogged with FFs.
All this was decided by the driver. But still there were very, very Chinese controllers who did not want to work. And the problem was the hardware, apparently. To solve this riddle, immediately two people sent me their controllers for testing. Thanks for this to Vladimir Kondratenko and keithelmcity from the GBATemp website. The answer turned out to be simple - just pull up the SDA and SCL lines with two resistors to the power supply. This is required by the I²C protocol, which is used in these controllers. The Chinese saved on a couple of resistors. Alas, to fix this is not as easy as updating the program, but many users without any problems were able to do it themselves. Fortunately, even the one who holds the soldering iron for the first time can probably solder two resistors.
Other
When I was already bored with this project, and the plans for further development were almost over, I finally shared a table with lines from the program for people, and almost two dozen people from different parts of the world translated it into their languages. You know, this is a very strange feeling when you work on a program for a long time, and then you see it in French, German, Spanish, Italian, Greek, and a bunch of different languages:
Perceived as something native, and someone else at the same time.
I still had some ideas and plans. For example, connect the SD card to the free contacts that are inside the NES Mini. But I really do not like for a long time to engage in any one project, and the people have already lost interest. So I’ll probably make further changes only after the release of the SNES Mini, if it is hacked.
hakchi2 on github: github.com/ClusterM/hakchi2
Source: https://habr.com/ru/post/333418/
All Articles