Decompilation of RNC ProPack 5 years long

Greetings, friends!

In this article, I will tell you how for several years I worked with reverse 46 KB (it seems - just that!) An executable file from AmigaOS , learned a lot of new things for myself, tried many different technologies, and, as a result, got it - turned decompiled Motorola M68000 assembly code in C-shny code that can be used by anyone .

Prehistory

There is such a thing as RNC ProPack (Rob Rorthen Compression). This is a very common data packer. They packaged games for MS-DOS , Sega Genesis / Mega Drive , Game Boy , SNES , Sony Playstation , and perhaps some other platforms. Actually, because of the Sega Mega Drive, I took up the study of the packer.

The LHA archive by reference contains executable files under AmigaOS , and MS-DOS , which, in today's reality, working under 64-bit operating systems is not great, but an obstacle to the normal use of the utility.

Yes, there is a source code, and the supposedly working version of the RNC version 1 decompressor , but in fact it works crookedly, compresses worse, and there are some other bad things that I didn’t want to use this section.

Of course, there are all sorts of DOSBox- s, WinUAE , but this turns out to be somehow very cumbersome in fact.

Quest for truth

First of all, I tried to find the author, contact him, find ready source codes that would be easier to port to modern realities. But the author turned out to be quite difficult to find. Rob Northen is now a middle-aged grandfather, he no longer has the source code, and that link with Aminet leads to an archive with programs that he did not even compile (according to the author, the source code was in pure assembler, and everything else is not his).

UPD. :

As far as I understood, the source code for the packer was delivered to game developers for money. There were several different versions of RNC ProPack under MS-DOS , packed with different versions of protectors of executable files. The AmigaOS version was also packaged (by the RNC ProPack itself ).

Approach number one

At first I tried to start debugging this junk. At that time (well, about 5 years ago), MS-DOS was closer for me, so I decided to start with it, use IDA Pro , analyze the executable file and do everything quietly, without touching anyone. :) It didn't work .. .

First problem: the file was protected by some packer of executable files under MS-DOS . Most modern PE analyzers ( PEID , Exeinfo PE ) had no idea about what to do with dosovsky rarities. Therefore, they kept silent in response:

Only Detect It Easy helped:

I don’t remember exactly how, but through DOSBox with the debugger enabled I managed to somehow unpack the WWPack :

But Borland C ++ signatures in IDA did not really want to recognize most of the standard library functions (as I understood it: on top of the usual fopen calls there were wrapper functions that did not receive names). Then I decided on my own with the help of Flare - IDA utility to collect signatures. That helped a little.

Still not the way I wanted ... I wanted the convenience, the source code, the dynamic debugging of the code ...
Here, it seems, IDA Pro can work with segments, selectors, 16-bit code ... Yes. But, firstly, the standard debugger does not. Secondly, the only debugging plugin available at that time (which Ilfak, well-known to all, subsequently added as a reference in the SDK ):

#define DEBUGGER_ID_X86_DOSBOX_EMULATOR 10 ///< Dosbox MS-DOS emulator 

... works very crookedly, hangs, debugger-hints point to the wrong target addresses in relative addresses like: DWORD PTR DS:[EAX+58h] , etc.

I decided to rewrite it ! It is this version that I recommend to use when debugging MS-DOS applications.

But, having worked with this option, I realized that this was not the same ... Convenience increased, but the goal was also far away.

What if the second time work?

This time, after some time, I heard about such a wonderful project as amitools , and, in particular, about one of its components: vamos .

This is such a self-contained AmigaOS emulator (if you remember, the second executable file was just under it) in python . As an emulator for the M68k processor, Musashi is used there.

It seemed to me a great idea. Write a debugger plugin for IDA that pulls python code, executing the code I need inside. But, I didn't manage to figure out embedded python that time. Actually, the python itself was still then I almost did not know / did not apply.
Therefore, did not grow together.

Yes, and the packer executable file itself worked poorly in the emulator: repeatedly tried to pack the same file, some AmigaOS system calls did not work at all.

I even tried to rewrite vamos in C to embed it in the debager. But there are no cool python2c decompilers, and I quickly get tired of rewriting with my hands. The case was hopeless.

Tried even debugging through WinUAE , wrote UaeIDA . I built in my own code for the server side of the debugger module , trying to debug it, reporting bugs. It seems the case even somehow more or less acceptable move. But, the impassable bugs of WinUAE itself (as well as the author’s unwillingness to mess with the part of the source responsible for the debugger), in the end, I again reached a dead end.

Now it will be released

Having gathered my strength for the last time, after all these foolish attempts that didn’t lead to the desired result, I copied another Amiga Cruncher ( Imploder ) WinImploder to C by that time and decided to return to the Amitools variant again.

The project seems to have fixed some bugs, the assembly of the project has been simplified ... Therefore, the start was given! In the end, I got this frankenstein : Amitools_dbg . You start python- part with the -dbg flag, from IDA you launch the debugger module, and ... voila - we are at the entry point to the application. And we can even do debugging!

Only system API calls were still a line in the disassembler of the form:
JSR -$3A(a6) , where a6 is the base of the system library, and $ 3A is the offset of the API call.

Having rummaged through the Internet, I found such a once actual project as ReSource . Archives with the program can still be found on the Internet, and run them in WinUAE . So there were fd- files for each library with an indication of offsets. Amitools just had a parser of these fd files, and I took it myself. As a result, my debugger got more recognition of such system calls, which helped me in identifying some of the wrappers over API calls.

Over all previous years, I managed to figure out approximately the C compiler that someone used to get the AmigaOS executable file: SAS / C of some ancient version. But the attempt to incite lib files from the delivery to IDA did not bring success, so I did it with my hands (yes, not quite as much as I wanted, but also nothing).

Sources

If someone is not in the subject line, then one cannot simply take and rewrite the assembler code in C :

A bunch of nuances: the names of variables, functions, fields of structures. And so it was still beautiful. Therefore, the naming of all this takes a lot of time.

How do you like that?

 #pragma pack(push, 1) typedef struct huftable_s { uint32 l1; // +0 uint16 l2; // +4 uint32 l3; // +6 uint16 bit_depth; // +A } huftable_t; #pragma pack(pop) typedef struct vars_s { uint16 max_matches; uint16 enc_key; uint32 pack_block_size; uint16 dict_size; uint32 method; uint32 pus_mode; uint32 input_size; uint32 file_size; // inner uint32 bytes_left; uint32 packed_size; uint32 processed_size; uint32 v7; uint32 pack_block_pos; uint16 pack_token, bit_count, v11; uint16 last_min_offset; uint32 v17; uint32 pack_block_left_size; uint16 match_count; uint16 match_offset; uint32 v20, v21; uint32 bit_buffer; uint32 unpacked_size; uint32 rnc_data_size; uint16 unpacked_crc, unpacked_crc_real; uint16 packed_crc; uint32 leeway; uint32 chunks_count; uint8 *mem1; uint8 *pack_block_start; uint8 *pack_block_max; uint8 *pack_block_end; uint16 *mem2; uint16 *mem3; uint16 *mem4; uint16 *mem5; uint8 *decoded; uint8 *window; uint32 read_start_offset, write_start_offset; uint8 *input, *output, *temp; uint32 input_offset, output_offset, temp_offset; uint8 tmp_crc_data[2048]; huftable_t raw_table[16]; huftable_t pos_table[16]; huftable_t len_table[16]; } vars_t; 

It looks awful, as for me. But now the code is working ... Maybe in the future, I will fix something else in this code.

Yes, the bugs were after looking all over, did a couple of tests. Somewhere < , but should be <= , the constants are not so adapted, etc. But, now sort of fixed it.

Of course, I did not begin to rewrite the packaging of MS-DOS applications, AmigaOS applications with this packer (there was such functionality). I just wanted the algorithm itself.

findings

The most banal: if you want to achieve something, try, do not be afraid. Do this step by step. And you all turned out.

Link to IDB- file for AmigaOS binary:
https://mega.nz/#!3Z1TBA6J!ZOOfQf4Jmp_I704yYynzMVI_3To3etMjqRC2eF9b0Jg

RNC Propack Source : https://github.com/lab313ru/rnc_propack_source