Analysis of keygenme from Ra $ cal on the basis of a virtual machine

0. Info

KeygenMe page on crackmes.de
Crackme with simple vm. Key check algorithm is simple, so the main target is vm.
It is a pause to start. it looks like it’s like a regulator, registers, commands =)
Good luck and have fun.
Difficulty: 4 - Needs special knowledge
Platform: Windows
Language: C / C ++

We will start our research by viewing the code that is executed when we click the “Check” button on the form. Since CrackMe is based on the usual dialog box, we first look at its registered window function.


Figure 1. Part of the window dialog function responsible for handling WM_COMMAND.

Clicking the button causes the application to send a WM_COMMAND message with the ID of the element at which the event occurred. In this case, the received values ​​are received using the GetDlgItemTextA API, and then the function that checks the entered values ​​is called.
')

Figure 2. The function Check called from the window function.

Looking at the large number of NOP commands (403F3E - 4035A9 = 995h), it can be assumed that the source code of the algorithm that was chosen was originally located here, and was later jammed.
The sub_4017C0 function is a wrapper over a virtual machine call, the main loop of which is located in the sub_401890 function , the pseudo-code of which is shown below.


Figure 3. The main loop of the virtual machine in the sub_401890 function.

All these functions inside switch / case are command implementations.

The virtualized code itself is stored in the .rvmpc section, starting at address 00407000.

1. Command format

Determining the structure of a virtual machine, determining its architecture, as well as the format of the commands it uses, is the first and most important task for code virtualization.
This virtual machine is a registered virtual machine, a stack is used for temporary storage of variables.
The command format of this virtual machine is not of fixed length and is rather redundant. All types of commands have a general header of the following form:

Bias	Type of	The size	Description
+0	word	2	operation code
+2	dword	four	Team ID
+6	byte	one	argument size
+7	dword	four	ID of the next command
+ Bh	dword	four	unknown

The “Team ID” field contains a unique value for the entire bytecode. Accordingly, to move to the next command, the virtual machine takes a value from the “ID of the next command” field and searches in the entire code array for a command in which the ID matches the one specified. After that, using switch / case on the value of the “operation code” field, each command is interpreted.

The instruction set of the virtual machine under consideration allows operating its internal registers, native processor registers, and executing the native code.

opcode	Act
0x00	call
0x01	call
0x02	Conditional transition
0x03	push dword
0x04	sub / add ESP
0x05	push dword
0x06	mov [esp], dword
0x07	jmp ID
0x08	mov [ebp +?], dword
0x09	native
0x0A	native
0x0B	-
0x0C	Modification of the internal flag dw4052AC
0x0D	-
0x0E	-
0x0F	-
0x10	mov REGn, dword-reg / mov REGn, dword [reg + X]
0x11	mov dword [addr], (reg / dword) / mov dword [REGn], (reg / dword)
0x12	Various forwarding commands mov
0x13	Various forwarding commands mov
0x14	MOV _32 [A], unpack (REGn)
0x15	MOV REGn, pack (_32 [A])
0x16	XOR _32 [A] [i], _32 [B] [j]
0x17	mov [REGn], reg / mov reg, [REGn]
0x18	XOR _32 [A] [c: d], _32 [B] [e: f]
0x19	MOV _32 [A], 0

Commands 0x14 - 0x19 work with an additional area of ​​memory that allows performing bitwise operations on 32-bit numbers.

2. Disassembling VM Code

Now when we have a description of the format of commands and we know what functions they perform, it is worth writing a disassembler for this virtual machine. Commands may not be at all like the command system of previously known architectures, so you can translate into mnemonics that you understand.

Disassembler sources using BeaEngine library (+ exe)
The disassembly command listing has made 961 lines.

3. Devirtualization

This step is optional if the zavrutalenny code is small and you can understand the algorithm without re-encoding it into the native code. As already mentioned above, the process of devirtualization consists in recoding the code for the virtual machine into the code for the native processor. To do this, the disassembler instead of mnemonic virtual machine should produce one or more mnemonic native processor. Then you should compile the resulting code. This operation will provide support for existing tools - debuggers, decompilers, etc.

In principle, I disassembled the algorithm at the most according to disassembler listing, but I wondered how HexRays would behave if the resulting listing was compiled into the native x86 code. To do this, I got rid of all internal registers in command mnemonics, rewrote some of the code that works with bit values, and compiled everything in exe. I did not achieve ideal behavior, so there are some logical errors in the decompiled code below.

The resulting source for x86 asm
The source is built using FASM .

So for example a couple of commands

MOV REGn, EBP-x MOV reg, [REGn] 

turn into one

 MOV reg, [EBP-x] 

and commands to move in the opposite direction

 MOV REGn, EBP-x MOV [REGn], reg 

turn into

 MOV [EBP-x], reg 

Most of all, the construction of the form is reduced:

 MOV REG9, DWORD PTR [ebp-E0] MOV eax, DWORD PTR [REG9] MOV [REG2], eax MOV _32[1], 0 MOV _32[0], unpack(REG2) XOR _32[0][0:7], _32[1][18:1F] XOR _32[0][8:F], _32[1][10:17] XOR _32[0][10:17], _32[1][8:F] XOR _32[0][18:1F], _32[1][0:7] MOV REG2, pack(_32[1]) MOV eax, [REG2] MOV REG9, DWORD PTR [ebp-E0] MOV DWORD PTR [REG9], eax 

On the x86 assembly language, it will look like this:

 MOV eax, DWORD PTR [ebp-E0] BSWAP eax MOV DWORD PTR [ebp-E0], eax 

Then I set the Hex-Rays plugin on the compiled EXE and get the source as in the picture below:


Figure 4. Decompiled code rebuilt under the x86 architecture.

4. Inversion of the algorithm

This is the saddest part of the study, because the algorithm is nothing interesting and is based on XOR and cyclic shifts.

The key is as follows:
SSSS-11111111HHHHHHHH22222222 - ???
where:
SSSS is a numerical value in a 10 numeral system;
11111111, 22222222, HHHHHHHH - values ​​in the 16th numeral system;
??? - arbitrary value, arbitrary length

Algorithm:

1. We count the sum of the characters of the entered name
   

    for ( i = 0; i < nLenName; i++ ) dwNameSum += name[i] 

2. Compare the value obtained with the first key block;
3. We consider hash on behalf of the computer
   

    for ( j = 0; j < nCompNameLen; j++ ) { dwHashCompName ^= j ^ szCompName[j]; dwHashCompName = __ROL__(dwHashCompName, 3); } 

4. From sections 11111111, 22222222 and HHHHHHHH we get from the binary representation (hexdec) - analogue of dwHash1, dwHash2, dwHash3.
5. For each of the values ​​we perform the operation:
   dwHashN = dwHashN xor dwHashCompName xor htonl(dwHashCompName)
6. The key hash is calculated as follows:
   dwHashSerial = dwHash3 xor dwHash1 xor htonl(dwHash1) xor dwHash2 xor htonl(dwHash2) xor dwHashCompName xor htonl(dwHashCompName)
7. Hash on behalf of is calculated by the algorithm:
   

    for ( idx = 0; idx < nLenName; idx++ ) { if ( idx % 2 ) dwHashName ^= 0x4F620AEC ^ (idx + dwHash1) ^ szName [idx]; else dwHashName ^= 0x4F620AEC ^ (dwHash2 - idx) ^ szName[idx]; dwHashName = __ROL__(dwHashName, idx); } 

All we need to do is to randomly generate 2 values ​​for blocks 1 and 2. Then consider dwHashName and dwHashSerial, taking dwHash3 equal to 0. And it remains only to calculate the missing value using the formula dwHash3 = dwHashName ^ dwHashSerial

And of course the result of the whole study:



Download EXE and keygen sources

PS Yandex burns somehow direct links, so if it shows 404, open links in a new tab.