Exploring the format of the level map of the Jackal NES game
The article will describe a non-standard way to search for data about levels in NES games — by sequentially changing all the data in an image and examining the consequences (“correlation” in terms of romhackers). For example, I will show you how to find the level data in the game Jackal for NES and add one of its levels to the CadEditor editor. This method allows you to explore any game with a block level structure (almost any game on NES), without knowledge of the X8502 assembler, only initial skills of working with scripting languages ( Lua and Python ) are required.
The idea of the method is to alternately change all the bytes in the game one by one and see if any of them change the level has changed. Hands to do this for a very long time, so you need to use auxiliary tools, namely, the Lua scripting language built into the FCEUX emulator, its capabilities and limitations will be discussed below.
In addition, some difficulty is the process of checking the result - a modified ROM image must be loaded into the emulator, and then the game should be completed before the first level is launched. Moreover, it is impossible to simply load the already saved emulator made on the original image at the moment after the start of the level - the data from the image will already get into the video memory and RAM, and the first level screen will remain unchanged, it will be impossible to visually distinguish it from the original. Therefore, it is necessary either to record in the FCEUX emulator the repetition of all keystrokes that causes the game from the start screen to start the level (this is done using the File-> Movie-> Record Movie ... menu), or it is simpler to do - to save directly to level loading moment. After clicking the "Start" button in the main menu of the game, before the start of a level, the image on the screen is dimmed for a few moments, at which time the first game screen of the level is created.
To save at the right time, you can slow down the time several times ( NES-> Emulation Speed-> Speed Down menu, I recommend setting up hotkeys in the Config-> Map Hotkeys menu to conveniently speed up / slow down the game from the keyboard). It is necessary to wait for the screen to black out and at this point (before the game screen appears) to save the game (say in slot 1, with the Shift + F1 key combination). Also, for further work, you need to detect which frame was saved and on which frame the game screen finally appeared (you can turn on the display of the current frame-frame on the screen using the menu item Config-> Display-> Frame Counter ). This will be needed in order to take a screenshot of the level screen at the moment when it is already created, so as not to remove the black screen earlier.
In the screenshot, the level data is loaded from the image into memory at the moment between frames 454 and 560.
')
Thus, the algorithm for searching data about levels will look like this:
In most cases, the image on the screen will not change, sometimes it will not exist at all, since in some versions of ROM images we will inevitably break the game code, sometimes there will be a strange porridge from tiles, the ultimate goal is to find the version in which the screen will be changed only one macroblock (recall again, the level screens in NES are made up of macroblocks ). When such a screenshot is found, it will become clear to which address the array of block numbers describing the screens is stored.
The Lua language built into the emulator allows you to operate with processor memory and registers; however, it is designed to work with the already loaded ROM cartridge image, so there are no operations either to load modified versions of ROM images or to change the image itself after loading, but with the help of it implement steps 2,3,4 of the algorithm.
The rest of the first version of the automatic corrapter I decided using the Python language. However, in that version there were several serious flaws - several thousand ROM-generated images took up a lot of space, the emulator was constantly launched from the command line as a separate application and closed, which caused focus to switch to it, so working on a machine with a running script was inconvenient, as close it in the course of work. Therefore, for the article I decided to add the necessary functionality to the Lua modules in order to change the byte within the ROM image directly from the emulator.
Fortunately, the source code for the FCEUX emulator is available , so you can download the source code and add your own modifications.
We find the function to read the byte from the ROM image (file fceu.cpp , function FCEU_ReadRomByte ) and add the same version for writing:
Next, we "forward" the ability to call this function from Lua (file lua_engine.cpp ):
Thus, we were able to change the data bytes from the scripts and got rid of the need to create and load thousands of different versions of ROM images.
The next stage is to teach Lua to analyze screenshots before saving them. To do this, instead of saving the standard screenshot to the file, the gui.savescreenshot () function will leave it in memory using the gui.gdscreenshot () function, and then check whether the same screenshot has already been taken (for this you have to store the hashes of all the screenshots already taken ), and save to disk only unique ones. This will allow not to store thousands of identical screenshots, in which a change in one byte did not affect the first screen of the game.
To save the screenshots, I used the gd library (the compiled version can be taken here or compiled from the source itself), the unpacked files should be put in the folder with the compiled emulator. To calculate the hashes, I used a little trick - I threw the calculation function from the emulator itself (there it was used to calculate the checksums of ROM images):
Now, the preparatory work is finally finished and you can write a Lua-script for the image corrupt (if the comments at the end of the lines are not displayed completely, you can view the commented scripts on the githabe, the link at the end of the article):
To run the emulator with a script, you can use the batch file with the following content:
fceux -turbo 1 -lua corrupt.lua "Jackal (U) [!]. nes"
(The turbo key will allow the emulator to run as fast as possible).
The script on my machine processes all data in 8 minutes. If it works too long, you can increase the STEP step to a larger one, up to 64 per screen, and also make a more accurate save of the game, in which the time between the launch frame and the frame on which you need to take a screenshot will be minimal.
A few recommendations for setting up a script for different games: data on screens often start from the beginning of banks (at addresses that are multiples of 0x2000 or 0x4000 ), these zones can be explored in more detail; If there are video banks ( CHR-ROM ) in the ROM image, you can not explore them, since they store only video memory. Video banks are always at the end of the ROM image, their number can also be viewed in the header (the first 16 bytes of the ROM image).
For the game “Jackal”, the script finds 235 unique screenshots, which present a wide range of various graphic glitches. However, of interest are screenshots made from images with modified bytes at addresses 0x1058, 0x105D8, 0x105E8 :
It is clear from them that:
What can be done with the data obtained? For example, prepare pictures of all level macroblocks that use them in the block editor CadEditor to create a level map editor.
To do this, we need to slightly rewrite the script of the corrush so that it records all possible byte values at the address that causes the block to change on the screen (for example, 0x105C8 ) and take screenshots of all the blocks. I will not give the full text of the script, it is in the example archive at the end of the article ( corrupt_byte.lu a ). Unfortunately, the gd library is not designed for convenient processing of parts of images, so to “bite out” the macroblock image from the screenshot and combine them for convenience into one long “tape” I had to write another Python script (with the PIL library installed for image processing):
The last part remains - it is necessary to create a configuration file for the CadEditor editor, which would use the resulting image. It uses C # as the scripting language (using the CSScript library).
From the screenshots, we calculate the beginning of the line in the macroblock map - if the address 0x105C8 changes the 4th macroblock in the line, then the address 0x105C5 is responsible for the first one . Next, create a template config:
The level map loaded into the editor with this config looks strange, although it resembles the real one:
After studying the result, it turns out that the lines of the screen of 16x8 macroblocks are stored in the order from bottom to top, but the screens themselves are from top to bottom, because of which it turns out that every 8 lines of the screen are swapped. Fortunately, the editor has a large number of methods that allow you to specify how exactly the level will be loaded from the ROM image. In this case, you need to set two special functions that will control exactly how the macroblock number will be read from the card and, accordingly, how it will be written back by the editor.
Everything, now the map is displayed correctly and you can redraw the level geometry:
The search method is applicable to almost all NES-games, you can use the scripts from the archive with an example to explore your favorite games!
In addition, with some modifications, the method is also applicable for the Sega Mega Drive and SNES platforms (the difference is that it is not the ROM image that needs to be modified, but the memory of the device, often the unpacked level map is stored in it).
In the next article I will show, using an example of some game, a device of game logic and methods of searching for game objects, in order to display and arrange them on the map.
References:
Archive with an example
Commented scripts
Theory
The idea of the method is to alternately change all the bytes in the game one by one and see if any of them change the level has changed. Hands to do this for a very long time, so you need to use auxiliary tools, namely, the Lua scripting language built into the FCEUX emulator, its capabilities and limitations will be discussed below.
In addition, some difficulty is the process of checking the result - a modified ROM image must be loaded into the emulator, and then the game should be completed before the first level is launched. Moreover, it is impossible to simply load the already saved emulator made on the original image at the moment after the start of the level - the data from the image will already get into the video memory and RAM, and the first level screen will remain unchanged, it will be impossible to visually distinguish it from the original. Therefore, it is necessary either to record in the FCEUX emulator the repetition of all keystrokes that causes the game from the start screen to start the level (this is done using the File-> Movie-> Record Movie ... menu), or it is simpler to do - to save directly to level loading moment. After clicking the "Start" button in the main menu of the game, before the start of a level, the image on the screen is dimmed for a few moments, at which time the first game screen of the level is created.
To save at the right time, you can slow down the time several times ( NES-> Emulation Speed-> Speed Down menu, I recommend setting up hotkeys in the Config-> Map Hotkeys menu to conveniently speed up / slow down the game from the keyboard). It is necessary to wait for the screen to black out and at this point (before the game screen appears) to save the game (say in slot 1, with the Shift + F1 key combination). Also, for further work, you need to detect which frame was saved and on which frame the game screen finally appeared (you can turn on the display of the current frame-frame on the screen using the menu item Config-> Display-> Frame Counter ). This will be needed in order to take a screenshot of the level screen at the moment when it is already created, so as not to remove the black screen earlier.
In the screenshot, the level data is loaded from the image into memory at the moment between frames 454 and 560.
')
Thus, the algorithm for searching data about levels will look like this:
- Run the modified ROM in the emulator
- We load the saving prepared in advance on the original ROM image (at the time of loading the data from the image into the video memory in order to get the changed level on the screen).
- We are waiting for a measured number of frames (for this we marked the Frame Counter value at the moment when the image appears on the screen).
- Make a screenshot of the screen.
- Repeat the process for the next version of the modified ROM image.
- We analyze the screenshots made (in the name of the screenshot made, you must specify the address of the byte that was changed, so that it was clear what change led to the data on the screen changed).
In most cases, the image on the screen will not change, sometimes it will not exist at all, since in some versions of ROM images we will inevitably break the game code, sometimes there will be a strange porridge from tiles, the ultimate goal is to find the version in which the screen will be changed only one macroblock (recall again, the level screens in NES are made up of macroblocks ). When such a screenshot is found, it will become clear to which address the array of block numbers describing the screens is stored.
The Lua language built into the emulator allows you to operate with processor memory and registers; however, it is designed to work with the already loaded ROM cartridge image, so there are no operations either to load modified versions of ROM images or to change the image itself after loading, but with the help of it implement steps 2,3,4 of the algorithm.
The rest of the first version of the automatic corrapter I decided using the Python language. However, in that version there were several serious flaws - several thousand ROM-generated images took up a lot of space, the emulator was constantly launched from the command line as a separate application and closed, which caused focus to switch to it, so working on a machine with a running script was inconvenient, as close it in the course of work. Therefore, for the article I decided to add the necessary functionality to the Lua modules in order to change the byte within the ROM image directly from the emulator.
Making tools
Fortunately, the source code for the FCEUX emulator is available , so you can download the source code and add your own modifications.
We find the function to read the byte from the ROM image (file fceu.cpp , function FCEU_ReadRomByte ) and add the same version for writing:
//new
void FCEU_WriteRomByte ( uint32 i, uint8 value )
{
if ( i < 16 + PRGsize [ 0 ] ) PRGptr [ 0 ] [ i - 16 ] = value ;
else if ( i < 16 + PRGsize [ 0 ] + CHRsize [ 0 ] ) CHRptr [ 0 ] [ i - 16 - PRGsize [ 0 ] ] = value ;
}
Next, we "forward" the ability to call this function from Lua (file lua_engine.cpp ):
static int rom_writebyte ( lua_State * L )
{
FCEU_WriteRomByte ( luaL_checkinteger ( L, 1 ) , luaL_checkinteger ( L, 2 ) )
return 1 ;
}
static const struct luaL_reg romlib [ ] = {
{ "readbyte" , rom_readbyte } ,
{ "readbytesigned" , rom_readbytesigned } ,
// alternate naming scheme for unsigned
{ "readbyteunsigned" , rom_readbyte } ,
{ "writebyte" , rom_writebyte } , // new function
{ "gethash" , rom_gethash } ,
{ NULL , NULL }
} ;
Thus, we were able to change the data bytes from the scripts and got rid of the need to create and load thousands of different versions of ROM images.
The next stage is to teach Lua to analyze screenshots before saving them. To do this, instead of saving the standard screenshot to the file, the gui.savescreenshot () function will leave it in memory using the gui.gdscreenshot () function, and then check whether the same screenshot has already been taken (for this you have to store the hashes of all the screenshots already taken ), and save to disk only unique ones. This will allow not to store thousands of identical screenshots, in which a change in one byte did not affect the first screen of the game.
To save the screenshots, I used the gd library (the compiled version can be taken here or compiled from the source itself), the unpacked files should be put in the folder with the compiled emulator. To calculate the hashes, I used a little trick - I threw the calculation function from the emulator itself (there it was used to calculate the checksums of ROM images):
// file lua_engine.cpp
static int calchash ( lua_State * L ) {
const char * buffer = luaL_checkstring ( L, 1 ) ;
int size = luaL_checkinteger ( L, 2 ) ;
int hash = CalcCRC32 ( 0 , ( uint8 * ) buffer, size ) ;
lua_pushinteger ( L, hash ) ;
return 1 ;
}
static const struct luaL_reg emulib [ ] = {
// part of the code is missing
// ...
{ "readonly" , movie_getreadonly } ,
{ "setreadonly" , movie_setreadonly } ,
{ "print" , print } , // sure, why not
{ "calchash" , calchash } ,
{ NULL , NULL }
} ;
Corrupt script and its use
Now, the preparatory work is finally finished and you can write a Lua-script for the image corrupt (if the comments at the end of the lines are not displayed completely, you can view the commented scripts on the githabe, the link at the end of the article):
- we load library gd
require "gd"
- the initial address for the corrupt (immediately from the header of the ROM image)
START_ADDR = 0x10
- the end address for the corrupt (depends on the mapper on which the cartridge is made, you can simply set the file size)
END_ADDR = 0x20010
CUR_ADDR = START_ADDR
- end frame number, after which you need to take a screenshot (when the game screen is already displayed for the player, the number is measured for the created save)
FRAME_FOR_SCREEN = 7035
- test value that will be written instead of game byte
WRITE_VALUE = 0x33
--Step, which will be used to make a corrupt, to save search time
- (it is not necessary to change each byte on the screen, a large number of macroblocks can be displayed on the screen, it is enough to detect at least one of them).
STEP = 8
- table for saving hashes of all unique screenshots
shas = { }
- remember the value that will be corrupted by the corrupt, in order to restore it later
lastValue = rom . readbyte ( START_ADDR )
- load the previously prepared saving from the FIRST slot (the numbering of slots in the emulator starts from 0, and in Lua - from 1).
s = savestate . create ( 2 )
savestate . load ( s )
while ( true ) do
--If the screen has loaded and is already displayed
if ( emu . framecount ( ) > FRAME_FOR_SCREEN ) then
--save screenshot in memory
local gdStr = gui . gdscreenshot ( ) ;
- count it hash
local hash = emu . calchash ( gdStr , string.len ( gdStr ) ) ;
--If there was no such screenshot
if ( not shas [ hash ] ) then
print ( "Found new screen" .. tostring ( hash ) ) ;
local fname = string.format ( "% 05X" , CUR_ADDR ) .. ".png" ;
local gdImg = gd . createFromGdStr ( gdStr ) ;
- save a screenshot to a disk with the specified byte address in the name
gdImg : png ( fname )
shas [ hash ] = true ;
end ;
- restore previous byte value
rom . writebyte ( CUR_ADDR , lastValue ) ;
CUR_ADDR = CUR_ADDR + STEP ;
- Correspondent next byte
lastValue = rom . readbyte ( CUR_ADDR ) ;
rom . writebyte ( CUR_ADDR , WRITE_VALUE ) ;
- loading save again to allow the game to load data from the new image into the memory
s = savestate . create ( 2 )
- display of progress
savestate . load ( s )
gui . text ( 20 , 20 , string.format ( "% 05X" , CUR_ADDR ) ) ;
--When all addresses are processed, stop the emulator.
if ( CUR_ADDR > END_ADDR ) then
emu . pause ( ) ;
end
end ;
- flush the emulation to the next frame
emu . frameadvance ( ) ;
end ;
To run the emulator with a script, you can use the batch file with the following content:
fceux -turbo 1 -lua corrupt.lua "Jackal (U) [!]. nes"
(The turbo key will allow the emulator to run as fast as possible).
The script on my machine processes all data in 8 minutes. If it works too long, you can increase the STEP step to a larger one, up to 64 per screen, and also make a more accurate save of the game, in which the time between the launch frame and the frame on which you need to take a screenshot will be minimal.
A few recommendations for setting up a script for different games: data on screens often start from the beginning of banks (at addresses that are multiples of 0x2000 or 0x4000 ), these zones can be explored in more detail; If there are video banks ( CHR-ROM ) in the ROM image, you can not explore them, since they store only video memory. Video banks are always at the end of the ROM image, their number can also be viewed in the header (the first 16 bytes of the ROM image).
For the game “Jackal”, the script finds 235 unique screenshots, which present a wide range of various graphic glitches. However, of interest are screenshots made from images with modified bytes at addresses 0x1058, 0x105D8, 0x105E8 :
It is clear from them that:
- The game uses a macroblock system of 2x2 blocks (4x4 tiles).
- Screens are described by lines of 16 macroblocks wide (the difference between two adjacent addresses).
- The lines are stored in the ROM image in order from bottom to top.
What can be done with the data obtained? For example, prepare pictures of all level macroblocks that use them in the block editor CadEditor to create a level map editor.
To do this, we need to slightly rewrite the script of the corrush so that it records all possible byte values at the address that causes the block to change on the screen (for example, 0x105C8 ) and take screenshots of all the blocks. I will not give the full text of the script, it is in the example archive at the end of the article ( corrupt_byte.lu a ). Unfortunately, the gd library is not designed for convenient processing of parts of images, so to “bite out” the macroblock image from the screenshot and combine them for convenience into one long “tape” I had to write another Python script (with the PIL library installed for image processing):
# - * - coding: utf-8 - * -
import Image
def cutBlock ( pp ) :
im = Image. open ( pp )
# upload a screenshot to any graphics editor to view the coordinates of the beginning of the block
X = 96
Y = 96
# cut from the screenshot block given coordinates
imCut = im. crop ( ( X , Y , X + 32 , Y + 32 ) )
imCut. save ( "_" + pp )
for x in xrange ( 256 ) :
cutBlock ( r "% 03d.png" % x )
BLOCK_COUNT = 102
MAX_BLOCK_COUNT = 256
imBig = Image. new ( "RGB" , ( 32 * MAX_BLOCK_COUNT , 32 ) )
for x in xrange ( BLOCK_COUNT ) :
im = Image. open ( "_% 03d.png" % x )
imBig. paste ( im , ( 32 * x , 0 , 32 * x + 32 , 32 ) )
# increase macroblock size to 64x64, requirement for use with CadEditor
imBig64 = imBig. resize ( ( MAX_BLOCK_COUNT * 64 , 64 ) )
imBig64. save ( "outStrip.png" )
Adding a game to the level editor
The last part remains - it is necessary to create a configuration file for the CadEditor editor, which would use the resulting image. It uses C # as the scripting language (using the CSScript library).
From the screenshots, we calculate the beginning of the line in the macroblock map - if the address 0x105C8 changes the 4th macroblock in the line, then the address 0x105C5 is responsible for the first one . Next, create a template config:
using CadEditor ;
using System ;
using System.Collections.Generic ;
using System.Drawing ;
public class Data
{
/ * calculate the correct address by alternately moving the boundaries of the lines up and down until
until the correct level map appears in the “window”.
From the starting address 0x10625 we fall back 96 lines up.
1 - the number of game screens at the level
16 * 96 - the size in bytes of one game screen
* /
public OffsetRec getScreensOffset ( ) { return new OffsetRec ( 0x10625 - 16 * 96 , 1 , 16 * 96 ) ; }
public int getScreenWidth ( ) { return 16 ; } // set the screen width
public int getScreenHeight ( ) { return 96 ; } // set the screen height
public int getBigBlocksCount ( ) { return 256 ; }
// connect a strip with pictures of macroblocks
public string getBlocksFilename ( ) { return "jackal_1.png" ; }
// turn off the macroblock sub-editors and enemies that are not implemented for this game
public bool isBigBlockEditorEnabled ( ) { return false ; }
public bool isBlockEditorEnabled ( ) { return false ; }
public bool isEnemyEditorEnabled ( ) { return false ; }
}
The level map loaded into the editor with this config looks strange, although it resembles the real one:
After studying the result, it turns out that the lines of the screen of 16x8 macroblocks are stored in the order from bottom to top, but the screens themselves are from top to bottom, because of which it turns out that every 8 lines of the screen are swapped. Fortunately, the editor has a large number of methods that allow you to specify how exactly the level will be loaded from the ROM image. In this case, you need to set two special functions that will control exactly how the macroblock number will be read from the card and, accordingly, how it will be written back by the editor.
// Telling the editor to use a special function to get the macroblock number from
// maps and records back
public GetBigTileNoFromScreenFunc getBigTileNoFromScreenFunc ( ) { return getBigTileNoFromScreen ; }
public SetBigTileToScreenFunc setBigTileToScreenFunc ( ) { return setBigTileToScreen ; }
public int getBigTileNoFromScreen ( int [ ] screenData, int index )
{
int w = getScreenWidth ( ) ;
int noY = index / w ;
noY = ( noY / 8 ) * 8 + 7 - ( noY % 8 ) ; // transformation of the Y-coordinate of the macroblock
int noX = index % w ;
return screenData [ noY * w + noX ] ;
}
public void setBigTileToScreen ( int [ ] screenData, int index, int value )
{
int w = getScreenWidth ( ) ;
int noY = index / w ;
noY = ( noY / 8 ) * 8 + 7 - ( noY % 8 ) ; // transformation of the Y-coordinate of the macroblock
int noX = index % w ;
screenData [ noY * w + noX ] = value ;
}
Everything, now the map is displayed correctly and you can redraw the level geometry:
The search method is applicable to almost all NES-games, you can use the scripts from the archive with an example to explore your favorite games!
In addition, with some modifications, the method is also applicable for the Sega Mega Drive and SNES platforms (the difference is that it is not the ROM image that needs to be modified, but the memory of the device, often the unpacked level map is stored in it).
In the next article I will show, using an example of some game, a device of game logic and methods of searching for game objects, in order to display and arrange them on the map.
References:
Archive with an example
Commented scripts
Source: https://habr.com/ru/post/259483/
All Articles