How did I manage to hack and unpack the resources of the old game for PSX

“I’d like to unpack these game archives and see what's inside!”, He probably thought to himself, at least once, every gamer who wanted to understand how his favorite game is arranged.

Fortunately, today, most developers not only do not interfere with the study of their games, but on the contrary, they do everything so that players change and complement the games themselves. But even if there is no official documentation, for 99% of the games you can find ready-made programs for unpacking.

I decided to write this article in order to show that even if you are faced with a very rare, old or useless game, the archives of which are not taken by any “unpacker”, then even with minimal knowledge of a programming language, it is quite possible for you manages to cope on his own and become the first who can gut this game down to the bones.
')
Since the device description of the game archives is unlikely to be of any benefit, I will describe the entire path that I made during the study of the game archives, the course of my thoughts, as well as the mistakes that led me to a standstill.

Outset

It all started when I was a fan of SquareSoft games as a child, and rumbling about idleness on January 1, I got interested in information about early prototypes of Final Fantasy and was surprised to learn that the Japanese version of Final Fantasy VII had a bonus disc for PlayStation with all sorts of additional materials, including sketches and screenshots of the game in the early stages of development.
Immediately downloading his image from the "torrents" and running on the emulator, I found with some disappointment that, firstly, the disk is terribly slow in the emulator (despite the new hardware), and secondly, the entire interface in Japanese and the devil will break my foot through it this is a menu.



Then I had an idea, to gut the disk and study its content directly, and not to use the interface proposed by the developers.

Primary investigation

First of all, I decided to study what files are generally contained on the disk.


The first thing that caught my eye was the extension of the NPK and LPK files, which indirectly indicated that these were archives.

Therefore, one of the first ideas that occurred to me was to use all sorts of universal analyzers and unpackers. I downloaded everything I could find for PSX and just in case, even Dragon UnPACKer, for PC.
But the catch was insignificant, the files in the STR folder turned out to be videos in the standard PSX STR format, and the three files in the S1G folder turned out to be pictures, in the standard one again, as I later managed to google, for the Sonya TIM format. only with a different extension.
The NPK and LPK archives, which, as I assumed, contained the main content of the disc, did not manage to take a single program, including software focused on “Final Fantasy VII”.

The time has come to search for at least some information about these NPK archives, since, most likely, I am not the first to take them, and at least some mention should have been made.
The result turned out to be quite entertaining, trying this way and that and making requests and going through heaps of search rubbish, I could find only two references to these archives in the context of this disk.



Moreover, both mentions were on the same forum, related to, being in Down, at the time of this writing, the site dedicated to the software for modding to the games of the FF series.

One of the threads was an idle discussion on the topic “oh, how good it would be to tear out models” with a bunch of stubs instead of screenshots that the hoster removed a long time ago.
But the second thread , dating back to 2005, gave very good clues. In it, some user suggested chelendzh (oh, I “love” such freeloaders who want to get everything by someone else’s labor) to hack this same drive in order to tear out resources from it.

Qhimm (the owner of the forum and the author of many programs unpackers for FF) was blown up to study these archives, but it all ended somewhat suddenly ...

It turned out that another software developer for gutting PSX (snailrush) resources has already made a utility for unpacking these archives. It would seem “Hurray”, follow the link and download the utility, but if everything were so good, then this article would not exist. In general, the link was dead, and even archive.org did not help. The snailrush site went down many years, almost a few days after it made this damn utility, and no snapshots of that page survived.



Thus, I had only Qhimm's post (which for many years had not appeared on my forum), that he managed to understand the structure of these archives. And that meant that there was nothing particularly difficult about them.

We start to dig by yourself

It's time to open the files in the HEX editor. Immediately struck by some numbers at the very beginning, which indicated that the file has something like a header.

Here I would like to make a small digression. Despite the fact that at the very beginning I said that special knowledge is not necessary for digging into resources. There is one "small" exception. It is necessary to have at least a basic idea of ​​how exactly any data can be stored in memory or on a disk, in the simplest form - numbers. In other words, you need to know what Big Endian and Little Endian are .

I will try to explain the essence as briefly as possible:
Very often, numbers or characters are stored in memory in reverse order. For example, the number "16830", which in HEX looks like "41 BE", and being a variable of int type as "00 00 41 BE" in memory itself can be stored as "BE 41 00 00", I will not explain why this is so (you can google yourself), I’ll just say that "the computer is more convenient." And accordingly, in the same way, in the reverse order, the data can be stored in a file on disk.

And at first glance, almost all files, regardless of the extension, and LPK and TZL and FFD, had a very similar structure at the very beginning.

Several numbers of type int or short, then the sequence "DF 10 00 00 00 09", then some kind of a jumble of bytes, which were apparently data.





Since the first numbers of all the files were different, I assumed that these were most likely some kind of displacement or size of the internal blocks.

For a more detailed study, I decided to concentrate on one file and chose NPK_WEP.NPK, because from the thread it was clear that firstly it was possible to unpack it, and secondly it was known that it stores pictures in the TIM format.

It is time to take a closer look at the TIM format which I was going to look for inside.
Fortunately, TIM is well documented at the moment, including because Sony’s official documentation has leaked.
And the good news right away, TIM has a standard heading “10 00 00 00



I open at the same time in HEX with those 3 files that I already had in open form.



Wow, here in all of them "10 00 00 00 09" at the very beginning, just like in my archive.
It seems this cannot be a coincidence, here it is! Tim file is right inside, but why didn't automatic analyzers find it then? and why right after the “10 00 00 00 09” hash?
I try to search for “10 00 00 00 09” in the file and find 127 matches. Moreover, all places with "10 00 00 00 09" are similar to each other.

Here, for example, the second place in NPK_WEP.NPK where the sequence “10 00 00 00 09” was found:



After a heap of zeros, I saw the exact same structure as at the very beginning of the file.

So, now you can make some preliminary assumptions and conclusions:

1. If “10 00 00 00 09” is really a signature that speaks about the beginning of TIM files, then there are 127 such files in the archive, which is in general logical and similar to the truth.
2. The NPK_WEP.NPK archive, as such, does not have its own headers (which would describe its structure) and most likely is a jumble of somehow structured data.
3. TIM files (if they are anyway) are somehow compressed in the archive, since the data after the header “10 00 00 00 09” doesn’t look like real files.

Now I decided to study the structure of the NPK archive in more detail and find out what the numbers mean before the “10 00 00 00 09”.

The very first number was:
37 00 00 00> 00 00 00 37 (HEX)> 55 (DEC) It does not seem to be an offset, since offset 37 is already inside what I currently consider compressed data, so skip it.

The second number was short F7 03> 03 F7 (HEX)> 1015 (DEC)
The third is also a short F6 04> 04 F6 (HEX)> 1270 (DEC)
These two are already similar to the displacement, try to find something in the area of ​​these places.

Looking for 03 F7:



And bingo! The hike is exactly where the data block ends, since from this place there are zeros.
And what's the line below? 36 00 00 00 FD 03 2F 08 - It does not look like compressed data, but it is very similar to the same header as at the beginning.

Then what was the third offset? since it indicates the middle of the next block, I’ll assume for now that this is the size of the data after decompressing.

But now there is an assumption, which meant the very first number 37 (HEX) or 55 (DEC)
Apparently, this is the number of sub-blocks until the end of the packed file; for each next block it will decrease by 1. The check confirmed this assumption.

Only one thing remains unclear, if we have offsets that speak about the length of a block, then why not start the next block immediately after the end of the previous one, why this whole bunch of zeros? Besides, the beginning of the second block 0x0400 does not indicate anything, then how to understand where the next block begins? On the other hand, the 0400 is a very suspicious bias, too even. I wonder how much it is in DEC?



Wow! exactly 1024, this is hardly a coincidence. let's see what we have through another 1024 bytes at 0x800 offset:



Now everything is clear, the size of each sub-block is 1 kilobyte.

So, the structure NPK_WEP.NPK is almost completely decoded.
The entire file is divided into blocks of 1024 bytes.

The first 4 bytes of each block is the number of sub-blocks to the end of one file contained in the archive.

The next 2 bytes are the offset from the beginning of the sub-block to the end of the payload.
Another two bytes - most likely the size of the data after unpacking (apparently to allocate space for them in memory).

It seems to be true, it only remains to understand how the data itself is compressed or encrypted.

Understanding Compression

Before that, I had never encountered compression or encryption and had no idea or experience whatsoever.

Therefore, I began to look for information in two directions at once:
First, how to understand what compression is used in the file, if it is not known.
And secondly, what kind of compression is used most often for the PlayStation as a whole, or for SquareSoft games and the FF series in particular. The clue was that Qhimm quickly figured out the file, which meant it was something familiar to him.

Starting with a general search “as an eye to determine how a file is compressed,” the most useful thing I found is this list:
zenhax.com/viewtopic.php?t=27 , in which the eye caught on

lzss
Parts of the original data uncompressed.

that in general was like what I had.
And secondly, it was possible to immediately exclude any zip-related and other methods with some standard headers, since nothing similar to the header of the compressed data was observed.

The search for compression used in Final Fantasy VII led to the fallen site wiki.qhimm.com, which snapshots fortunately were available via the Wayback Machine.

It said that in FF VII, for example, two types of compression GZIP and LZS are used (some specific subspecies LZSS). I immediately dismissed GZip, as he was supposed to have a leader, but I decided to stop at LZSS in more detail, since everything pointed to him.
But then I made a mistake and I could get into “Wikipedia” (when editing the article I noticed that it was not Wikipedia, but another wiki site) to read about LZSS . And the problem was that one thing was the compression principle of the 80s described in textbook, and the other is its concrete implementation. I didn’t take it at all that, while preserving the principle of compression, it can be implemented in a bunch of different ways, but in textbooks, including most articles where it is described on the Internet, it is described as follows: 1 bit is read if this is 0, then the next 8 bits are read and entered as is to the buffer (output), and if it is 1, then the next 7 bits are read, part of which is offset in the buffer, and part is the length of the data that needs to be read from the buffer and put into the output.

Those. as if everything is clear and logical, but not a damn thing is similar to what I have, and even more so it does not correspond with the idea that the beginning of the data should look as if they are “not compressed”. (Well, simply because with this implementation all data would look like porridge, since the first byte would contain 1 signal bit and 7 bits of uncompressed data, the 8th bit of which would be the first bit of the second byte.)

I did not immediately realize that the specific implementation of the algorithm very much depends on the size of the buffer, and what is described in the textbooks is suitable for a buffer of negligibly small length.

In general, having lost several hours in an attempt to add 2 + 2, I decided to go on the other side and look for some implementation thread “in the code” of the compression used in FF VII.
And immediately stumbled upon github.com/cebix/ff7tools - a set of tools on python to translate ff7, which included lzss and unlzss. For verification, I decided to set lzss on one of my tim files. It turned out something like this:



At first glance - nothing good, but I clung to 09 and the sequence behind it, which painfully reminded me of what my files looked like in the archives. Here I was almost completely convinced that the pictures were compressed by some of the implementations of LZSS, it remains only to figure out which one.

For this, I returned to wiki.qhimm.com and decided to read more closely about “this is their LZS”.
After careful reading and re-reading - the puzzle has developed! BINGO!

The format is as follows:

The whole date is divided into blocks, the length of which can vary from 9 bytes to 17 bytes.
The first byte of the block indicates what all the other bytes of the block will be. This is encoded at the bit level. 1 means that the byte is sent as it is to the buffer and to the output. 0 - means that you need to read 2 bytes and they will be a reference to the buffer. And the reading bit comes from the end.
Thus, if the first byte of the block is FF, (11111111), this means that the next 8 bytes must simply be read as is and written to the output (buffer).
And, already native for me, the DF byte, facing 10 00 00 00 09, in bit form looks like 11011111, which means that the 5 bytes following it need to be read as they are (those 10 00 00 00 09), then There is a 2-byte link (EF F0), and again 2 bytes must be read as is (0C 02).
Further more interesting. The format of the link strongly depends on the size of the buffer. In this implementation, the buffer is 4 kilobytes, and the link in bits looks like this (again, fucking bits, I hate them already!)
OOOO OOOO OOOO LLLL (O = Offset, L = Length)
Well, or without bits, then OO OL (at the byte level), with the first 4 bits of the second byte being the first byte of the offset, because you need to read again from the end. But that's not all, you need to add three to the length, just such a hardcode. Because the lengths 0, 1 and 2 do not make sense, since in this case the link will take up more space than the data read from it. So 0 is 3, 1 is 4, and so on ...

I understand that it is difficult to understand, so I will explain with a real example:

The DF byte tells us that the 6th and 7th byte behind it is an offset, we have this EF F0.
We cut off the last 4 bits (length) and turn over, we get 0xFEF - offset in the buffer, and the remaining 0 + 3 = 3 - length.

Thus, the link tells us to read 3 bytes on the 0xFEF offset from the buffer. If you scroll higher and see what a normal TIM file looks like, then right after 09 there are three zero bytes, that’s what they are copied from the buffer. (Apparently, those 3 zero bytes that were there, literally “just”, are written previous actions).

Also, in that article, a complex formula was described for calculating the real offset that needs to be read from the buffer. This was explained by the fact that the author intended to use ordinary I / O streams for writing and reading, and the compression format implied the use of a ring buffer, and it was necessary to somehow somehow transform the offset.
But I was courageous, and decided that “well, what for him”, I’d better implement a ring buffer in myself than I will understand these strange formulas. Of course, I regretted it, but at the same time, when I began to understand why it “does not work,” I found a much more beautiful solution than the author of that article.

In addition, the article wrote about two "chips" that need to be borne in mind, references to the buffer can go both into the past and into the future, it was suggested to bypass all sorts of hacks.
As I later found out with the correct implementation of the buffer, no "hacks" are needed.

Writing Code

Well, for now, I uncovered Eclipse (I decided to write to JAVA, since it’s more convenient for me)
First you need to decide how to implement the ring buffer. The fact is that I immediately understood that it could be implemented in very different ways, so I decided to just try my luck and chose a similar implementation from the more or less standard standard library CircularFifoQueue from apache.commons.collections. It was a FIFO buffer that allows you to read the offset internally. Other implementations out of the box, most often on offset, were not allowed to read.
As a result, the code came out like this (I threw out all the garbage to simplify reading and shortening):

  static CircularFifoQueue<Byte> circularBuffer = new CircularFifoQueue<>(4096); private static void processNPK() { FileInputStream fis = new FileInputStream(new File("NPK_WEP.NPK")); ByteBuffer b1kb = ByteBuffer.allocate(1024); while (fis.read(b1kb.array()) != -1) { int blocksInMegaBlockCount = EndianUtils.swapInteger(b1kb.getInt()); int compressedDataLenght = EndianUtils.swapShort(b1kb.getShort()); int decompressedDataLenght = EndianUtils.swapShort(b1kb.getShort()); while (b1kb.position()<compressedDataLenght) { readLZSBlock(b1kb); } if (blocksInMegaBlockCount == 1) {dumpFile(result);} } } private static void readLZSBlock(ByteBuffer bb){ byte controlByte = bb.get(); for (int i = 0; i < 8; i++) { if (isLiteral(controlByte,i)) { byte b = bb.get(); circularBuffer.add(b); result.write(b); } else result.write(getFromCBuffer(bb)); } } private static byte[] getFromCBuffer(ByteBuffer bb) { byte first = bb.get(); byte second = bb.get(); Integer lowNibble = Integer.valueOf(second & 0x0f); Integer highNibble = Integer.valueOf(((second & 0xf0) >> 4)); int offset = EndianUtils.readSwappedShort(new byte[] {first, highNibble.byteValue()}, 0); byte[] bytesFromBuffer = new byte[lowNibble+3]; for (int i = 0; i < (lowNibble+3); i++) { byte b = circularBuffer.get(offset); circularBuffer.add(b); bytesFromBuffer[i]=b; } return bytesFromBuffer; } private static boolean isLiteral(byte controlByte, int position) { if (((controlByte >> position) & 1) == 1) return true; else return false; } 

You can describe his work as follows:

1) The file is cut into pieces of 1024 bytes,
2) For each read int, short, short
3) The remainder is transmitted to the loop with the unpacker (readLZSBlock)
4) It reads the first byte, finds out what to do with the rest, and either writes them as is, or, if it is a link, it gives it to the buffer reference handler (getFromCBuffer), which reads from the buffer.
5) When one file inside the archive is read, it is merged onto a disk.

The result of the execution was:



On the one hand, this is good, the outlines of the sword are visible, and the file has opened in the viewer. On the other hand, it is obvious that the nicher buffer does not work, you can guess that all the data that was read from it is garbage, and you can see how the zeros were at first and the top of the file is almost white, and then, as the buffer was filled, the garbage became colored. At the same time, the outlines of the sword are visible, which pleases.

I was not very upset, as I expected that the buffer would have to be finished. Just in case, I drove through all the other archives on the disk through the program and was surprised to find that the first file in the NPK_WEP.NPK archive was the only one that opened in the tim file viewer. So I was very lucky that I started with him.

The next step was an attempt to understand exactly where the failure occurred. For this, I, having armed myself with the description of the TIM format , manually began to go through my file in the HEX editor. The hope was that the failure would already be found in the headline and I would be able to find out exactly what the value should have been, and which one actually recorded, and knowing the status of the buffer, make some kind of correction.

A noticeable error was found on the 0x214 offset. in a normal TIM file, there should have been an int corresponding to the size of the data in it, given that the TIM files are not compressed by default and the size depends on the resolution, and the resolution in turn for the pictures of the playstation stretched across the entire TV screen is the same, I , .

Here is an example of a normal file:



Data size 00 01 E0 0C.

And here is my file:



instead of a normal number, I have some kind of garbage there.

Now, by changing the code a bit, I got the data related to the processing of this place.
The offset read from the file was 4092, length - 3 bytes.

This is how my buffer looked at the moment when it was necessary to get 00 01 E0 out of it



(Actually, this is the end, since the entire buffer did not even fit into my screen, but at the beginning there are just more zeros than in the picture.)

Finding Buffer the necessary data, I was confused.
The buffer was pushed 527 bytes of "useful" date. The desired one was found on the offset 15 from the beginning of the data entry, and approximately on the offset 3569 from the beginning of the buffer. How could they be considered on the 4092 offset issued to me, I could not understand. Also, remembering that the article mentioned the number 18, I tried to add and remove it from various offsets, but I did not get anything meaningful.

In summary, scratching the turnip, and realizing that the original buffer was arranged in a completely different way than mine, I returned to the article and a description of how to get a “real offset” when using reading from streams. The formula looked so awful that I just copied it as it is, without trying to penetrate and still attached a couple of extras of cars to it to pull it onto my buffer, in the end it turned out such a horror:

  //      int offset = EndianUtils.readSwappedShort(new byte[] {first, highNibble.byteValue()}, 0); //   ( OUTPOS -     ) int realoffset = OUTPOS - ((OUTPOS + 0xfee - offset) & 0xfff); //      " " int realoffset2 = 4096 - OUTPOS + realoffset; 

(Realoffset and realoffset2 could certainly have been reduced, but since I was not sure that it would work, I left it as it is for debugging.)

And oddly enough, it worked!



This could have stopped, but the implementation curve of the buffer did not give me rest. And most importantly, I wanted to better understand how the buffer actually worked, because it was obvious that the developers themselves didn’t have any dirty hacks when reading on an offset, and if they received an offset of 4092, then the necessary data were really located at 4092.

I confess that I thought for a long time, drew a lot on paper and arrays and rings to understand exactly how, and most importantly, why it was implemented exactly as it was.

The result was something like this:

 public class ByteRingBuffer { private byte[] rBuf; // ring buffer data public int rBufSize = 4096; // ring buffer size public int writeReadOffsetCorrection = 18; public int lenghtCorrectionLZS = 3; private int rBufWrPos = rBufSize - writeReadOffsetCorrection; // position of first (oldest) data byte within the ring buffer public ByteRingBuffer() { rBuf = new byte[rBufSize]; } public byte write(byte bt) { rBuf[rBufWrPos++] = bt; if (rBufWrPos == rBufSize) rBufWrPos = 0; return bt; } public byte read(int offset) { return rBuf[offset & 0xfff]; } } 

Simply put, the buffer was a regular byte array. Reading occurred from the beginning of the array, and the read marker never moved. The only hack was that the initial starting point for writing to the array was not 0, but the size of the array is minus 18. And everything worked as it should!
That is, it was enough to shift the recording point to 18 at the very beginning and after that you could safely read and write without any hacks. in this scenario, the 15th byte written to the array really turned out to be 4092 on the offset.

Well, besides, we didn’t need any hacks related to reading from the “past” or “future”.

The reason that the shift to 18 was generally necessary was most likely due to the fact that when the buffer is filled during compression, some of the data already in the buffer will be overwritten. And the following situation is possible:

1) Data intended for compression is searched in the buffer.
2) Found out there.
3) A link to them is created, and they are written to the buffer.
4) But they overwrite, in fact, themselves, and partly
5) And as a result, proper unpacking now becomes impossible.

But the surprises from the bonus disc did not end there either. As I moved to other archives, I discovered a strange “anomaly”.

Remember those blocks of 1024 bytes, the data in which took a little less space? I had a code that checked the rest of the block and issued a notification if there were not only zeros. On the first file - NPK_WEP.NPK, this piece of code never worked, but on the next and almost all the others, permanent warnings popped up. Since the files were unpacked correctly, I decided to see what was there for the “extra” data in the tails. And a little bit ofigel. Without hesitation, I added the code, which all the garbage from the archive dumped into a separate file, and this is what I found among this garbage:


Most likely, the unallocated useful info place, in the RAM, was dumped to disk "as is" from the memory, and contained the previous information. In addition to the source code, there was the usual garbage that you expect to see in the computer's memory, images of processes (including the executable file of the disk itself) and so on.

Other archives on the disk had either the exact same structure or a similar one, so there is no sense in analyzing them.

Just in case the link to the full version of my "govnokod".

That's all, I hope it was interesting to read and someone will come in handy.

PS: I apologize for a ton of spelling and punctuation errors, if you notice something absolutely blatant, then write and I will correct it.