FF7PSX\\FIELD

[Kenyai]

Hey, everyone. I've been visiting this forum for some time now without registering or posting. But today I decided I should go ahead and appear out of nowhere.

You guys are incredible! On my own, I would never have been able to figure out LZS decompression, let alone the compression (which I still know nothing about). I must thank Ficedula immensely for his helpful documents. However, I felt like a leech by reading all these threads without adding any of my own contributions. I probably don't have a lot to add that isn't already known, but whatever. I think it's better to give out well-known information than to not give out unknown information.

These are the functions I wrote for getting the size of a decompressed LZS file and actually decompressing the file. If you see errors in these, please inform me, but they have worked fine for me, and the file sizes match those from Fice's LZS decompressor.

Code: [Select]

DWORD LZSG(BYTE *lzs, DWORD li)
{
BYTE cb=0;
DWORD cs=0;
for(DWORD ci=0;ci<li;){
cb=lzs[ci++];
for(BYTE bt=0;bt<8&&ci<li;bt++,cb>>=1){
if(cb&1){cs++;ci++;}
else{ci++;cs+=(lzs[ci++]&0xF)+3;}}}
return(cs);
}

void LZSD(BYTE *dec, BYTE *lzs, DWORD li)
{
BYTE cb=0,cl;
DWORD co=0;
for(DWORD ci=0;ci<li;){
cb=lzs[ci++];
for(BYTE bt=0;bt<8&&ci<li;bt++,cb>>=1){
if(cb&1)dec[co++]=lzs[ci++];
else{
int o1;
cl=(lzs[++ci]&0xF)+3;
o1=((lzs[ci]&0xF0)<<4)+lzs[(ci++)-1];
o1=co-((co-o1-18)&0xFFF);
for(;o1<0&&(cl--);o1++)dec[co++]=0;
for(;cl--;)dec[co++]=dec[o1++];}}}
}

First of all, the only game I'm focusing on is FF7. I only have the PSX US version of the game, and that's what I've been working on. I suppose the only real files I've been concentrating on are the FIELD directory files. As I found out, there are differences between this version of the game and the well-documented PC version. This is what I know about the decompressed files.

BSX - I have no idea about these files. Haven't looked all that much.

MIM - I've looked at these guys a lot, and it seems to me that all they contain is an 8BPP TIM of the "unconstructed" tiled picture for the field backgrounds. I call this the raw background TIM. Anyway, the MIM skips the ID tag DWORDs for TIM and 8BPP. That is, if you look at the 8BPP TIM file structure starting with the DWORD containing the size of the CLUT section, that right there is the structure of the MIM file. However, I always saw 4 extra 00 bytes at the end of the file for some reason. Now, it seems that there are subsections of the raw background TIM. The pixels with an x lower than 256 are all related and share a CLUT. The pixels with an x from 256 to 512 share the next CLUT, etc. However, there is a problem. It's not always 256. Sometimes the sections are only 128 pixels wide. I cannot find in the DAT where it says how wide each section of the raw TIM is!

DAT - I got frustrated with these guys for a long time. After probably a week of staring at the file in a hex editor, I figured out how to always get the exact location of where the text starts. There is a word at offset 0x20 that stores an offset to the text section header, but you have to add 0x1C to it. (Yes, I found this out independently somehow, but am now aware it is already written somewhere.) It seems like this will be easier to describe in code.

Code: [Select]

ifstream datfile;
WORD offset, numstr, headsize;
datfile.open(".\\FILE.DAT", ios::in | ios::binary);
datfile.seekg(0x20);
datfile.read((char *)&offset, 2);
datfile.seekg(offset + 0x1C);
datfile.read((char *)&numstr, 2);
datfile.read((char *)&headsize, 2);
datfile.seekg(offset + 0x1C + headsize);
//You may now begin reading text. The number of strings should be numstr.

More recently, I figured out what the first 0x1C bytes are for. They're offsets to the sections of the DAT (seems PSX has different sections than the PC; I only see 7 sections here) but there's a couple of really weird things going on. First, there's a useless 0x8011 WORD every other WORD. Second, looking at the even WORDs instead, you have to subtract 0x5000 from their value and then add 0x1C to get correct offsets in the file. I think the background construction section where it tells you how to load the background picture using the MIM is the third such offset, the WORD starting at offset 0x8 in the file.

Code: [Select]

ifstream datfile;
WORD offset;
DWORD headsize;
datfile.open(".\\FILE.DAT", ios::in | ios::binary);
datfile.seekg(0x8);
datfile.read((char *)&offset, 2);
datfile.seekg(offset - 0x5000 + 0x1C);
datfile.read((char *)&headsize, 4);
datfile.seekg(offset - 0x5000 + 0x1C + headsize);
//This is where the first construction section starts.

I don't know what the header before the image construction does. There seems to be more than one DWORD besides headsize. If you read the one after that instead and add it like you did there, it takes you to a differently structured section, but it still seems to be image construction. Anyway, for the first image construction section you should see this repetition... (Note that like in the PC version, the destination's (0, 0) is the middle of the screen.)

Code: [Select]

short dstx;
short dsty;
BYTE srcx;
BYTE srcy;
WORD section;

If you take (section - 0x7800) / 64 you should get which section to get the tile from. I haven't figured out any more image construction sections. As of now, the only way I can figure to get the final background width is to look at every dstx in the image construction section and look for the largest x and smallest x, then find (lrgx - smlx + 16) as the width. Same method for height. But this isn't wide/high enough sometimes. I guess sprites can stick out past the bottom layer sometimes? If there is an easier way to find the image dimensions, please let me know!

I think that's it for this post. If you see errors anywhere, correct them as you wish in a reply. If you can help me with something I didn't know about, by all means, go ahead. I am not too proud to accept help. If this post was long AND useless, I apologize, but I know I couldn't find anything this detailed about these files for the PSX version when I searched through this forum. I hope this helps people.

[Darkdevil]

First things first, Welcome to the forum!

Second things first, I have to say....That look DAMN impressive from where im sitting, but then again, im not really good at programming/ reverse engineering.

We owe All the progress made to the people who live in this board really...They have done *so* much work on the game its unreal.  

Halkun, Terrence and everyone else.  I SALUTE YOU!

heh

[Kenyai]

Thanks for the welcome!

EDIT: My LZSD function used to display incorrectly, but I improved the function and now it shows up correctly. If you want to know the details of what code used to be there, just ask me.

[halkun]

Welcome welcome!

So the MIM files only hold the picture background layers with the CLUT data?

Is it just a collection of TIM files? (Multi-tIM)

I know that when field backgrounds are loaded, each "unconstructed" layer is placed into a 256x256 "slot" in video memory. After that any animated sprites are loaded after that in thier own 256x256 slots. When the background image is displayed each square is blitted from off-screen video memory to the view window, starting with the bottom layer, then the 3d characters are drawn, and the upper layer is drawn on that, painting over the character that's "behind" something.

IF you are running epsxe on a windows PC, run the game in 1024x512. Then hit the F12 key so you can see the vram while you play the game.

The Gold saucer Tram station. (The one inside the GS) has a really simple CLUT and when you look at the video memory, the colors are not using a broken "pallet" so you can see how the MIM file looks in VRAM

I was hoping that the walkmesh and camera data was held in the MIM file there. I wonder, if DAT files only contain the script, that leaves the walkmesh in the BSX file if it does.

I have a question, as you are hacking the PSX data, how are you altering your data to test your therioes? Are you using the save state/memory alter think with emulation, or are you flipping bits on an ISO image?

Also could you create a quick table loosely showing which sections of a PC field file corrispond to the section in BSX/MIM/DAT ?

I'm thinking about woriking on the Field section of Gears after I finish poking around in Menu and giving Kernal an overhaul

Being a semi-literate programmer, I do have problems reading code. For example I would love to add the PSX movie format to the Field section, but don't know how to read the decompressor's code.

[Cyberman]

Halkun the BSX files contain NPC character models like the BCX files contain PC character models.  I think they contain the animations as well.

MIM are the background image files, I didn't think they were decapitated tims. Hmmm I'll have to look at it some more.

DAT are the script and I think other things as well.

The BSX file is likely to contain the walkmesh because there are places where there are no NPC's but there is ALWAYS a BSX file.

Cyb

[Anakin]

Question:
Isn't it a bit of waste of time/typing, etc to have the NPCs coded de novo in the respective BSX file every time they need to be present in a scene?
Why would you think is that?
Does anybody have any idea?
Or in other words Are you sure the NPCs are coded in BSX again and again? I remember that when I tried to compare BSXs with Hexedit I found very small identical pieces in scenes that actually contained the same NPCs (for example the scenes of Nivelheim entrance)

[Kenyai]

Ah, yes, I think you're right about the MIM being a collection of TIMs, halkun. Looks like there can be more than one 8BPP image data section in one MIM. But it seems there's only one CLUT section per file, which is at the beginning.

This may be pretty clumsy on my part, but I haven't used a PSX emulator at all to test these theories. However, I did it other ways.

To test my theory about the method of finding where the text starts, I created a program to extract the text from the PSX DAT files, and it looks like it got it right, as there were no unreadable characters before the text started. Also, I looked at the DATs in the hex editor, followed the method manually, and it also checked out then. I don't know if this is known or not, but I think at least on the PSX, the 0xA9 byte translates to ... in the dialogue.

As for the background image construction, I admit I'm a lot less certain about how it all works. I was trying to start out simple by seeing if I could load simple backgrounds like in JUNON, ignoring sprite/animation layers entirely for now. I wrote a program to do this, and it exported the JUNON background to a bitmap file perfectly. When I tried it on GONGAGA, it seemed to work there as well. But JUNONL1 gave me trouble. When I altered my additions to consider that some sections might be 128 pixels wide instead of the standard 256x256 blocks, the image turned out perfectly in the bitmap file. Of course, since it was only the bottom layer, there was a lack of coloring where the door sprites should be, etc. NIVL also didn't work properly if I assumed 256x256 blocks.

As for viewing all these files in VRAM, I wrote a program for that too. I can choose any file to load as a picture, then I can alter the BPP and the width until it works out. That's how I found the image data in the MIMs. When I was in 8BPP, I recorded the correct width of the image and when I looked in the file, I saw half that number consistently in the same area.

I haven't looked at the videos yet, but I know there are players for them already out there, so it's not impossible to decipher them. I vaguely remember something about XA being mentioned in regards to them...

Now, comparing the PC and PSX formats... I don't have the PC version at all, but looking at Fice's document on the PC field file structure...

Section 1 - For the dialogue on the PSX, it's found in the DAT as I described above. If there is more to this section, I don't know where that's stored on PSX... Unless this is the one with the author name and the names of characters who can lead the party on that field level. That stuff is at the beginning of the PSX DAT file.

Section 2-3 - I dont know what these are in PC.

Section 4 - Palette data for the background is stored in the MIMs.

Section 5-8 - I don't know what these are in PC.

Section 9 - The 8BPP image data is in the MIMs, but I don't see those 6 byte headers before tiles, at least in the bottom layer with no sprites or animations. If the 6 byte headers are only for sprite tiles, that would explain this. The information to construct the image is located in the DAT file at the offset stored at 0x8.