FIELD.BIN (exploration) thread

[Cyberman]

The complete list of field locations (ONLY) is contained starting @ 0003A5B8 of the uncompressed data.  This is the LSN and size of each file IN field.bin
0003A5B8 to 0003EF80 2 pairs of uint32 (little endian) Logical Sector Number then Size.  Again Field files ONLY (this does not include BCX files in other words).  2361 files total for that.   0003F08C to 0003F0DC has the BCX files indexed.
I haven't found anything else YET but that's what this thread is for what's where in it.
Anyhow looks as though the game uses this index for loading field files (Akari keep that in mind ).

Cyb

Edit 2/21/2007
FIELD.BIN is located at sector 55000 it's the first record in the \FIELD directory.
The first file listed in FIELD.BIN at offset 003A5B8 is DUMMY.DAT.  The ordering of the files are thus DAT MIM then BSX.  I'm not sure if this is related to the PC's ordering of the data for each field location (7 sections in the DAT + section 8 being the actual field data and section 9 being the BSX field character data or something like that).  The final entry at 003EF78 is ZTRUCK.BSX. So this table covers JUST the field location files.   The section from 003F08C to 003F0DC are the BCX files + the FIELD.TDB file (Texture data). This leaves 2 files not index and those are DSCHANGE.X and ENDING.X.

There are a number of uint32 numbers within the FIELD.BIN file that are suspicous when read in HEX. They appear to be offsets in the playstations memory.  In addition the entire list of Opcodes are contained much like what was in the PC executable.  To get to the point FIELD.BIN is likely the FIELD engine program that is loaded on top of the kernel.  When porting the merged the FIELD engine section and KERNEL section into one ... conglomerate.

[Micky]

(Sorry about the thread necromancy...)
The first 4 byte of field.bin is the uncompressed size, the gzip header is at offset 8. I don't think there are multiple files in that bin, as the input buffer is empty when zlib is finished.
The file is decompressed to 0x0a0000 in memory whole. It's quite possible that it is is some kind of overlay/DLL instead of an archive. Does it exist in the PC version?

The normal table is at offset 0x3f520, which means the sine/cosine table are before that as well.

[Gemini]

The first 4 byte of field.bin is the uncompressed size, the gzip header is at offset 8.

The first 4 bytes are more like the real size of the decompressed file, while the second WORD is the gzip decompression counter. They used a double counter to save room for empty bss segments (they can be recreated by simply filling the remaining size with zeroes). Other overlays like BATTLE.X use the same trick.

[Micky]

The final entry at 003EF78 is ZTRUCK.BSX. So this table covers JUST the field location files.

I just ran a complete file table against all words in field.bin, and it looks like the final entry is 3ef80,2903,280222,('WORLD/WM.PRE', 2903, 280222). There are a few more matches, but because the file size doesn't match for them I assume they're just random.
I'll see if I can find some more free time to see if any other files contain a file table.

Code: [Select]

world.bin: 0x273e4 - 0x2752C, txz files, bot files, bsz files, map files, ev files and field.tbl.


[halkun]

Does this mean that soon we could remaster a PSX disk with updated field files?

[Micky]

Does this mean that soon we could remaster a PSX disk with updated field files?

Aren't the fan-translation teams doing that anyway? Or are they just trying to inject files into the old sector positions?
But, yes, in theory if we know all file tables then you could patch them after creating the disk layout.

[Gemini]

As long as all common files on disk share the same position (less MOVIE [in part], MINT, and SNOVA), regenerating the whole thing is very possible. Lasyan's tools rebuild the FIELD folder from scratch already.

[Cyberman]

I believe I read through field.Bin before (as noted I started the thread eons ago) and found several tables in it. 

I suspect that the LBA table for the real file db is in there and I believe I found corelation to the files on the disk. I just had a system crash and it wiped all my work out is all. Have to love SIS motherboards (that's what killed my data a bad ATA interface on the board). This is the same machine with a different mother board works just fine without it

Anyhow judging that the engine uses a series of numbers internally for field files it has to use something to translate that information into a physical location.  I would hazard to guess that would FIELD.BIN for the rooms (which is the majority of the game content apart from movies). Battle scenes and characters have very little in the way of scripting.  The majority of the script is in the \FIELD directory.  That being said everything therefore must have an index for the script system FIELD.BIN seemed (or is?) the most likely candidate for the table for that.

Cyb

[Lazy Bastard]

Apologies for resurrecting a dead thread, but I can't for the life of me seem to un-gzip (or gunzip) FIELD.BIN.

FIELD.BIN doesn't seem to make ff7dec very happy (it spits out an error stating that FIELD.BIN isn't a gzip file). I've followed the instructions and used the command:

Code: [Select]


ff7dec d field.bin

...but, no dice. I'm tempted to try it manually (destroy first 8 bytes, then treat it like a normal gzip file).

However, even after trying the "manual" method, it fails. All I get is

Code: [Select]

C:\Users\LB\Desktop\FF7 Modding Tools\ff7_gzip>ff7dec d field.bin

Creating file field.bin0!
gunzip: field.bin0.gz: not in gzip format

Creating file field.bin1!
gunzip: field.bin1.gz: not in gzip format

Creating file field.bin2!
gunzip: field.bin2.gz: not in gzip format

Creating file field.bin3!
gunzip: field.bin3.gz: not in gzip format

Creating file field.bin4!
gunzip: field.bin4.gz: not in gzip format
I'm using ff7dec (as mentioned above), but it doesn't seem to have a version number listed anywhere. It was last modified on 5/7/2003, and is 168 KB, if that's of any use as a distinction. Is there a newer (or older yet working) version somewhere, or have I incorrectly used the program?

Any help would be appreciated.

[Vehek]

Running ff7dec doesn't sound like doing it manually to me. I always considered manual extraction to be using a regular gzip extractor on the separated gz files from the bin archive.

FIELD.bin is not like the other BIN archives. See here:

The first 4 byte of field.bin is the uncompressed size, the gzip header is at offset 8.

Compare that with the standard format. ff7dec probably fails because it wasn't designed to handle this situation.

[Lazy Bastard]

Running ff7dec doesn't sound like doing it manually to me. I always considered manual extraction to be using a regular gzip extractor on the separated gz files from the bin archive.

I attempted to use 7Zip to extract it after deleting the first 8 bytes, as mentioned earlier in THIS thread, but that failed as well.

Any advice on specifically how I should go about extracting FIELD.BIN?

[Gemini]

That's pretty much the procedure: delete 8 bytes, rename to FIELD.gz, use any decompression program capable of detecting gzip files and you're done.

[Lazy Bastard]

That's pretty much the procedure: delete 8 bytes, rename to FIELD.gz, use any decompression program capable of detecting gzip files and you're done.

Should I actually delete the first 8 bytes, or should I simply delete the values therein? In other words, should I be changing the file size here, or keeping it the same? I'd just test it right now, but I'm at work...

[Gemini]

Delete=physically removing them from the file, so you'd obtain the old file size -8. What you've probably done is replacing the first 8 bytes with zeros, which definitively doesn't help in making any codec recognize the gzip signature. Delete those extra bytes, that's all you gotta do.

[Lazy Bastard]

Delete=physically removing them from the file, so you'd obtain the old file size -8. What you've probably done is replacing the first 8 bytes with zeros, which definitively doesn't help in making any codec recognize the gzip signature. Delete those extra bytes, that's all you gotta do.

Nope, I was actually deleting the first 8 bytes (and thus changing the file size). HxD is pretty explicit about the distinction, and the file size was indeed decreased. Come to think of it, though...I may not have changed the extension to .gz, assuming that 7-Zip would detect the format automatically. I'll verify that tonight if possible (busy night).

[Micky]

I'm using this code to open field.bin:

Code: [Select]

import zlib, struct

def loadBin(name):
    # read compressed data
    f = open(name, "rb")
    buf = f.read()
    f.close()
   
    (size,) = struct.unpack_from("<I", buf, 0)

    data = zlib.decompress(buf[8:], 16 | 15)
    if len(data) != size:
del buf
del data
return None
    return data
It skips the first 8 byte and changes the window size.
http://docs.python.org/library/zlib.html#zlib.decompress
I'm not sure where I got the wbits value from, most likely from somebody else's document or decoder.

[Lazy Bastard]

Heh, changing the file's extension to .gz was all that was required. I can't believe 7Zip didn't auto-detect the format. It does with Zip files and various other things.

Thanks for the help.

[Lazy Bastard]

Also, to corroborate what Cyberman said many moons ago (the first post in this thread), FIELD (ungzipped) contains sector numbers referring to the FIELD files on disc. These are recorded in little-endian format (swap the bytes in each 'word'), and of course in hex. So, sector/LBA 58890 (MD1STIN.DAT) is recorded as 0AE6 (which, in big-endian, is E60A). Immediately before this in FIELD is F6E5, which in big-endian is E5F6, or decimal 58870; 58870 is the sector number/LBA for WHITEBG3.BSX, the file immediately before MD1STIN.DAT on the FF7 disc.

[Jenova's Witness]

.

[halkun]

What do you mean by same position? If they are the same size as they originally were, if they have the original name, and if they are in the original directory?

No "same position" means where the bytes of data are located on the psychical disk. This is not always guaranteed when you go through a filesystem. When you delete a file and make a new one, most filesystems will mark the old file as "deleted", but not remove the data. Then, if you make a new file it will create the data somewhere else and "link" the directory entry to the new location...

FF7 does not use the directories,  folders, or filenames to access data. That's why they got rid of them in later games. What happens is the FF7 kernel tells PSX where to put the laser head and reads the ones and zeros directly from the media.

I'm simplifying a little, but that's what's happening in a nutshell.

[Lazy Bastard]

Jenovah's Witness: As halkun said, FF7 uses raw disc sectors to reference data, rather than file names and directories. Luckily, in this case, sectors are synonymous with LBA (Logical Block Address), so all we have to do to see what file is at what sector is fire up something like ISOBuster (or one of many other apps), load the disc image, and take a look at LBA for each file.

[Lazy Bastard]

As a test, I deleted the first 8 bytes of FIELD.BIN, ungzipped it, noted the references to the three MD1STIN.* files (.DAT, .MIM, and .BSX), noted the references to the three MOGU_1.* files, and overwrote those of MD1STIN with those of MOGU_1. In other words, I told FIELD.BIN that when it wants to call MD1STIN, it should point to MOGU_1; this should make the game start at Mog House, rather than the Sector 1 train station.

I then gzipped it again, added the original 8 bytes to the beginning of the file again, renamed it to FIELD.BIN, launched CDMage, loaded a pristine copy of FF7 Disc 1 (as a Mode 2/2352-byte image), and imported FIELD.BIN directly over the top of the existing one. Probably due to different gzip implementations, my file was 2 KB smaller than the original, so CDMage asked if I'd like to pad the file with zeros, to which I responded no. It successfully imported the file (to the same LBA/sector as the original), and I saved the new image as a new file (in the same format I loaded it as, Mode 2/2352-byte image; I also told it to ignore sectors that couldn't be 'converted', as otherwise, it would immediately error out...I assume this is related to FMVs and Mode2/XA data or something). I then launched PCSX-Reloaded (a PSX emulator), loaded the ISO, and it worked fine. However, once I hit NEW GAME, the emulator crashed.

My first thought was that the new FIELD.BIN's different gzip compression could be making FF7 angry...quite possible, of course. So I tried simply loading a pristine copy of FF7 Disc 1 in CDMage, in the same way I did earlier, and then saving another copy, without modifying it. This yielded the same result. For some reason, something CDMage does when saving the image corrupts something, and this something isn't a problem until FIELD.BIN itself is loaded...even if FIELD.BIN isn't intentionally modified.

Does anyone have another program that will allow me to insert a file at a specific LBA, and won't cause the issue I'm experiencing?

[Gemini]

What do you mean by same position? If they are the same size as they originally were, if they have the original name, and if they are in the original directory?

No, I mean they must share the same LBA and size on all three disks. There's no point in retaining the original size (and whatever other crap) if you're planning on regenerating an expanded ISO.

Probably due to different gzip implementations, my file was 2 KB smaller than the original

Check if the gzip signature is correct and if the header contains any extra information like time stamps and file names (FFVII doesn't like 'em), just in case.

It successfully imported the file (to the same LBA/sector as the original), and I saved the new image as a new file (in the same format I loaded it as, Mode 2/2352-byte image;

You don't save a new image after injecting a file, it's unnecessary. The ISO is already changed as soon as you confirm the injection.

[Lazy Bastard]

You don't save a new image after injecting a file, it's unnecessary. The ISO is already changed as soon as you confirm the injection.

Negative; rather than choosing Save after making modifications, I chose Save As, and created a new image. This way, I could repeatedly test different output methods without having to use another copy of the image elsewhere on my hard drive or make copies multiple times. The "last modified" information for the original image I loaded in CDMage did not change - changes are not applied until you save them.

[Bosola]

Luckily, in this case, sectors are synonymous with LBA (Logical Block Address)

Actually, not quite - LBA values are always 150 lower than the corresponding sector IDs, because the first 150 sectors of a PSX disc are used for license data. That means if you're hunting look ups, you have to account for the numeric difference.