Qhimm.com Forums
Miscellaneous Forums => Scripting and Reverse Engineering => Topic started by: Cyberman on 2007-02-20 02:07:55
-
The complete list of field locations (ONLY) is contained starting @ 0003A5B8 of the uncompressed data. This is the LSN and size of each file IN field.bin
0003A5B8 to 0003EF80 2 pairs of uint32 (little endian) Logical Sector Number then Size. Again Field files ONLY (this does not include BCX files in other words). 2361 files total for that. :D 0003F08C to 0003F0DC has the BCX files indexed.
I haven't found anything else YET but that's what this thread is for what's where in it.
Anyhow looks as though the game uses this index for loading field files (Akari keep that in mind ;) ).
Cyb
Edit 2/21/2007
FIELD.BIN is located at sector 55000 it's the first record in the \FIELD directory.
The first file listed in FIELD.BIN at offset 003A5B8 is DUMMY.DAT. The ordering of the files are thus DAT MIM then BSX. I'm not sure if this is related to the PC's ordering of the data for each field location (7 sections in the DAT + section 8 being the actual field data and section 9 being the BSX field character data or something like that). The final entry at 003EF78 is ZTRUCK.BSX. So this table covers JUST the field location files. The section from 003F08C to 003F0DC are the BCX files + the FIELD.TDB file (Texture data). This leaves 2 files not index and those are DSCHANGE.X and ENDING.X.
There are a number of uint32 numbers within the FIELD.BIN file that are suspicous when read in HEX. They appear to be offsets in the playstations memory. In addition the entire list of Opcodes are contained much like what was in the PC executable. To get to the point FIELD.BIN is likely the FIELD engine program that is loaded on top of the kernel. When porting the merged the FIELD engine section and KERNEL section into one ... conglomerate. :)
-
(Sorry about the thread necromancy...)
The first 4 byte of field.bin is the uncompressed size, the gzip header is at offset 8. I don't think there are multiple files in that bin, as the input buffer is empty when zlib is finished.
The file is decompressed to 0x0a0000 in memory whole. It's quite possible that it is is some kind of overlay/DLL instead of an archive. Does it exist in the PC version?
The normal table is at offset 0x3f520, which means the sine/cosine table are before that as well.
-
The first 4 byte of field.bin is the uncompressed size, the gzip header is at offset 8.
The first 4 bytes are more like the real size of the decompressed file, while the second WORD is the gzip decompression counter. They used a double counter to save room for empty bss segments (they can be recreated by simply filling the remaining size with zeroes). Other overlays like BATTLE.X use the same trick.
-
The final entry at 003EF78 is ZTRUCK.BSX. So this table covers JUST the field location files.
I just ran a complete file table against all words in field.bin, and it looks like the final entry is 3ef80,2903,280222,('WORLD/WM.PRE', 2903, 280222). There are a few more matches, but because the file size doesn't match for them I assume they're just random.
I'll see if I can find some more free time to see if any other files contain a file table.
world.bin: 0x273e4 - 0x2752C, txz files, bot files, bsz files, map files, ev files and field.tbl.
-
Does this mean that soon we could remaster a PSX disk with updated field files?
-
Does this mean that soon we could remaster a PSX disk with updated field files?
Aren't the fan-translation teams doing that anyway? Or are they just trying to inject files into the old sector positions?
But, yes, in theory if we know all file tables then you could patch them after creating the disk layout.
-
As long as all common files on disk share the same position (less MOVIE [in part], MINT, and SNOVA), regenerating the whole thing is very possible. Lasyan's tools rebuild the FIELD folder from scratch already.
-
I believe I read through field.Bin before (as noted I started the thread eons ago) and found several tables in it.
I suspect that the LBA table for the real file db is in there and I believe I found corelation to the files on the disk. I just had a system crash and it wiped all my work out is all. Have to love SIS motherboards (that's what killed my data a bad ATA interface on the board). This is the same machine with a different mother board works just fine without it ;)
Anyhow judging that the engine uses a series of numbers internally for field files it has to use something to translate that information into a physical location. I would hazard to guess that would FIELD.BIN for the rooms (which is the majority of the game content apart from movies). Battle scenes and characters have very little in the way of scripting. The majority of the script is in the \FIELD directory. That being said everything therefore must have an index for the script system FIELD.BIN seemed (or is?) the most likely candidate for the table for that.
Cyb
-
Apologies for resurrecting a dead thread, but I can't for the life of me seem to un-gzip (or gunzip) FIELD.BIN.
FIELD.BIN doesn't seem to make ff7dec very happy (it spits out an error stating that FIELD.BIN isn't a gzip file). I've followed the instructions and used the command:
ff7dec d field.bin
...but, no dice. I'm tempted to try it manually (destroy first 8 bytes, then treat it like a normal gzip file).
However, even after trying the "manual" method, it fails. All I get is
C:\Users\LB\Desktop\FF7 Modding Tools\ff7_gzip>ff7dec d field.bin
Creating file field.bin0!
gunzip: field.bin0.gz: not in gzip format
Creating file field.bin1!
gunzip: field.bin1.gz: not in gzip format
Creating file field.bin2!
gunzip: field.bin2.gz: not in gzip format
Creating file field.bin3!
gunzip: field.bin3.gz: not in gzip format
Creating file field.bin4!
gunzip: field.bin4.gz: not in gzip format
I'm using ff7dec (as mentioned above), but it doesn't seem to have a version number listed anywhere. It was last modified on 5/7/2003, and is 168 KB, if that's of any use as a distinction. Is there a newer (or older yet working) version somewhere, or have I incorrectly used the program?
Any help would be appreciated.
-
Running ff7dec doesn't sound like doing it manually to me. I always considered manual extraction to be using a regular gzip extractor on the separated gz files from the bin archive.
FIELD.bin is not like the other BIN archives. See here:
The first 4 byte of field.bin is the uncompressed size, the gzip header is at offset 8.
Compare that with the standard format (http://wiki.qhimm.com/FF7/Kernel/Low_level_libraries#BIN_archive_data_format). ff7dec probably fails because it wasn't designed to handle this situation.
-
Running ff7dec doesn't sound like doing it manually to me. I always considered manual extraction to be using a regular gzip extractor on the separated gz files from the bin archive.
I attempted to use 7Zip to extract it after deleting the first 8 bytes, as mentioned earlier in THIS (http://forums.qhimm.com/index.php?topic=6473.msg81408#msg81408) thread, but that failed as well.
Any advice on specifically how I should go about extracting FIELD.BIN?
-
That's pretty much the procedure: delete 8 bytes, rename to FIELD.gz, use any decompression program capable of detecting gzip files and you're done.
-
That's pretty much the procedure: delete 8 bytes, rename to FIELD.gz, use any decompression program capable of detecting gzip files and you're done.
Should I actually delete the first 8 bytes, or should I simply delete the values therein? In other words, should I be changing the file size here, or keeping it the same? I'd just test it right now, but I'm at work...
-
Delete=physically removing them from the file, so you'd obtain the old file size -8. What you've probably done is replacing the first 8 bytes with zeros, which definitively doesn't help in making any codec recognize the gzip signature. Delete those extra bytes, that's all you gotta do.
-
Delete=physically removing them from the file, so you'd obtain the old file size -8. What you've probably done is replacing the first 8 bytes with zeros, which definitively doesn't help in making any codec recognize the gzip signature. Delete those extra bytes, that's all you gotta do.
Nope, I was actually deleting the first 8 bytes (and thus changing the file size). HxD is pretty explicit about the distinction, and the file size was indeed decreased. Come to think of it, though...I may not have changed the extension to .gz, assuming that 7-Zip would detect the format automatically. I'll verify that tonight if possible (busy night).
-
I'm using this code to open field.bin:
import zlib, struct
def loadBin(name):
# read compressed data
f = open(name, "rb")
buf = f.read()
f.close()
(size,) = struct.unpack_from("<I", buf, 0)
data = zlib.decompress(buf[8:], 16 | 15)
if len(data) != size:
del buf
del data
return None
return data
It skips the first 8 byte and changes the window size.
http://docs.python.org/library/zlib.html#zlib.decompress
I'm not sure where I got the wbits value from, most likely from somebody else's document or decoder.
-
Heh, changing the file's extension to .gz was all that was required. I can't believe 7Zip didn't auto-detect the format. It does with Zip files and various other things.
Thanks for the help.
-
Also, to corroborate what Cyberman said many moons ago (the first post in this thread), FIELD (ungzipped) contains sector numbers referring to the FIELD files on disc. These are recorded in little-endian format (swap the bytes in each 'word'), and of course in hex. So, sector/LBA 58890 (MD1STIN.DAT) is recorded as 0AE6 (which, in big-endian, is E60A). Immediately before this in FIELD is F6E5, which in big-endian is E5F6, or decimal 58870; 58870 is the sector number/LBA for WHITEBG3.BSX, the file immediately before MD1STIN.DAT on the FF7 disc.
-
.
-
What do you mean by same position? If they are the same size as they originally were, if they have the original name, and if they are in the original directory?
No "same position" means where the bytes of data are located on the psychical disk. This is not always guaranteed when you go through a filesystem. When you delete a file and make a new one, most filesystems will mark the old file as "deleted", but not remove the data. Then, if you make a new file it will create the data somewhere else and "link" the directory entry to the new location...
FF7 does not use the directories, folders, or filenames to access data. That's why they got rid of them in later games. What happens is the FF7 kernel tells PSX where to put the laser head and reads the ones and zeros directly from the media.
I'm simplifying a little, but that's what's happening in a nutshell.
-
Jenovah's Witness: As halkun said, FF7 uses raw disc sectors to reference data, rather than file names and directories. Luckily, in this case, sectors are synonymous with LBA (Logical Block Address), so all we have to do to see what file is at what sector is fire up something like ISOBuster (or one of many other apps), load the disc image, and take a look at LBA for each file.
-
As a test, I deleted the first 8 bytes of FIELD.BIN, ungzipped it, noted the references to the three MD1STIN.* files (.DAT, .MIM, and .BSX), noted the references to the three MOGU_1.* files, and overwrote those of MD1STIN with those of MOGU_1. In other words, I told FIELD.BIN that when it wants to call MD1STIN, it should point to MOGU_1; this should make the game start at Mog House, rather than the Sector 1 train station.
I then gzipped it again, added the original 8 bytes to the beginning of the file again, renamed it to FIELD.BIN, launched CDMage, loaded a pristine copy of FF7 Disc 1 (as a Mode 2/2352-byte image), and imported FIELD.BIN directly over the top of the existing one. Probably due to different gzip implementations, my file was 2 KB smaller than the original, so CDMage asked if I'd like to pad the file with zeros, to which I responded no. It successfully imported the file (to the same LBA/sector as the original), and I saved the new image as a new file (in the same format I loaded it as, Mode 2/2352-byte image; I also told it to ignore sectors that couldn't be 'converted', as otherwise, it would immediately error out...I assume this is related to FMVs and Mode2/XA data or something). I then launched PCSX-Reloaded (a PSX emulator), loaded the ISO, and it worked fine. However, once I hit NEW GAME, the emulator crashed.
My first thought was that the new FIELD.BIN's different gzip compression could be making FF7 angry...quite possible, of course. So I tried simply loading a pristine copy of FF7 Disc 1 in CDMage, in the same way I did earlier, and then saving another copy, without modifying it. This yielded the same result. For some reason, something CDMage does when saving the image corrupts something, and this something isn't a problem until FIELD.BIN itself is loaded...even if FIELD.BIN isn't intentionally modified.
Does anyone have another program that will allow me to insert a file at a specific LBA, and won't cause the issue I'm experiencing?
-
What do you mean by same position? If they are the same size as they originally were, if they have the original name, and if they are in the original directory?
No, I mean they must share the same LBA and size on all three disks. There's no point in retaining the original size (and whatever other crap) if you're planning on regenerating an expanded ISO.
Probably due to different gzip implementations, my file was 2 KB smaller than the original
Check if the gzip signature is correct and if the header contains any extra information like time stamps and file names (FFVII doesn't like 'em), just in case.
It successfully imported the file (to the same LBA/sector as the original), and I saved the new image as a new file (in the same format I loaded it as, Mode 2/2352-byte image;
You don't save a new image after injecting a file, it's unnecessary. The ISO is already changed as soon as you confirm the injection.
-
You don't save a new image after injecting a file, it's unnecessary. The ISO is already changed as soon as you confirm the injection.
Negative; rather than choosing Save after making modifications, I chose Save As, and created a new image. This way, I could repeatedly test different output methods without having to use another copy of the image elsewhere on my hard drive or make copies multiple times. The "last modified" information for the original image I loaded in CDMage did not change - changes are not applied until you save them.
-
Luckily, in this case, sectors are synonymous with LBA (Logical Block Address)
Actually, not quite - LBA values are always 150 lower than the corresponding sector IDs, because the first 150 sectors of a PSX disc are used for license data. That means if you're hunting look ups, you have to account for the numeric difference.
-
Negative; rather than choosing Save after making modifications, I chose Save As, and created a new image.
You don't need to use Save, changes are instantly applied no matter what command you use after import. With Save As all you do is creating yet another identical copy of the ISO.
Actually, not quite - LBA values are always 150 lower than the corresponding sector IDs, because the first 150 sectors of a PSX disc are used for license data. That means if you're hunting look ups, you have to account for the numeric difference.
There's no difference to be taken into account, you were probably mislead by CDMage's sector scroller (in which sector 150=LBA 0=beginning of the ISO). Let's go by examples: LBA=0 means something is located at sector 0, byte 0; LBA=23 means the sector you need to seek to is the 23rd, byte 54188 (23*2352=54096, where 2352 is the size in bytes of each sector). Licensing data is much smaller than 150 sectors.
-
You don't need to use Save, changes are instantly applied no matter what command you use after import. With Save As all you do is creating yet another identical copy of the ISO.
Ack, you're right. If you choose Save As, it saves both the initial copy AND the new copy...ugh. I stand corrected.
-
Check if the gzip signature is correct and if the header contains any extra information like time stamps and file names (FFVII doesn't like 'em), just in case.
After resolving the issues I was having with importing files using CDMage, etc (and successfully importing various other files without issues), I've decompressed FIELD.BIN, recompressed it without making any modifications, then imported it back into FF7 Disc 1, and this still caused a crash after choosing NEW GAME. This leads me to believe that there indeed is an issue with the gzip header, or something else related to the gzip algorithm. I've tried using both 7Zip's gzip support and ff7dec and its bundled gzip, with the same results. Of course, I re-added the original 8 bytes that I stripped off in the first place, verbatim.
Is there anything in particular that I should be looking for? I've looked around at the gzip RFC, various articles concerning gzip headers and signatures, and general gzip documentation, but I haven't found anything that describes headers in such a way that I could hex edit and eliminate/modify them in a meaningful way...
-
Grab the command line gzip and make sure to use the proper options (should be a couple flags, IIRC) to exclude unwanted stuff from the header.
-
After resolving the issues I was having with importing files using CDMage, etc (and successfully importing various other files without issues), I've decompressed FIELD.BIN, recompressed it without making any modifications, then imported it back into FF7 Disc 1, and this still caused a crash after choosing NEW GAME. This leads me to believe that there indeed is an issue with the gzip header, or something else related to the gzip algorithm. I've tried using both 7Zip's gzip support and ff7dec and its bundled gzip, with the same results. Of course, I re-added the original 8 bytes that I stripped off in the first place, verbatim.
Is there anything in particular that I should be looking for? I've looked around at the gzip RFC, various articles concerning gzip headers and signatures, and general gzip documentation, but I haven't found anything that describes headers in such a way that I could hex edit and eliminate/modify them in a meaningful way...
I edited the gzip header like this and it worked ingame (tested on a emulator)
http://www.youtube.com/watch?v=ocE73qwTlgE
All i basically did was copy the header of the file from the original FIELD.BIN over to the new one compressed in gzip, hope this helps.
EDIT:- just rechecked, on the re-compressed Gzip all you have to do is add the original 8 bytes back into the file (the ones removed so we can decompress) and it should work (providing you also compressed the same way i did in the video above) so just make sure that when you use the command line Gzip you use
gzip -9 -n file
to compress the file (never use the 7zip gzip for this) then add the original 8 bytes back, this works for me (also when i added it into CDmage i also said YES to zero bytes when replacing)
-
Grab the command line gzip and make sure to use the proper options (should be a couple flags, IIRC) to exclude unwanted stuff from the header.
so just make sure that when you use the command line Gzip you use
gzip -9 -n file
to compress the file (never use the 7zip gzip for this) then add the original 8 bytes back, this works for me (also when i added it into CDmage i also said YES to zero bytes when replacing)
Yep, those gzip flags (-9 -n) did the trick :) I was already adding the 8-byte header back at the end (I mean, the end of the process...beginning of the file, heh), but apparently my gzip process was adding extra crap. Thanks.
I was able to recompress and run with an emulator without issue...however, the edits I attempted resulted in my getting a black screen and a repeating FF7 Error sound, when I chose NEW GAME. No matter, this just means I need to go back and check my work. And really, I have no need for editing FIELD.BIN quite yet...I just want the capability so that when the time comes, I can do so. Also, pointing FIELD.BIN directly to the first FIELD\*.DAT file I want to load, rather than hacking the first naturally-loaded one (MD1STIN.DAT) should be slightly faster, and much more correct...not that it would really matter.
Well, off to work. Thanks again for the help :)
-
.
-
Jenova's Witness: Yep; I've seen the Python docs on gzip. Also, before you posted that, you and Gemini had exactly the same number of posts :-o
Just an FYI for those interested: as far as I can tell, replacing the value of one field's LBA in FIELD.BIN with that of another field doesn't allow you to 'trick' FIELD.BIN into loading the field you want. Something seems to go wrong, and you just end up with a black screen and a repeated FF7 error sound. I've also tried replacing the LBA values for all three field files per field (.BSX, .MIM, and .DAT), to no avail.
This could be because FIELD.BIN isn't just choosing a file to load, but passing it arguments, etc...some of which may not make sense relative to a different field. I haven't been able to identify the exact cause yet, but I do know this: if I'm able to cause it to error out by changing the value of a field's LBA in FIELD.BIN, and it only errors out when I edit the field that's about to be loaded, that would imply that FIELD.BIN at least references that value before loading a field. Therefore, I'm on the right track toward modifying which field is loaded. In any case, jumping straight from MD1STIN.DAT to the field I really wish to load (using Makou Reactor or Hack7) isn't a huge problem, as long as I never want to use MD1STIN.DAT legitimately...and even if I do, I can always jam in a variable and If/Else to determine whether a new game has just been launched or not.
OK, back to work on that prequel mod, unless someone has something else to add.
-
I think field.bin stores both position and size... it has been ages since I looked at it, though. You could try to match the whole table of position/size against the disc image and the directory to confirm.
-
I think field.bin stores both position and size...
That's right.
-
That's right.
It's been eons since I've fiddled with this stuff.
Just don't have as much time as I use to.. Anyhow haven't seen you in a while.
FF7 may be doing a check to see if the data is correct when extracted. As the others have noted there is a size field as well as the LBA. If the size doesn't exactly match the real file size this will indicate it could be the uncompressed size. I don't remember which too be honest it's been quite a while.
I suggest you verify what lasyan3 and Mickey said by being sure you change the size information too match the new location.
It may be helpful to create a tool to read the table data and allow you to swap the data around with LBA and size information together.
So looking at it logically the table is 6 x N locations (6 columns N locations for rows)
IE
LBA | SIZE | LBA | SIZE | LBA SIZE
If you want to swap the data or have a row point to another location it's probably a good idea to swap the whole row or change the whole row (6 values instead of 3).
I don't remember anything about data checks such as CRC's etc. for the files, for all I remember there may be.
I can't remember where the size data is however, I'm so tired these days it's sad :D
Cyb