Figuring out the field files

[Discostew]

I've browsed the forums here the past couple of months, but decided that I needed some help, so this is my first post here.

I've been working on and off on my own personal FF7 project, using the files/data from the PSX version, and while I've got a good number of the stuff for displaying fields working to an extent, there are a few things I'd like to ask.

First of all, I know that the PSX doesn't have a z-buffer in it, so I am assuming that everything is rendered via painter's algorithm (whatever is considered the furthest away is drawn first, and the closest is drawn last, showing up above the rest). The tilemap section of the DAT files has it's own individual sections pertaining to drawing the field, as structured here, but how does it know whether a 3D model is in front or behind a tile? is there a particular variable that contains the tile's "depth" on the screen, or something to help it sort it out? The wiki is somewhat old, and I've already seen some differences in a few of the structures.

Also, some files are 2Kbytes big, and look to be uncompressed, but I have no idea how to work with those.

I'll be hanging around here, and will more than likely ask more questions, but this is good for a start. Thanks in advance.

[Aali]

Look for a fixed-point value, divided by either 10000000 (as is the case for the PC version) or 4096 (another common fixed-point format used in FF7/FF8).
If you find something that for *most tiles* adds up to a number near 1.0 (0.99 or 0.98) that's your Z coordinate.

:EDIT:
You might want to take a look at the q-gears source, I think it does this properly for PSX data already.

[Discostew]

Thx. I'll take a look at it.

[Discostew]

Alright, I've been scanning what I could with the source, but I found nothing that helped solve the mystery with the 2Kbyte files (such as both DAT and MIM files of BLIN69_2, a few BLACKBG, and many WM files). I'm not sure if they are compressed files that are different from LZS, or actual uncompressed data. If it is the latter, how would I know where to place it, as it looks like there are no pointers in it that designate the separate sections?

[Discostew]

Did some more looking around with the 2KByte files in the FIELD directory, and I still haven't figured it out, though there are some files that I assume aren't part of the normal field files (such as the wm files which must be related to the world map). The ones that have their names in the first 8 bytes of those files have almost exact data that is 764 bytes long, after the first 24 bytes (doesn't matter the filename or type), which include multiple values of 0xFF7F, 0xFF00, or 0x0101. After the first 780 bytes, the data following is exact to the data of other files under that same name (if they are also 2KByte in size). The others contain a small value at the first 4 bytes, of which afterwards the data alternates every dword (4 bytes) between some value, or 0.

It just doesn't make any sense to me, nor do they look structured like LZS files. Even looking in the q-gears source, any place that references loading a map also involves extracting them as if they are lzs files, whether or not the file size (not including the first 4 bytes) matches the first 4 bytes in the file itself. Are they being loaded some other way?

[Vehek]

On his blog, VanishedOne regarded those tiny files as unused. They (files like blin69_2) don't even exist in the PC version.

[VanishedOne]

On his blog, VanishedOne regarded those tiny files as unused.

Well, it's not as though I've made certain the game never reads them; but based on the filenames, the fact that at least one map ID seems to have no corresponding map in the final version (the unused 'From pinball' scene crashes because the game tries to jump to a map that won't load), and the uniform file sizes (compare SaGa Frontier, which has loads of dummy map files that are just wavy lines and ASCII copyright notices), it looks like the likely explanation, with the possible exception of the WM#s.

That said, if you look at the 2KB FALLP in 7mimic, it does show a still from the parachute scene...

[Discostew]

That said, if you look at the 2KB FALLP in 7mimic, it does show a still from the parachute scene...

Which is interesting because that seems to be one of the few file groups that the DAT file is 2KByte in size, but the MIM is bigger, and is LZS-compressed. In fact, these kind of file groups in the DAT files each have a small value as the first dword, then follows with alternating values of non-zero and zero valued dwords, or with a bit more structure with what looks like actual 8-byte tile data at the end, uncompressed.

[halkun]

Lookup tables maybe? For the data on the disk?

[Micky]

First of all, I know that the PSX doesn't have a z-buffer in it, so I am assuming that everything is rendered via painter's algorithm (whatever is considered the furthest away is drawn first, and the closest is drawn last, showing up above the rest). The tilemap section of the DAT files has it's own individual sections pertaining to drawing the field, as structured here, but how does it know whether a 3D model is in front or behind a tile? is there a particular variable that contains the tile's "depth" on the screen, or something to help it sort it out? The wiki is somewhat old, and I've already seen some differences in a few of the structures.

Is there a chance you could post the differences you found? I just had a look at my PSX field viewer again, and with some very tricky coding I can now display quite a lot of fields. One difficulty seems to be that some background reference more tiles than the number that is actually in the file.
Just to make a start:
Section 1:
0x7FFF - no argument - start new layer
0x7FFE - u32 argument - index of texture page data from texture page section
otherwise:
u16 maybe_depth, u16 start_index, u16 count - how many tiles to draw, data is taken from the background section for the first layer, and from the sprite section from the following layer. Depth is a guess.

I assume the depth is not very precise, and the whole thing is just managed by the layers and the walkmesh. So the tiles in a layer cover a whole depth range, but the walkmesh is laid out in a way that you can never walk in front of the tiles you're not supposed to.

That said, if you look at the 2KB FALLP in 7mimic, it does show a still from the parachute scene...

Which is interesting because that seems to be one of the few file groups that the DAT file is 2KByte in size, but the MIM is bigger, and is LZS-compressed. In fact, these kind of file groups in the DAT files each have a small value as the first dword, then follows with alternating values of non-zero and zero valued dwords, or with a bit more structure with what looks like actual 8-byte tile data at the end, uncompressed.

If they are not used in the game there is a chance they were used during development and written out in an earlier version of the file format. Once they were removed from the game they were not updated with new builds of the source file, but maybe somebody forgot to delete the old files and they still ended up in the disc image.

[VanishedOne]

That said, if you look at the 2KB FALLP in 7mimic, it does show a still from the parachute scene...

Which is interesting because that seems to be one of the few file groups that the DAT file is 2KByte in size, but the MIM is bigger, and is LZS-compressed. In fact, these kind of file groups in the DAT files each have a small value as the first dword, then follows with alternating values of non-zero and zero valued dwords, or with a bit more structure with what looks like actual 8-byte tile data at the end, uncompressed.

If they are not used in the game there is a chance they were used during development and written out in an earlier version of the file format.

If you want to investigate possible changing versions of the DAT format, it might be worth looking at the PSX demos made during development. The early demo bundled with Tobal No.1 doesn't even have DAT and BSX files; it has ATE, BSC, CA, ID and MAP files instead. (It still has MIM; I haven't made sure that's the same MIM format as the final game's. I know some aspects of the character encoding are different: kana are mostly the same, but kanji have different codes from the final game's; I can't remember whether the English version's encoding differs much. I've heard it's a similar story with text in the Chrono Trigger Prerelease, so maybe it's a Square habit.)

[Micky]

(I know some aspects of the character encoding are different: kana are mostly the same, but kanji have different codes from the final game's; I can't remember whether the English version's encoding differs much. I've heard it's a similar story with text in the Chrono Trigger Prerelease, so maybe it's a Square habit.)

The Kanji thing could be just a result of text changes for the final build. Based on the amount of available memory you may not want to blow your memory budget on a full Kanji set, so what people do is they run a tool over the text to make a list of only the characters required, and then produce a minimal font that only contains the required characters.
In many cases the demo is taken from a far-from-complete build of the game, before final bugs a squashed, all features are implemented, and before the "oh crap how are we going to fit all this stuff onto a CD?" pass.

[VanishedOne]

Uh-huh. I just thought I'd mention it anyway to save anyone else looking at the demo files the trouble of rediscovering it.

[Cyberman]

There is some sort of "build" file dumped in the FIELD section of the PSX disks. This has a lot of data but I don't believe it's used by the game at all. I'm trying to remember it's name. Oh well if it's not a DAT MIM or BSX  (something like that) that's likely to be close to what that file is. It's been a long time since I messed with the data BCX are characters (heh look at the names) I believe it's FIELD.BIN not positive though as I said it's been a while.

Cyb