Author Topic: "Cracking" FFVIII model data  (Read 35568 times)

Afrotronics

  • *
  • Posts: 27
    • View Profile
    • http://www.afrotronics.com
"Cracking" FFVIII model data
« on: 2003-11-10 16:16:11 »
Yo er'body.  I was chillin at my computer last night and I was messin' around with the data files from FFVIII, specifically the model files (.mch).  In my programming classes (I got to Eastern Michigan University BTW) I'm learning how to read from different file type and finding algortihms and that good stuff.  So I decided to challenge myself.  I wanted to see if I could figure out the structure of an FFVIII model file.  After tens of minutes of hex editing and consulting with my friend Mo (who is an excellent 3d and AI Programmer BTW) I notice some patterns that may be vertex data.  Looking through the file you'll come across lines that have -D and %D in an obvious pattern.  A majority of the time the -D or %D (it's usually %D) will have six characters before it.  I think these characters may be the x, y, and z coordinates (but where is w?). Before and after each instance of the 6char+%D there is (in hex) 0000 0100.  I think this may be the defining space container for each vertex.  There were some other patterns I found but I can't remember them right now (I'm in the computer lab at school so no FFVIII data is accessable).  But the thing I'm forgetting is pretty important if I recall.  My friend Moe pointed it out to me.  I'll post it when I get home tonight.  I don't know if I'm even remotely on to something ...shoot I could be looking at the wrong files...If I am, I'm definitely going to start writing a file reader for this file type.  But I hope this is useful to y'all.  Off topic--You know how you can hexedit a lot of programs and plain english will turn up in a file?  Am I not seeing any English in these files (except for the .cnf files and the grp file) because the game was programmed in japanese or is there just no human readable data in any of the files? Anyways, I gots to get back to programmin' my stupid/hard e-commerce recommendation system project for my java programming II class.

Qhimm

  • Founder
  • *
  • Posts: 1996
    • View Profile
    • Qhimm.com
"Cracking" FFVIII model data
« Reply #1 on: 2003-11-10 16:26:09 »
Vertex data and texture (including mappings) can be read through an old editor I once wrote... so those values you've seen is probably the vertex coordinate data, yes. :) I never quite figured out where the skeletal layout/animation is stored however, I think it's in the individual scene files where they're used...

Afrotronics

  • *
  • Posts: 27
    • View Profile
    • http://www.afrotronics.com
More info
« Reply #2 on: 2003-11-10 16:33:54 »
Oh yeah....I was also hexediting the FFVIII executable and there are craploads of Direct 3d calls in there.  I have no clue what they all do cuz I don't know directX programming all that good stuff very well.  I'm assuming (not a good idea, eh :wink: ) that somewhere there's a call for getting model data.  

-Just for clarification this ISN'T the pattern that I forgot the details (because this isn't a pattern at all, it's just an observation).  Yo, if you have any extra info and whatnot pllllllleeeeeeeeeeeeaaaaaaaase post!

Afrotronics

  • *
  • Posts: 27
    • View Profile
    • http://www.afrotronics.com
Sorry bout the double post
« Reply #3 on: 2003-11-10 16:39:29 »
:( :erm: Sorry bout the double post I wasn't quite sure which topic it would go under.  I would've thought that it fits Tech-related more because tech related includes programming and inner-workings,  but I wasn't sure because I saw subjects like mine in the Programming feedback topic.  I hope people follow the link in the other one to get to this one because I think it would be really cool if we could get out hands on that there FFVIII Model data and make it useful :D

Cyberman

  • *
  • Posts: 1572
    • View Profile
Couple of Things
« Reply #4 on: 2003-11-10 17:14:24 »
First Qhimm mentioned he has an old program for doing this

Second yes it would have a lot of Direct 3d calls in it. Of course this must mean you are talking about the PC version.

I'm still in the process of wadding through the PSX version of FF7's data :)

For the PSX version of anything beyond FF7 well Square decided to PACK everything tightly so no one could easily access or read the data (I guess they didn't like people ripping data from there games :) ).

Qhimm-
FF8 PC version does it have higher polygon count models for it?
or is it the same as the PSX in terms of model data?
does it have improved textures?
Do you have the 'prolog' information for the model data on the PC?

Cybie

mirex

  • *
  • Posts: 1645
    • View Profile
    • http://mirex.mypage.sk
"Cracking" FFVIII model data
« Reply #5 on: 2003-11-12 11:23:45 »
I tried to rip ff8 data apart, but i didnt succeeded yet. I was able to extract only model textures ... no polygon / vertex data yet. I was trying .dat and .c?? files IIRC ... but im not sure about that. I thought that monster polygon / vertex data are stored in those files aswell, because in each there are textures for one monster and lots of other data i dont understand.

Afrotronics

  • *
  • Posts: 27
    • View Profile
    • http://www.afrotronics.com
"Cracking" FFVIII model data
« Reply #6 on: 2003-11-14 23:33:31 »
were you looking in the .mch files?

mirex

  • *
  • Posts: 1645
    • View Profile
    • http://mirex.mypage.sk
"Cracking" FFVIII model data
« Reply #7 on: 2003-11-16 14:26:50 »
me not sure about that ... maybe ...
could you guys post info that you found ?

Qhimm

  • Founder
  • *
  • Posts: 1996
    • View Profile
    • Qhimm.com
"Cracking" FFVIII model data
« Reply #8 on: 2003-11-16 16:31:39 »
From what I've seen, the model data in FF8 PC is identical to the PSX version in terms of quality.

Cyberman

  • *
  • Posts: 1572
    • View Profile
"Cracking" FFVIII model data
« Reply #9 on: 2003-11-17 03:03:29 »
Ahh.. ok that means they are have similiar data..

Not that it's terribly informative.

However.. here is what I've found about the PSX models in FF7.
There is ALWAYS a TIM section in them and it's ALWAYS the last section in the file. If FF8 data is stored the same as FF7 then finding one of the characters TIM data location that's stretched on them, will find also there model data.  It's backwards admitedly however I think it has potential to work  better than anything else I have at least (hehehe). Are the background images the same format.. I wonder.  They are all uncompressed too as I understand it. More exploring I guess :)

Cyb

mirex

  • *
  • Posts: 1645
    • View Profile
    • http://mirex.mypage.sk
"Cracking" FFVIII model data
« Reply #10 on: 2003-11-18 17:24:38 »
yes they are ... i found out that there are usually 11 parts in the file, i think file is like this: (i dont have proper info on this cpu, so its unprecise)
first there is 4bytes(long integer) - number of file parts. Then for each part there is its offset (4bytes). I know that part 11 (and higher) are TIM textures (model texture, i found some buildings textures, so i think that scenes are stored like this aswell) ... i can extract model textures, it is implemented in biturn, but nothing more ... i hasn't been able to decode model info..
i'll post more proper info when i get it.

Cyberman

  • *
  • Posts: 1572
    • View Profile
"Cracking" FFVIII model data
« Reply #11 on: 2003-11-20 21:19:26 »
Oh... That's good news.
FF7's file format for models is much more varied and have a lot more parts to the objects.

The only exception in FF7 about the TIM at the end is HICLOUD.LZS this has it as Item 96 of 113.  It's actually a 8 bit texture (instead of the 4 bit ones used on all the other ones).

Normally if the model doesn't use a texture it has a 64 byte TIM that's appears to be TV color bar colors. 16 pixels in all.

I assume all FF7 objects have colored vertices and use Guarad shading save for the few with actual textures?

I'm still working on the PSX model data.. darn hard to figure out.

Cyb

Haruhiko

  • *
  • Posts: 24
    • View Profile
"Cracking" FFVIII model data
« Reply #12 on: 2003-11-21 09:07:07 »
I've done some work with the FF7-format. Yes, the last section is always a 4-bit TIM with no pixel data (only palette) when the model has no texture. I know also about the vertex/color data and the offsets that pointing to them. FF7 model files contains also lines and quads. it seems there are also the hierachy and skeleton data in the files but i haven't decoded them 100%.

Cyberman

  • *
  • Posts: 1572
    • View Profile
"Cracking" FFVIII model data
« Reply #13 on: 2003-11-21 13:52:21 »
Which section has all the directive information for the file?
Is it section 1 or 2? I haven't pulled my hair out looking for it, but I've thought about it, back to adding my search function to my TV tool. (Originally Tim View. Now it views TIM's LZS compressed TIMs .DAT (partially resolved still needs more work) MIM <DITTO> and dumps BSX files. and of Course Battle Models).

Anyhow I can partially decode MIM files at least to see all the blocks that consist of the actual image.

Is it me or would all the character models be cramed in with each BSX or DAT file? I think that's where all the data for the cutscenes is.

Cyb

Afrotronics

  • *
  • Posts: 27
    • View Profile
    • http://www.afrotronics.com
"Cracking" FFVIII model data
« Reply #14 on: 2003-11-25 15:11:52 »
Yo, I'm thinkin' that none of the model data is actually stored where the textures are.  The mag20383xx... files are just the texture and motion files.  I'm thinkin that all of the model data is stored in the .mch files.  Wouldn't it make sense to keep the model data separate from animation and textures to cut down on size.  If each scene had to store an individual model then the game would be so much larger.  Seeing as that the models themsleves are pretty much static throughout the game you could just keep them in a separate file.  For a model to be the actual model it needs to well, stay as the model that it is.  It would really suck for the programmers to have to reference a different model for each seen instead saying "We have this model that we're going to put in this scene and in this scene the model will be doing this (which is the motion data) and will use this(these) texture(s)".  This is only a guess.  But I'd have no other explaination what the .mch files are in a directory called "model" and everybody is having such a crazy hard time figuring out where exactly the vertex data is in the other file.

Afrotronics

  • *
  • Posts: 27
    • View Profile
    • http://www.afrotronics.com
"Cracking" FFVIII model data
« Reply #15 on: 2003-11-26 00:22:18 »
Well, i guess I was kinda wrong.  I just found that the .mch files contain texture data at the very begining.  it looks like after that though, there is vertex data or mapping data.  It seems as though that the main character each have their own .mch file.

Qhimm

  • Founder
  • *
  • Posts: 1996
    • View Profile
    • Qhimm.com
"Cracking" FFVIII model data
« Reply #16 on: 2003-11-30 00:14:19 »
Pretty much all of the main characters have their own .mch files which are referenced in the actual field data files (rather than having to include the entire model in each field it's used). Other character models reside in the chara.one files found in the field files. Also, all animation data resides in the chara.one files, even for main character models. The animation data (found in the beginning of each model block in the chara.one files) is pretty much the only part of the model format that still eludes me; I have decoded the vertex data, face data, skeleton data and texture mapping data. Once I've understood the format of the animation data, I might compile a document on it, or write a small program to read the models. (just pleeease don't bug me about it ;) )

Until then, here's a basic layout on the .mch files:
  • Index block, 64 DWORDs. Contains the offsets of each texture in the file, then a 0xFFFFFFFF, then the offset of the model data.
  • Texture data. Standard TIM format.
  • Model data header, 8+8 DWORDs. Contains counts and offsets of different types of model data (skeletal joints, vertices, unknown, faces, unknown, vertex-to-bone links, 3 x unknown).
  • Skeletal joint data. 64-byte blocks of mostly zeros, contains a 1-based parent index and a bone length value. Probably a bunch of skeletal deformation data used in-game, but whose values are initially stored as zeros.
  • Vertices. Blocks of 4 shorts, the first three being x-, y- and z-coordinates and the fourth unused.
  • Faces. Huge blocks containing vertex indices, color data (unused), edge data (I think), texture mapping data and texture ID. Faces can be 3-point or 4-point polygons, decided by the first DWORD (actually it's a PSX GPU command ID).
  • Vertex groups, or "limbs". These decide which vertices are connected to which skeletal node. Groups of 4 shorts, with the index of the first vertex, the number of vertices and the bone ID (1-based) to which they belong.[/list:u]
    Teaser image of a decapitated Squall

    Side notes: The animation data somehow stores the internal rotations for the model's skeletal nodes using 4 bytes per node. I'm speculating that the format would be one which a PSX could handle, in line with the lazy programmers theorem. Any ideas?

    Also, the battle model format is vastly different from the normal field model format. I haven't even been able to find any vertex data in it, I'm almost suspecting it's compressed or something.

Afrotronics

  • *
  • Posts: 27
    • View Profile
    • http://www.afrotronics.com
"Cracking" FFVIII model data
« Reply #17 on: 2003-11-30 17:23:41 »
WooooooHOOOOOOOOOO!!!!!!! Qhimm you so totally rock!  Thank you to everybody that helped in this effort.  I love seeing collections of knowledge be put to work!   Oh yeah, I was checkin' out some of the field data.  I forgot which file it was exactly BUT it looks like the texture and model data for the vattle scences.  The texture for battle scenes are 256x256 and for some reason or another each line of texture is separated by a line of a solid color. Just thought I'd throw in another two cents.

Afrotronics

  • *
  • Posts: 27
    • View Profile
    • http://www.afrotronics.com
"Cracking" FFVIII model data
« Reply #18 on: 2003-12-09 16:41:51 »
in .mch files:

7F 7F 7F 00 7F 7F 7F 00 7F 7F 7F 00 7F 7F 7F 00
<data>
7F 7F 7F 00 7F 7F 7F 00 7F 7F 7F 00 7F 7F 7F 00


is this some sort of container of some sort?

Qhimm

  • Founder
  • *
  • Posts: 1996
    • View Profile
    • Qhimm.com
"Cracking" FFVIII model data
« Reply #19 on: 2003-12-09 19:17:43 »
Nope, that looks like the vertex colors actually.

Afrotronics

  • *
  • Posts: 27
    • View Profile
    • http://www.afrotronics.com
"Cracking" FFVIII model data
« Reply #20 on: 2003-12-10 11:20:36 »
Using d001.mch I tried the whole 4shorts for the x,y,z coordinates and when I put the verteces in to a 3d space it looks like a vase or something kinda shaped like a vase, or a guy crouching.  Is it supposed to look like a vase?

Cyberman

  • *
  • Posts: 1572
    • View Profile
"Cracking" FFVIII model data
« Reply #21 on: 2003-12-13 05:05:32 »
Quote from: Qhimm

Side notes: The animation data somehow stores the internal rotations for the model's skeletal nodes using 4 bytes per node. I'm speculating that the format would be one which a PSX could handle, in line with the lazy programmers theorem. Any ideas?

Also, the battle model format is vastly different from the normal field model format. I haven't even been able to find any vertex data in it, I'm almost suspecting it's compressed or something.


Qhimm I've analyzed the PSX FF8 data for a bit, it's identical to the PC data you have provided thus far. I suppose one can conclude they are the same surprisingly. Unlike FF7... For FF9 they used yet another format.. sigh at least we know :)

Cyb

mirex

  • *
  • Posts: 1645
    • View Profile
    • http://mirex.mypage.sk
"Cracking" FFVIII model data
« Reply #22 on: 2003-12-15 10:36:31 »
wow Qhimm great work.

mirex

  • *
  • Posts: 1645
    • View Profile
    • http://mirex.mypage.sk
"Cracking" FFVIII model data
« Reply #23 on: 2004-03-26 18:05:38 »
Hey i finally found a way how to read something of those ff8 .mch's ... model with textures and skeleton to be exact. No animations yet.
And battle models are still giving me headaches, so its only bad-looking models up-to-date.
Here are some pics i dumped:
http://mypage.sk/BIN/ff8f_1.jpg
http://mypage.sk/BIN/ff8f_2.jpg
http://mypage.sk/BIN/ff8f_3.jpg
http://mypage.sk/BIN/ff8f_4.jpg

http://mypage.sk/BIN/ff8f_6.jpg

mirex

  • *
  • Posts: 1645
    • View Profile
    • http://mirex.mypage.sk
"Cracking" FFVIII model data
« Reply #24 on: 2004-03-27 00:23:44 »
And here is how are ff8 battle models (.dat's) stored:

 Final fantasy 8 battle dat format

At the start of file there is 'file data table'.

- file data table -
long            number_of_file_parts
number_of_file_parts * long             offset to file part
long            end_of_file_offset

Some file data have 'file data table' or 'data sub table' at their start if they
contain more items, textures for example. 'data sub table' is similar to 'file data table',
except it does not have end_of_file_offset. All offsets are relative to table start offset.

- data sub table -
long            number_of_file_parts
number_of_file_parts * long             offset to file part


File data table shows that file consists of 11 parts usually for monster battle files.
Of these i know that 1st is bone model, 2nd is model geometry and 11th are textures,
3rd is probably model animation, 7th might be monster information.
I haven't decrypted whats in other parts, but i think there are animations, maybe monster
info (i saw monster name there somewhere).

- bone model -
header is 16 bytes long
2 byte short    number_of_bones
14 bytes                unknown

each bone info is 30h (48 dec) bytes long
2 byte short    bone parent
2 byte short    bone length, negative
44 bytes                sometimes zeroes, sometimes not

- geometry -
There is 'data sub table' at the start of data. Shows number of objects and
offsets to their data.

After table there are vertices. each is 3 * short = 6 bytes;
there ain't no number of vertices, but vertex data are 16 bytes padded, and
last 8 bytes are zeroes. before that there are numbers of 3polys and 4polys.
After that data are padded with zeroes to 16 bytes from start of vertex data.

then polygons follow


- texures -
There is table like 'file data table' at the start, from which you can see how many
textures are there, what are their offsets and sizes. They are FF8 tim's, 8bit paletized,
usually with sizes 128x128, so their size is 16,928 bytes. Data of these texture files
start with these bytes: '10 00 00 00  09 00 00 00  0C 02 00 00'