Author Topic: FF7 Crisis Core - File Format and Data Investigation  (Read 200497 times)

koral

  • Guest
Yaay!



But it seems to crash strangly when I try to load Genesis [file: 02170.raw], also with the Barrel model [file: 02450.raw]. Maybe it is something I am doing wrong somewhere with texture loading, because they were working fine before i started playing the multi-textures.
Apparantly, those files don't contain any 0x77s to seperate the texture-data headers from the textures, so it is probably just a case of mistaken identity.
Perhaps there is a texture count somewhere?


And you may notice something peculiar with Tifa here [file: 02192]:


No hat!
Remember my earlier attempts showed her hat, and everyones weapons too.

MrAdults, you suggested the possibility of multiple models right? I will have to look for them before making the release, but if the multi-textures are any guide, then they should follow on immedietly after the primary model data.


And NO, I will not be adding in any more export formats for textures, because it should be simple enough to convert them to other formats.
I may have to switch to X files later if we discover precise skeletal and animation data, but OBJ should suffice until then.

Maybe we are getting a little ahead of ourselves here  :-P
Thanks for the support everyone!  :-D


EDIT:
I figured out the Texture-Count thing, there is a SHORT at 0xe of the file-header giving a count of the number of texture-headers present:



now to look for the count and location of other models within the same file!  :lol:


EDIT2:

Figured that out too!



The file header contains a SHORT at 0xa of the number of models, and jumping to the vertex-data offset simply read in the offset to the start of the vertices, located every 8-bytes. There is an unknown DWORD before each offset, a count for something in each vertex group but no idea what.

I ran through ALL the character !-Files [files: 02163 - 02457] and they all load up correctly, although some have screwed up texture-maps (because of differing interlacing or something).

So I can now happily annouce that the next release is on its way!
ETA: sometime later today
 :-D
« Last Edit: 2009-03-14 14:28:41 by koral »

ultima espio

  • *
  • Posts: 1357
    • View Profile
 :-o :-o :-o Neocloudstrife is gonna die when he sees this!
« Last Edit: 2009-03-14 17:46:47 by ultima espio »

Aurenasek116

  • *
  • Posts: 155
    • View Profile
If this gets released you will see swarm of crisis core models ported to Jedi Academy.

koral

  • Guest
Here is the link: [link removed, baaaad file]
I have updated the first post too, get it now and extract to your heart's desire!  :lol:

Wouldn't it be cool if Marlie-chan (and her friends) could find their way into a new FF7 mod?
« Last Edit: 2009-03-14 19:54:40 by koral »

Aurenasek116

  • *
  • Posts: 155
    • View Profile
Uhm, I get error when trying to run the exe. It says the application has wrong configuration and can't be run...
I'm running Win XP Professional 32bit if that's needed.

ultima espio

  • *
  • Posts: 1357
    • View Profile
Koral this is amazing, thank you sooo much for your hard work!

ignitz

  • *
  • Posts: 44
    • View Profile
HUUUUUUU, KORALL GREAAAATT WOOOOOOORKK, you made it, yes, now you unleashed, XD

-------------------------------
Same problem with font and crash with 3D model, can you be fix these things?
« Last Edit: 2009-03-14 18:33:09 by ignitz »

koral

  • Guest
iiya, sorry about that  :oops:
I was messing with MFC and forms and stuff, probably some rubbish got left over, I will need to sort it ASAP!
Sorry

EDIT:
um, try this (v0.4b, GL): http://www.mediafire.com/?mtz0nyl2d3h
please tell me if this doesn't work, it must be something else then...

EDIT2:
this is v0.4c, which uses DX9 and is perfectly compatible with 3DRipperDX: http://www.mediafire.com/?yewwz3nnlwy

The program is extremelybuggy and likes to crash sometimes when you select a file.
There is nothing else I can do unless I completely revert away from irrlicht.

Sorry I couldn't make it easier :cry:
« Last Edit: 2009-03-14 20:25:10 by koral »

koral

  • Guest
[ignore]
« Last Edit: 2009-03-14 20:21:07 by koral »

MrAdults

  • Guest
Awesome work again, koral. :) I was looking for skeletal data again last night, it's definitely not there in any obvious form. It seems there are groupings of 3 ints and 4 floats in one of the files. There is a short that specifies the number of those groups (seems to be "3" a lot), followed by that many groups, and some other unknown data in between. Those 4 floats could be a quaternion, but the 3 ints don't make sense as coordinates in any kind of interpretation. Plus it would be very strange to hold bone orientations in these kinds of groupings. So, I'm not confident that these models actually have the bone data/skeleton in them, though I'm not discounting it yet. But it would not be uncommon to have the skeleton in another file with the character's animations.

It seems very possible that the skeleton is "flattened" with no hierarchy, and because matrices are base-relative, the animation data could just be a flattened series of orientations that were computed based on the base pose matrix, which is no longer needed at run-time. The benefit of flattening is that no hiererchical transform is needed for joint matrices, although there is a downside of requiring more storage space for orientations. However, for Crisis Core, this means they could get away with absolutely no CPU cost for bone transforms. I am guessing they could not resist that even in the face of some extra memory overhead. ;)

All of that said, I did find where the bone index values are stored for each drawn segment (I think). So now I know which bones each existing weight is referencing. With that knowledge, I can actually re-create a skeleton and approximate the original joint hierarchy. I'm going to go ahead with this approach, because it will allow the characters to be exported and posed with rough bone positions, keeping the original vertex weighting intact. Since weighting is the hard part, this is suitable to my personal needs, at least. If the skeleton has problems, that's easy to tweak and change hierarchy in a modeling program, while retaining weighting info.

Quote
In what way responsible for the multiplayer portion? Do/Did you work at Raven?
Yeah, I worked there for a few years. As you probably know, we had separate executables for singleplayer and multiplayer. I was the guy responsible for the multiplayer codebase (which meant writing all of the MP-specific engine/game code, porting over all of the SP code to be multiplayer-friendly, etc).

Aurenasek116

  • *
  • Posts: 155
    • View Profile
Ah I see. Well then you know it's sad the game never got patched further than 1.01 due to running out of license for Star Wars universe (Or atleast that's what I heard when I was still a staff member at jk3files.com). I know I probably shouldn't ask for that but many people in JK3 community always wanted SP source code, would it be possible to obtain it? That's all for offtopic.

I've noticed that when the model exports to obj, double faces are facing the same way, which means one has to copy mesh and flip normals on it to appear without holes, but then when merged together the mesh will have quadruple faces which sadly in game engine like Quake 3 when shaded looks like chessboard except with triangles instead of squares. Is it possible for model viewer for export the double faces correctly?

PS. If you can't understand what I mean here's the picture:


MrAdults

  • Guest
Quote
Well then you know it's sad the game never got patched further than 1.01 due to running out of license for Star Wars universe (Or atleast that's what I heard when I was still a staff member at jk3files.com). I know I probably shouldn't ask for that but many people in JK3 community always wanted SP source code, would it be possible to obtain it? That's all for offtopic.
Nope, unfortunately, it can't happen. I have been advocating releasing that code, along with multiplayer engine code, for years (well after leaving Raven/Activision). Unfortunately, LucasArts will not respond to my inquiries. It's all in their hands - Raven is happy to pack up the code if Lucas gives the go-ahead. I've even asked for John Romero's help (since he has played a part in many Quake-engine-related open source releases), and LucasArts would not reply to him either. Additionally, no one that was working at Lucas and supervising/producer JK2/JA is even there anymore, from what I can tell. They seem to completely cycle out employees quite often - apparently most of the Force Unleashed team is no longer there either. As you can tell, I'm no more pleased by the situation than the rest of the community, but my repeated efforts over the years have all failed.

BlitzNCS

  • *
  • Posts: 889
  • Master of nothing in particular
    • View Profile
    • My Youtube
OMGOMGOMG!!!
you guys are amazing! thanks so much koral!
i'm actually on a mobile phone so i can't try it, but i WILL :D

MrAdults

  • Guest
Good news! Even though I still can't find anything that could be a bone orientation in the file, that chunk we've known about for a while now at 0x80 contains hierarchy data. All of those 0xFFFF's (-1 short) initially indicated to me that the skeleton was flattened. However, I decided to ignore that number, and parse through those chunks up to numBones using the first value as the parent, and that's what it was. Additionally, the first WORD right before the number of models in the header appears to always equal "numBones-7". Why -7, I'm unsure, but even "static" models like the crate that have only 1 bone to reference with all 1.0 weights have a value of 8 here, making it consistent with the fully boned/skeletal models.

The bone orientations can be approximated with knowledge of the associated weights and the hierarchy, now, so we get a full skeleton for the vertex weights to interact with.

The only problem is, those values I thought indicated which bones a segment referenced don't appear to be right. They are 100% right for 1-weight segments, so stuff like the machinery and choppers has a nice perfectly-weighted skeleton. However, soft weights aren't using the values I thought for indices. I have no idea how to get the right value for that, so I'll be continuing to look for that.

Edit: Alright, I figured it out. For reference, bone index values are located starting the WORD before the number of vertices/primitives for the segment, and then skip 1 WORD between each index. Not sure what those WORD values are in between.

I'll post back here once my tools are all ready for distribution.
« Last Edit: 2009-03-15 04:28:42 by MrAdults »

Aurenasek116

  • *
  • Posts: 155
    • View Profile
Wow, that's a great progess there. With ability to import models with their weighs it will so much easier to pose them.

Silver Wings

  • *
  • Posts: 90
    • View Profile
You guys are just amazing, thank you so much  :-D
I've been exploring through out files containg the models (2012-2457) and make a list for you (at least, that's all I can do, hope it would help somehow ^^)
Just a note:
- I cant view files from 02012 to 02160, every time I choose the files, the viewer crashes immediately
- In the others left (2161-2457), there are some files didnt show up any models so I didnt list them
Code: [Select]
2162 Zack - Soldier 1st Class
2163 Zack - after Angeal's death
2164 Zack - Soldier 2nd Class
2165 Zack at the beach
2166 Angeal
2167 Angeal's clone (Hound form)
2168 Angeal w/ wing
2169 Sephiroth
2170 Genesis
2171 Genesis w/ wing
2172 Genesis being degenerated
2173 Genesis's clone w/o mask
2174 Lazard
2175 Lazard in Angeal form
2176 Tseng
2177 Cissnei
2178 Cissnei at the beach
2179 Hojo
2180 Hollander
2181 Hollander at the last stage of degeneration
2182 Aerith
2183 Cloud - the infantry man
2184 Cloud in Soldier 1st class uniform
2185 Infantry man - the captain
2186 Infantry man
2187 Soldier 2nd Class
2188 Soldier 3rd Class
2189 Gillian
2190 Reno
2191 Rude
2192 Tifa
2193-2194 Angeal and Zack with weird shadow on faces (actually, the 2 Soldiers in the Epilogue)
2195 Researcher
2196 Man in suite
2197 Woman in suite
2198 Man
2199 Woman in suite
2200 Boy
2201 Girl
2202 Man
2203 Woman
2204 Man
2205 Wutai warrior
2207 Spriggan
2208 Thunderbird
2209 One eye
2210 Angeal's clone (One eye)
2212 Angeal's clone (Sahagin form)
2214 Angeal's clone (Hound form)
2215 worm
2216 Stinger
2218 Flying machine
2219 Bomb
2220 Griffon
2221 Angeal's clone (Griffon form)
2222 Demon
2223-2224 Genesis's clone w/o wing
2225 Genesis's clone w/ wing
2227 Genesis's clone
2228 Genesis’s clone being degenerated
2229 Sweeper (machine)
2230 Tarantula (machine)
2231-2232 Tank
2234-2235 Wutai's giant monster
2236 Ifrit
2237 Bahamuth
2238 Fury Bahamuth
2239 Angeal Penance
2240 Genesis Avatar
2246 Chair
2250 Truck
2252 Aerith
2253 Sephiroth
2254 Zack - Soldier 2nd Class
2255 Angeal
2256 Genesis
2257 Mask of Genesis's clone
2259 Needle machine
2260 Chain machine
2261 Missile machine
2262 Sweeper (machine)
2263 Wutai's giant monster
2264 Thunderbird
2265 Hound
2267 Dual horns
2368-2370 Saucer (machine)
2371-2372 Alert Head (small machine w/ armour)
2373-2274 Drill machine
2275 Tarantula (machine)
2276 Support machine
2277 Bug
2280 Sephiroth
2281 Genesis's clone (Predator)
2282 Tseng
2283 Sephiroth
2284 Zack - Soldier 1st Class
2285 Zack - after Angeal's death
2288 Man
2289 Woman
2290 Boy
2291 Girl
2292 Angeal w/ wing
2293 Angeal
2294 Zack - Soldier 2nd Class
2295 Genesis's clone
2296 Genesis’s clone being degenerated
2297-2298 Genesis's clone (Predator)
2300 Files
2304 Lazard
2305 Lazard in Angeal form
2306 Cissnei
2307 Genesis w/ wing
2308 Helicopter
2309 Genesis being degenerated
2310 Angeal clone (Sahagin form)
2311 Cloud - the infantry man
2316 Heel
2317-2318 Wutai warrior
2320 Hornets (bees)
2321 Death Claw
2322 Mover
2324 Dorky face
2325 Hungry
2327-2328 Grangalan
2329 Chocobo
2330 Magic pot
2331-2333 Tonberry
2334 CaithSith
2335 Hollander
2336 Motorcycle
2337 Angeal w/ wing
2341 Infantry man (the captain)
2343 Sweeper (machine)
2345 Genesis’s clone w/o wing
2346 Genesis’s clone w/ wing
2347 Man
2348 Woman (the shopkeeper)
2349 Boy (the little thief)
2352 Woman
2353 Boy
2356 Woman
2357-2358 Boy
2359 Juice-can
2361 Book (cyan)
2362 Dumbapple
2363 Book (magenta)
2366 Wutai warrior
2367 Genesis's clone (Infantry man form)
2368 Shinra rocket
2369 Cissnei at the beach
2370-2371 Genesis's clone (Infantry man form)
2372 Genesis’s clone (Predator)
2373 Moogle
2374 Zack at the beach
2375 Rooms
2376 Normal Behemoth
2377 King Behemoth (golden)
2378 Cloud in Soldier 1st class uniform
2379 Genesis’s clone (Dominator)
2380 Hollander at the last stage of degeneration
2382 Worm
2383 Dual horns
2384 Item box
2394 Jenova
2398 The monster in tube
2399-2401 Genesis’s clone (Shadow scythe/knight/mage)
2402 Camera
2405 Top floor
2406 Path?
2407 some kind of machine
2408 Hollander at the last stage of degeneration
2410-2411 Genesis at the last stage of degeneration
2412 Wutai warrior
2414 Tarantula
2415 Sahagin
2416 Stinger
2418 Spriggan
2419 Alert Head (small machine w/ armour)
2420 Hound
2421 Demon
2422 Dorky face
2424 Death Claw
2425 One eye
2426 Bug
2427 Bowl?
2428 Water pot
2429 Strong box
2430 Golden Coin?
2431 Minerva (The Goddess)
2432 Malboro
2434 Black woman in suite
2435 Black man in suite
2436 Yuffie
2438 Documentation?
2439 Statue
2440 Hollander's bag
2441 Plain?
2444 Genesis’s clone (Dominator)
2443 The monster in tube like
2444 Genesis’s clone w/ wing
2445-2447 Flower Wagon
2448 Sniper
2449 Truck
2450 Barrel
2451 Helicopter
2453 Potion
2454 Helicopter
2455 Worm
2456-2457 Zack's last moment
« Last Edit: 2009-03-15 14:22:45 by Silver Wings »

MrAdults

  • Guest
Alrighty, a download link to the tool and all of the information you need to get going is here:

http://www.richwhitehouse.com/index.php?postid=30

Unless there are terrible bugs with the tool, I'm stick-a-fork-in-it-done with this format. :) I don't intend to go find the bone orientations/animations, although if anyone else ever does, I'll probably go ahead and add support for them to my tool. Currently the exported bones are generated based on the weighting data, and since they do have the original hierarchy, they seem to be quite sufficient for posing the models.

Also, my tool does preview, but it's very barebones functionality. It's mainly intended to be a powerful exporter. If you just want to look at the game content, you should probably stick to koral's tool.

DiNeRo

  • Guest
this is epic

im still a noob at weighing, but youll see lots of crisis core models in JK3 soon  :roll:

ultima espio

  • *
  • Posts: 1357
    • View Profile
The Zack and Angeal with weird heads, they are infact weiss and nero :wink:

Silver Wings

  • *
  • Posts: 90
    • View Profile
Oh, I totally forgot the 2 strangers in the epilogue :D thank you for pointing it out ^^

koral

  • Guest
Awesome findings MrAdults!
I haven't had a chance to look into the skeleton stuff yet, but it seems to make perfect sense.
I tried your viewer too, works with no problems. I suppose that was one way of making the viewer easier to use!  :-P

The character/monster/event-model files are the only ones which parse correctly at the moment (as SilverWings has just confirmed  :-D).
I am sure that all the other files before that (02012-02160) must be Locations/Map-models which are similar to the viewable ones, minus animation data.

Sorry about the problems with my viewer Aurenasek116, but that is happening because of how the data was read from the file, and I have been too lazy to resort the polys in a more efficient way. Maybe I will look into it in the future, or you could just use MrAdults's tools (as and when they appear).

Since Rich (MrAdults) is doing such a fabulous job on the skeletal and animation front, I may as well delve deeper into those other files (02012-02160) and see what I can dig-up.

Thanks everyone!
 :-D :-D

Aurenasek116

  • *
  • Posts: 155
    • View Profile
I'm using MrAdult's extractor already and it does a perfect job with extracting models so no problem there koral, great work nonetheless since I prefer to use your viewer as it has a GUI to view models, and MrAdult's for extracting.

koral

  • Guest
I agree, MrAdults's tool is a much better choice to extract what you find!

Just shouting out that I have updated the first post and the wiki a little.

I haven't got around to documenting these !-models fully yet, but it will be quite a large task and I don't want to butcher it.
I owe much thanks to NeoCloudStrife, X-Dina and MrAdults for making it possible, and for their sake it will have to be a job well done.

Materia information is still hidden, which would be the first step to discovering if the Debug-Menu truly is called through a Materia, or if it is just a special kind of Event.
If it's the latter case (an Event), then there is a very high probability that a simple memory-hack cheat could be used to trigger it somehow.

The game itself does not seem to have anything more "mysterious" or unused that I have come across, which is what you would expect from games these days I suppose.

ff7rules

  • *
  • Posts: 423
    • View Profile
Alrighty, a download link to the tool and all of the information you need to get going is here:

http://www.richwhitehouse.com/index.php?postid=30

Unless there are terrible bugs with the tool, I'm stick-a-fork-in-it-done with this format. :) I don't intend to go find the bone orientations/animations, although if anyone else ever does, I'll probably go ahead and add support for them to my tool. Currently the exported bones are generated based on the weighting data, and since they do have the original hierarchy, they seem to be quite sufficient for posing the models.

Also, my tool does preview, but it's very barebones functionality. It's mainly intended to be a powerful exporter. If you just want to look at the game content, you should probably stick to koral's tool.

Wow thanks for this Mradults! im currently intersted in the psx version, model changing and stuff (see the replacing and adding new models thread)
I do have a question though would it be possible for you to make it so we can extract the PSX models so we can finally change them? :D Like all there data and will be just like they were when we compress them back to lzs? Also i noticed the weapons are in the middle of the model any chance of us ever been able to edit them? Thanks for this.

MrAdults

  • Guest
Wow thanks for this Mradults! im currently intersted in the psx version, model changing and stuff (see the replacing and adding new models thread)
I do have a question though would it be possible for you to make it so we can extract the PSX models so we can finally change them? :D Like all there data and will be just like they were when we compress them back to lzs? Also i noticed the weapons are in the middle of the model any chance of us ever been able to edit them? Thanks for this.
That FF7 LZS support has been in the tool for quite a long time, I did it a year or two ago, I believe it was. You could use the tool to convert those models to .smd, but they would lose the vertex coloring. I don't know much about what's out there currently to extract FF7 PSX models, I figured there was already quite a myriad of tools to view and convert the models. I believe .ase export would support everything FF7 models need, including vertex colors and node (mesh) orientations, but my tool doesn't currently export to that format.