FF7 Crisis Core - File Format and Data Investigation

[Aurenasek116]

Just wanted to show you some progress on converting to JK3:



It still needs work, mainly weighs on hips and textures to be added on hands, since in JK3 it's simplier to use basic hands from already default base model rather than the ones from crisis core models. Wing can be turned off in .skin file allowing for easier management ingame. (That was pretty much directed only at those who know about JK3 modding).

Cheers and thanks to Koral and MrAdults aswell as others that helped making this project possible.

[ff7rules]

There's no Tools with the ability to make so we can create and add to old ones yet. I did some testing and the models seem happy to replace with other stuff, Like i replaced Ruby weapons model with Sephiroth worked, same with Cloud and Tifa. I could also replace Nib Sephiroth with anyone i wanted so the models seem quite flexable as far as moving them goes. Any Chance of been able to export to .ase format in the future? so we can edit them and add to them, Also are the weapon's just meshes from bones or are they separate from the model?

[DiNeRo]

to add to Aurenasek's post (we are working together, me as his weigher)

currently we have weighed for jka genesis and sephiroth(although both have major weighing issues, with clipping mainly - i am still working to fix that)

[MrAdults]

There's no Tools with the ability to make so we can create and add to old ones yet. I did some testing and the models seem happy to replace with other stuff, Like i replaced Ruby weapons model with Sephiroth worked, same with Cloud and Tifa. I could also replace Nib Sephiroth with anyone i wanted so the models seem quite flexable as far as moving them goes. Any Chance of been able to export to .ase format in the future? so we can edit them and add to them, Also are the weapon's just meshes from bones or are they separate from the model?

Ah, it sounds like you want the ability to convert back to the FF7 PSX model format, too. I'm not sure how well that would work out, given that you'd probably have size and address restrictions to maintain the alignment. In any case, it's not something I plan to do, as it would be a lot of additional from-scratch functionality. I've seen PC version replacement models, though, so on that side of things there must already be a tool-chain for what you want to accomplish.

BTW, cool to see the models making their way into Jedi Academy.

[Satoh]

I picked the wrong week to not check up for a few days O_O;

That's absolutely amazing, Koral.

Now I just have to rip my disc and start digging through the models for myself!! *starts cackling like Hojo*

EDIT: Ok after much floundering about with my PSP... I successfully...

gave up and downloaded a copy... but I do have the game legally, hell, I bought a PSP JUST for crisis core.

anyway, I have now ripped a series of models XD; I'm ecstatic!

Anyway I noticed the viewer tends to crash a lot after trying to switch from 2D image into 3D renders... not the other way around... Not a big issue...

The one thing that does rather annoy me is the files starting at 02012 and up (on the front page this is the start of the model section) seem to auto-crash me... however working from the bottom up, I can view a lot of models... Is there a possibility of this being just me, or is it the files themselves, or something else entirely?

Btw, I should say again that I am dumbfoundedly impressed with this as is. I especially like the OBJ export function!

I hope that when you're satisfied with this format you'd be willing to investigate some others. (time and relaxation permitting that is. After all... you've done a lot of work already, you deserve a vacation... and a cookie.)

EDIT2: 02012 - 02161 seem to be offending files on my side... I haven't checked every one of them... but the first and last 10 with several sample tests in between all crash me as soon as I even click on the number.

[ff7rules]

Ah, it sounds like you want the ability to convert back to the FF7 PSX model format, too. I'm not sure how well that would work out, given that you'd probably have size and address restrictions to maintain the alignment. In any case, it's not something I plan to do, as it would be a lot of additional from-scratch functionality. I've seen PC version replacement models, though, so on that side of things there must already be a tool-chain for what you want to accomplish.

BTW, cool to see the models making their way into Jedi Academy.

Yeah, I'm thinking of turning my full attention to the PC version  since PSX can't be modded to the fullest like PC can. The formats are allot different PSX uses LZS compression as you know, and PCs version's files are in pieces. I know one way i could get the Model though 3D printscreen, but its pointless as a loss  of data for animations and such is inevitable. As far as i know, there is no way to keep the data. Anyway thanks for the reply.

[koral]

I have been wondering why those files (02012-02160) crash and after looking into them I think I may be onto something.

Those files have the exact same file-headers as the viewable character ones, and the offsets also lead to exactly the correct parts of the files.
There are multiple textures present in there too which i can confirm are identically organised as the viewable files (albeit with different interlacing with most of them).

The reason why they crash though is because their Vertex-data is completely different.
In the viewable-model files when we jump to the Vertex-data start (dictated by the offset at the beginning of the file) we always skipped 4-bytes that were always zeros.
In these other files, those four bytes have a different value: they represent a DWORD and hold a value of 1.
These files also contain many model-parts (50+ usually).

Looking at the data which followed the initial DWORD, there was nothing but a huge list of SHORTS, and those could be plotted as XYZ-vertices to produce almost recognisable shapes: Location Map models.

And these vertices represented QUAD-Lists, not Triangle-lists.

But here is the twist:
Only the first 8-vertices (8 times 3-SHORTS for XYX) at the beginning of every model-part seemed to be "real" vertices, the rest were "fakes" (potentially UV data).

And I can't seem to determine the group-strides, but I am extremely sure that these vertices will be organised in a far more logical manner.

I will continue looking into it, but if anyone would like to help then they are more than welcome 

Stay tuned, the characters were only the beginning 

[MrAdults]

I haven't actually looked at those files at all, but in pure speculation, I believe they should be embedded with some kind of scene graphing data structure. A quad-tree would seem quite suitable for those map layouts, but it could be anything.

It seems a good bet that the positional data you're looking at is actually part of that encapsulation tree, which is why it might happen to look right rendered as verts with quads, as the data nodes are probably in a specific order with a rather uniform alignment, especially if it is a quadtree.

If I'm right, there is probably an index or index list associated with most of those nodes into an actual surface bundle. I'm betting the actual vertex lists are still in the same PSP-GPU vertex format. They may be more versatile in storage than the character models, though, so you might have to use those GPU specs to support other types of coordinates and primitive indexing.

[Satoh]

maybe they're the walkmesh and wall collision models for the area maps...

That was my first though from your post anyway...

EDIT: Also I noticed a few models that seem to have broken textures...
02336 has a broken texture on the wheel of the bike, can you confirm this to make sure it isn't just a bad dump?

[koral]

Those "broken textures" are textures which have different interlacing modes, and it looks like most of those models use them. The viewer doesn't support that interlacing mode yet. 

MrAdults is onto something there, because those quads are all aligned perfectly in space so most likely represent Bounding-Boxes of some kind per model-part.
I haven't seen any familiar TYPE-1 or TYPE-2 vertex structures though, but maybe the data will still parse through correctly.

[koral]

They contain nothing but TYPE-1 vertices afterall! All in a nice long triangle-list.




These places dont seem recognisable, but I probably missed a chunk of model-parts somewhere. And once the textures are in place they should become obvious.
All i need to do now is find the index to textures and we should have these location models in the bag too! 

[BlitzNCS]

Wow, this is amazing koral
I've never actually used a map in a 3d modelling program before O.o
BTW, number 2 looks like the training room from my guess....

[MrAdults]

Sweet. I wonder if there is visibility data in there too, specificying visibility of each of the quad (or whatever they are) nodes to each other. If we preserved that, we could automatically convert it over to a BSP'd, vis'd Quake 3 map or something. That would be pretty novel. Although they may just have another map format, where they placed all of those map bits together with portals or something to specify visibility regions.

[koral]

From what I have seen of the data so far, there is only a single BBox before the list of vertices, no kind of other visibility data at all. They probably used vanilla camera-frustrum checks for data strorage efficiency, or something.

I figured out the texture indices, and here are the results so far:



Most texture-maps are still corrupt because of different interlacing modes, but the correct maps are now applied to the correct vertices.

There are still lots of missing polygons though. 
I still haven't discovered where the precise count of vertices are located, because the vertex-group "header" seems to be different from the characters so the PSP-GPU method doesn't work quite as expected.

[MrAdults]

Ah, oh well. I was thinking of some of the bigger missions - particularly the Wutai complex inflitration one. In total that map had far more parts overlapping than they could have dealt with using exclusively frustum culling. But, I bet that level was made out of a bunch of these map geometry files, that is held together by another structure (which perhaps uses portals or something to isolate visibility sections).

About the verts: Since they have index values in there, the 32-bit vertex encoding value should be changed to reflect that. Have you found the location of that encoding value in these new files yet? It could be the key to decyphering the other vertices. If those verts are really type 1 except with the inclusion of index values, the exact 32-bit encoding value you should be looking for is 0x00000723 + (indexType<<11). indexType could be 1-3 depending on the index type.

As always, good luck.

[koral]

indexType could be 1-3 depending on the index type.

This statement had me scratching my head for a few minutes 
I will look into the encoding value thing when i get home.

And there most likely will some sort of visibility information somewhere, there is a lot we dont know yet about these !-model files.
We will bump into them sooner or later 

[Satoh]

You two are made of epic... I can't describe it any better...

[BlitzNCS]

*seconds*
ya, they do win. you're pretty cool too though, X-dina! 

by the way, i may have missed something here, but what exactly ARE the [&&] files?

[Satoh]

I was wondering that myself...

It says type [&&] Model,but are they really models at all?

[koral]

WOOHOO!!!


All the location models now load correctly! (geometry anyway)

Although these models had a different encode value (0x000011f) and an extra 0x30 bytes to skip (the bounding-box data) the clincher was the "next-vertex-index" which was 6 and not 5 like the character models used.

Each Map (as loaded in the game) is a seperate file, in most cases humongous models with several textures.

Texture interpolation is still a problem though. The only reason why i have repetedly only shows these four locations is because they have almost a "perfect" set of decoded textures.
As soon as I figure out how to correctly decode the other textures, expect to see a new release! 
(MrAdults might need to update his tool too, these are the same !-files after-all )

BTW
&&-files also contain model data and are very similar to these !-files. As to what models they would contain... beats me 

[Satoh]

So the new version is up and can view field areas now? If so... awesome. (I'll find out either way soon enough)

EDIT: OK guessing you didn't upload the newest version.

[Aurenasek116]

That's really interesting find, being able to view locations. Now if it was possible to extract them as models, props could be made and so on for other games. (I'd benefit from those so they could be used in JK3 maps )

[ignitz]

WOOOOO, it's not finished.... My GOD the people of Qhimm don't have limits.
 

[MrAdults]

Great work again, koral! I guess I might as well put proper support in for those files, too. I'll wait until you finish documenting your findings on the wiki, though, so I'm not re-doing the work you've done in finding the new offsets and data structures in these files.

I've actually been working on Silent Hill: Homecoming formats over the past few days in my spare time - I have their storage formats and eveything else figured out, and just got the models rendering properly the other day (though there are a few issues with the texture coordinates). So when that's done I'll jump back over to CC, and maybe we'll even have some Disssidia models to play with by then.

[Satoh]

speaking of using stuff from CC for other games, this is all perfect for a Turks game I was planning in my head... it would be an action RPG with stealth and platforming elements... (I guess you could say similar to devil may cry... but that's not my intention at all)

the player creates a character, name, weapon proficiency, and appearance, does a multi-part tutorial level (involving elements of every weapon, skill, and play style, with multiple possible ways to finish) to help them decide on a 'class' so to speak.

and then the game would be carried out in mission format similar to Crisis Core's but with more plot behind each mission...
[/stuff no one cares about]

I'm really amazed at how fast this has progressed... I remember the Chrono Cross people had what seemed to be several months and in all likelihood over a year invested in just getting the character models figured out... and there were a few people working on it. I helped out with that too actually... nothing more than educated guesses like I do now... I never did quite figure out how to read all that stuff in the hex... I could only get as far as the basic header...

anyway, I'm glad I came back to Qhimm when I did, or I'd have never found this mighty project.

EDIT: also, I'd say your viewer is plenty well user friendly.