Author Topic: [Dissidia] Models - Viewer and Extractor Tool by MrAdults (2009-04-08)  (Read 150863 times)

Aurenasek116

  • *
  • Posts: 155
    • View Profile
Re: Dissidia Models
« Reply #25 on: 2009-03-25 01:53:18 »
I think it's same as with Atrac audio format, in atrac case it's simply wav but wrapped in new header to make it look like if it was new format. I don't know really, I can be wrong.

Also here are the Dissidia tools coded by Vash, it has a GIM extractor aswell as other stuff which you can have a look at, maybe it will be of any use.

http://rapidshare.com/files/213186921/Dissidia_Tools.rar.html

Rapidshare link for now since FileFront is doing some maintance.

MrAdults

  • Guest
Re: Dissidia Models
« Reply #26 on: 2009-03-25 05:55:10 »
Just had a look myself. Looks like a nice big batch of straight-up-uncompressed bone matrices near the beginning of the file, so that's good. I would have at least expected some quat compression, but I'm not complaining. :) First thing to do is probably locate offset bits for the image location since it's a known piece of data. Wherever that offset is, there are probably more offsets to other meaningful mesh data.

Edit: Alright, I've been working on it over the past few hours. The file is broken up into chunks, and I've figured out how to ID the chunks, be they bone data, meshes, or textures. Mesh data is a more complicated matter. I'm using the crystal model as a simple test case.

Strangely, it draws the verts properly as a raw triangle list, but there is a data segment right before the verts that looks very much like triangle strip indices. However, trying to use that index list does not look correct in any way, so I'm not sure what the point is. It can't be a list of weight indices, because the crystal doesn't need to be weighted (since from what I can tell it only has a single root bone, and if every index were the same that would be obvious).

Vertex data is also not stored at all in native PSP GPU format. Vertex positions are offset from the beginning of the list before UV coordinates, which is not an accepted hardware format. However, vertex formats do vary wildly from model to model. I'm probbaly skipping over a format descriptor somewhere along the line.

As a side note, the very large files appear to be purely skeletal animation data with no mesh data at all. But many of the character model chunks also include some skeletal animation data.
« Last Edit: 2009-03-25 13:32:20 by MrAdults »

Aurenasek116

  • *
  • Posts: 155
    • View Profile
Re: Dissidia Models
« Reply #27 on: 2009-03-25 14:35:39 »
That's a nice piece of work, but the fact about models not being in native PSP GPU format saddens me. I hope it is possible for you/koral or both to find a solution and hopefully make the models be extractable.

MrAdults

  • Guest
Re: Dissidia Models
« Reply #28 on: 2009-03-25 17:25:15 »
I seem to be able to reliably grab vertex data from all of the models now, including all surfaces and surface sub-meshes.

Next thing is to determine a reliable offset for the UV coordinates and other specific vertex data based on the vertex tag. I believe I've found the tag, it just doesn't correspond at all to the PSP hardware's encoding scheme. But it's probably still the key to determining the vertex layout. In any case, I still haven't even bothered to handle ripping the images, so I'll have to get that done sometime before I move onto testing UV's.

BlitzNCS

  • *
  • Posts: 889
  • Master of nothing in particular
    • View Profile
    • My Youtube
Re: Dissidia Models
« Reply #29 on: 2009-03-25 17:36:08 »
Holy crap
its been what, a day? and you've already almost figured out model data!?
Wow, you realy win dude :O
Epicness, i will keep up with this thread :D

koral

  • Guest
Re: Dissidia Models
« Reply #30 on: 2009-03-25 17:59:19 »
wow, that was really quick!  :-D
I haven't looked much at these files myself, so sorry I can't be of much help.

You said the vertices were not in PSP-GPU format, which is quite surprising, but what then did you mean by the "tags"?
Some sort of container mechanism for meshes in this game?

But great progress so far, I hope you will fill us in on some specifics soon!
or maybe a model viewer would be a better alternative  :wink:

Aurenasek116

  • *
  • Posts: 155
    • View Profile
Re: Dissidia Models
« Reply #31 on: 2009-03-25 18:27:35 »
This is trully professional work, great job there. You sure work lightning fast.

Satoh

  • *
  • Posts: 386
  • Assuming this statement is correct, I'm alive.
    • View Profile
Re: Dissidia Models
« Reply #32 on: 2009-03-25 19:44:50 »
It looks as though they are both holding their normal and EX mode weapons from that shot, maybe we'll be able to tell once we get some UV's figured out...

koral

  • Guest
Re: Dissidia Models
« Reply #33 on: 2009-03-25 22:14:26 »
UVs without texture-maps are useless  :lol:

I dont know much about TIM or GIM files, but I do know that we will have to manually parse and extract the textures for a proper kind of model-viewer.

Some further observations:
  • there appear to be NO obvious offsets at the begging of the files, so the files have to be parsed like X, B3D, 3DS (etc) data, reading and skipping chunks depending on thei "chunk-tags"
  • there are "16-mesh" file chunks which are probably 16-bit indexed geometry
  • ".tga" chunks, which are soon followed by "MIG 00 1PSP" ascii text, possibly a formal sort of header for the data.
  • the Texture data itself seems to be a long list of unpalleted RGBA list (initial impressions)
  • "camera" chunks are probably found only in files which are levels/locations/maps
  • "arrays" chunks are plentiful and confusing
  • "r-box" chunks are most likely bounding-boxes in animated models
  • "lambert" chunks are probably generic material properties.


I didn't set-out to fully document these files, but I can't seem to help myself  :-D

Aurenasek116

  • *
  • Posts: 155
    • View Profile
Re: Dissidia Models
« Reply #34 on: 2009-03-26 00:59:05 »
About that MIG stuff Koral, Vash said it's simply GIM.

MrAdults

  • Guest
Re: Dissidia Models
« Reply #35 on: 2009-03-26 01:31:27 »
Yep, I was actually going to liken them to 3DS myself. :) Those array chunks have vertex info in their chunk headers, and that's where I believe I've found the vertex tag. However, I'm purely guessing that that's what it is. But it seems that it changes in correlation with the vertex stride, so I'm guessing I can use it to get component offsets too.

I'm still unsure what the data in the mesh chunk is. It seems to have an offset to the corresponding array, and usually a series of what looks like triangle indices. But none of the array data is rendered with indices! And, those values always number up to numVerts in the array, so it's hard to imagine what they could possibly be other than triangle indices or weight indices (which they seemingly can't be, as I already mentioned). That data is very mysterious, although it evidently is not needed to render the models.

I have no idea what a GIM file is. But if it's straight-up RGBA, sounds easy enough. I'll get to work on that soon-ish.

Edit: Got textures loading, they're 8-bit paletted 32-bit RGBA with width/height and offset to palette in the chunk header. But it looks like that mystery data is triangle data after all, because even though the model's holes are filled in drawing it without indices, it becomes apparent that the triangle UV's are being stretched incorrectly, so the texture map on the model looks all cut up and only correctly offset from the first triangle vert.

By the way, koral, do you plan to make a viewer for these models? I'm going to write up a format spec eventually either way, but if you're planning to actively work on this, I'll make more effort to document my specific format findings and exact offset locations and so on.
« Last Edit: 2009-03-26 07:01:13 by MrAdults »

apz freak

  • *
  • Posts: 410
    • View Profile
Re: Dissidia Models
« Reply #36 on: 2009-03-26 06:04:53 »
You guys are absolutely amazing you know that? :-D It's interesting and facinating to watch you guys decode the data. But let's not stop the conversational flow for praise just yet, carry on!

koral

  • Guest
Re: Dissidia Models
« Reply #37 on: 2009-03-26 14:04:49 »
I wasn't planning to make a new viewer/extractor for these files.
If you provided some specs and stuff then I could always include the format into the CC-Viewer sometime in the future, but if your mesh2drm tool will support these files anyway, then there would be no need for you to document your findings at all.

I am sure everyone will be happy to use your tool to view/extract the models.
Right?  :?


But lets not jump the gun.  :wink:
You say there are some weird issues going on with the vertices/indices. Seems like it may be somehow related to the bone weights, but that shouldn't affect UVs?
Weird

MrAdults

  • Guest
Re: Dissidia Models
« Reply #38 on: 2009-03-27 09:31:00 »
There turned out to be all kinds of issues. Meshes indirectly reference textures through materials, so I had to utilize the material chunks to get textures working. Then it turns out that there are quads as well embedded within the mesh indexing stream, and finally the part that still plagues me, the UV's are scale-and-biased, or maybe just scaled. Since the UV's are shorts, it makes sense to have a per-surface scale-and-bias value to reduce texel precision errors from the range loss. But since that factor varies from model to model (and maybe mesh to mesh, to a lesser degree), I need to find it to handle UV's properly. I've been looking for quite some time now and I just can't figure out where it is.

I had to manually multiply his UV's on U by 4.0 to get him to look correct. :) Squall takes a 3.0 multiply on U, while Kefka takes a 4.0 on U and 1.5 on V, but some models like the Tonberry look just fine without any scale-and-bias. Once I figure out how that scale is stored, it should be ready for a release.

In addition, you can see that the U on the buster sword's UV is flipped. This is especially interesting, because the buster sword is all one surface and all one mesh, but the hilt still looks correct! I thought this might be a material property of some sort, but it doesn't seem to be, especially since a mesh can only reference one material. This in particular is very confusing. It could mean that my scale-and-bias factor theory is wrong, and something compeltely different is going on with the UV's.

koral

  • Guest
Re: Dissidia Models
« Reply #39 on: 2009-03-27 09:51:43 »
Great progress  :-D

So its just a case of unpredictable UVs? There should be a logical explanation somewhere, it isn't usually an issue in most cases.
The example with the buster-sword in particular: what comes to my mind is possible polygon-flipping, either the vertices are indexed incorrectly (unlikely, seeing as the mesh looks perfect) or the UVs are indexed to incorrect vertices.

I would love to help you try find the bias and scaling factors within the files themselves (if they exist at all) but I don't know which files are which.  :oops:
If you can tell me the filenames for the characters you have already mentioned (Squall, Kefka, and the guy above, Bartz from FF5 isn't he?) then I will try hunt for the values.

MrAdults

  • Guest
Re: Dissidia Models
« Reply #40 on: 2009-03-27 10:09:30 »
Whoa, I just e-mailed you, and you had already replied here by the time I got the e-mail off. :) I don't think the sword is an indexing error, because if I just use -U, it looks good, and if the index were off by 1, it would result in warped coordinates on a uniformly stretched surface like that. But, then again, I could be missing something there that would allow bad indexing to cause this. It could also be that the texture map is laid out such that the proper inverse bias could still allow the hilt to look correct while flipping the sword back around. That seems the most likely explanation, but I haven't yet studied the map in combination with the UV values to find out. I'm hoping I will find the magical scale-bias value and it will just fix itself. :)

Here are some of the character files:
Squall: File000854
The dude in the pic (I think he's some kind of special character in Dissidia, as he has everyone's weapon on him): File000899
Kefka: File001123
Tonyberry: File000895

The figures I gave to get the UV's for them are probably not exact, I just eyed it. Also for reference, the UV's are in the vertex array (which typically contains vertices of either size 14 or 16), and they are always the last two shorts of each vertex. Converting them to floating point UV's is a matter of f/65535.0f, but that 65535 could turn into another value for the scale-and-bias too, instead of it being a multiply factor. I've tried searching Squall for floats in the general 2.9-3.1 range, and shorts ranging from 21545 to 21945, but haven't found anything promising. I've also sniffed around all the headers for the surfaces, suface sub-meshes, and surface vertex arrays, and their tails (which seem to be non-existent, the next mesh/array always starts right after the triangle/vertex list, respectively).

If you need any help figuring out the chunks, let me know. For reference, here are the chunk types I've documented thus far:
0x0004 - Bone info
0x0005 - Surface (contains meshes and vertex arrays)
0x000a - Texture (each texture is in its own chunk)
0x0008 - Material list
0x000B - Mysterious, I think it may be arrays of data for bone indexing and/or vertex weights.
0x000F - Texture animation, for blinking characters and such. This could conceivably contain data needed to fix the UV's, but probably not. However, I haven't investigated it too much yet.
0x8015 - Unknown, seems to be a list of floats, but I don't think any are useful for the UV correction.

My chunk structure is as such:
Code: [Select]
WORD chunkType;
short unknownB;
int chunkSize;

koral

  • Guest
Re: Dissidia Models
« Reply #41 on: 2009-03-27 10:35:16 »
I was just passing through and noticed your post  :lol:

Thankyou for the info
Based on what you have said about the UVs being shorts (of such immense values) I have to agree with you that there may be more to the numbers than just the UVs.
And we also dont know if the PSP would use a peculiar mechanism to shrink the values down so they can be used for rendering.

Have you tried bit-shifting the shorts instead of straight divisions? That may yield some kind of natural offsets, or perhaps leave a spare bit or two
I am just making random guesses here, I cant look into the files myself right now, sorry


MrAdults

  • Guest
Re: Dissidia Models
« Reply #42 on: 2009-03-27 10:46:03 »
Have you tried bit-shifting the shorts instead of straight divisions? That may yield some kind of natural offsets, or perhaps leave a spare bit or two
Yeah, I had the same thought about bit shifting, but nothing there seemed to make sense. However, I may have dismissed chunk 0x8015 too quickly, as I believe these floats may have a relation to the UV scaling after all. There seems to be two different 0x8015 chunks in each file, and the second one seems to have what looks like texel centering data. It will take some more looking into to determine if they're what we're looking for.

I am just making random guesses here, I cant look into the files myself right now, sorry
No worries, I'm grateful for your input regardless. :) And there's no rush. I'm sure it's a small matter of time now til we've got this format in the bag.

Edit: A smaller matter of time than I thought!


That second 0x8015 contains the exact data I was hoping for! 4 floats in the form of U-Bias, V-Bias, U-Scale, V-Scale. But they're scaled based on 0.5 instead of 1.0, so I was scanning the files for the wrong value ranges all along. :) And, it did indeed fix the buster sword case as well. I'll have the mesh2rdm release up later on today, most likely.
« Last Edit: 2009-03-27 11:09:13 by MrAdults »

koral

  • Guest
Re: Dissidia Models
« Reply #43 on: 2009-03-27 11:27:19 »
Awesome!!  :-D

If you ever feel in the mood to document your findings then I can add the spec into the Quimm Wiki for future reference.
But I dont think most people care about how its done as long as they have a way of extracting the models via a tool :lol:

This was all your show MrAdults, full props to you and your hard work!
 :-D

Landarma

  • *
  • Posts: 152
    • View Profile
Re: Dissidia Models
« Reply #44 on: 2009-03-27 11:28:27 »
@MrAdults

That 'The dude in the pic' is Barts Klauser from FF5.  ;)  Anyway, the progress seems to be pretty fast.  :-D

Aurenasek116

  • *
  • Posts: 155
    • View Profile
Re: Dissidia Models
« Reply #45 on: 2009-03-27 13:44:40 »
That's a great piece of work, once you release I'll make in-game sprite based Kefka using Dissidia model as base (Since Dissidia one is based on artwork which I don't like mainly because of face). And yes, that char you described as unknown is Butz Klauzer (In japanese version) also known as Bartz Klauser (in english). I do have PSP and Dissidia and the reason why he has weapons of all other heroes is because his fighting style is Mime job, where he copies attacks and weapons of other heroes.

MrAdults

  • Guest
Re: Dissidia Models
« Reply #46 on: 2009-03-27 14:26:33 »
Oh yeah, that makes sense, since he's from FF5 where everyone could do anything via jobs. Although I remember his hair being bushier. ;) I'm still waiting for the North American release to play the game.

I'll write up a full spec in the coming days, koral. Documenting it will probably take a little while, as there is quite a lot of info required just to get a model rendering, so I'll just chip away at it in my leisure time. I could just send you the source module to handle the files, too, although it is roughly 10 times more horrifying and needlessly complicated than my Crisis Core module, so I would not wish to inflict that harm upon you. :-D

Aurenasek116

  • *
  • Posts: 155
    • View Profile
Re: Dissidia Models
« Reply #47 on: 2009-03-27 15:08:11 »
Well, all Dissidia characters are based on Amano's artworks, not their igame looks. His ingame sprite did indeed have different hair but on Amano's drawings it was like this.

MrAdults

  • Guest
Re: Dissidia Models
« Reply #48 on: 2009-03-27 18:18:38 »
Tool and what's-new/how-to-use here:
http://www.richwhitehouse.com/index.php?postid=33

It seems to work fine with environmental models and stuff too. There is one bug that I only just noticed, which is that sometimes pure static geometry that is symmetrical will get half of itself culled out. I believe they use a surface flag to indicate those surfaces and re-render them with inverse axis coordinates, to save on vertex space. But, I haven't gotten around to finding that flag yet. The only one of the characters that are affected from what I've been able to tell is Bartz. His cape must be "static" for some reason, maybe it's because there's special code handling to animate the verts or something. But generally, since characters are point-weighted, that optimization can't take place on them.

I think that's everything. Enjoy. :)

ultima espio

  • *
  • Posts: 1357
    • View Profile
Re: Dissidia Models
« Reply #49 on: 2009-03-27 18:34:49 »
Cerberus next plz :-P

I'm surprised at how fast this was done, nice job!