Qhimm.com Forums
Miscellaneous Forums => Scripting and Reverse Engineering => Topic started by: Chev on 2009-08-06 16:46:13
-
Greetings
I've been trying to build a model viewer for FF9 and to this end this forum with Zande and Zidane's posts and the wiki have been of great use.
I'm able to get bones, vertices and faces without any problem now, but as I started tackling animations I realized I have no idea how to tell which animation goes with which model. Right now I'm matching a given model with the animations present in the same DB data cluster but there's really no reason that should be right, and that makes it tricky to be certain I'm decoding the angles accurately in the first place... It's not an assumption I can make for textures anyway, since so far I haven't found a TIM file in the same DB cluster as models.
So I was wondering if there's anything about it that any of you may known on the subject but isn't wikified yet? I guess I should look into the six unknown bytes of the model header...
Here's a boneless taxi from lindblum, just to show:
(http://img19.imageshack.us/img19/6594/ff9taxi.th.png) (http://img19.imageshack.us/i/ff9taxi.png/)
EDIT: And an amusing victim of mad chocobo disease
(http://img40.imageshack.us/img40/9389/madchocobo.th.png) (http://img40.imageshack.us/i/madchocobo.png/)
-
I commend you sir!
EDIT: Now that dinner is done, I can say a bit more...
Anyway two things: Be sure to include an export function XD; I so badly want those FFIX models...
And there was a program around at one point that could render models with no bones and had full texture functionality... otherwise you could simply have the program load textures as a separate file and apply whatever texture is loaded to the current model...
Are are you referring to finding the UV information... UV maps I have no idea about, but I'm sure the battle models contain their textures inside them/directly after them.
Anyway, you definitely have my interest.
-
You probably want to stick to using the models and animations that are stored within the same block/cluster for now. This should work fine with playable characters and monsters, which stores meshes, animations and textures together in one place.
Getting the animation IDs for certain models can be abit tricky.
Monsters stores some common animation IDs (such as death and idle animations) inside the stats chunk (together with HP, strength, drops et cetera). Appart from that, animations monsters uses for attacks/abilities are stored within the event script data. Both the event data and stats (together with string and abit of other data) is stored at one place (stored on a battle formations basis) seperated from the mesh/animation and texture data.
The data for NPC and object meshes (like the Lindblum taxi and Chocobo) are stored together at one place together with alot of different data (sprites for example), and their animations IDs are usually found within large event scripts.
-
X-dina> I've got the UVs alright, I'm referring to the actual textures. I'll do my best for an exporter once I get the right data.
Zande> Thanks for the info! I guess I'll just put in a nice handy button for cycling through the available anims and match them visually.
I indeed noticed there seemed to be some pattern to how NPC data was grouped. Since here it's grouped with field data (in my taxi screenshot the last line is the contents of the DB cluster around the opened model) I think that in the directory I opened (04 IIRC) DB clusters represent rooms, which would explain just about all the model grouping and redundancy (looks like each room store their NPCs) that's happening.
Glorious EDIT: I get anims now. Well, mostly. Still having trouble with the lower four bits of each angle, I need to remodel my app a bit before I can do it the right way. ANimes look pretty good already though. I'll also have to make a faster animation routine, right now I'm doing it in a very slow way.
So, rooms indeed contain the anims for their occupants, but some also contain anims that don't match any of their contents. I guess those are for scripted scenes where the player characters must move in specific ways.
(http://img19.imageshack.us/img19/6991/chocobetter.th.png) (http://img19.imageshack.us/i/chocobetter.png/) (http://img505.imageshack.us/img505/1734/taxibetter.th.png) (http://img505.imageshack.us/i/taxibetter.png/) (http://img14.imageshack.us/img14/1039/garnet.th.png) (http://img14.imageshack.us/i/garnet.png/) (http://img22.imageshack.us/img22/261/steiner3.th.png) (http://img22.imageshack.us/i/steiner3.png/)
-
Good job! The textures for playable characters and enemies should be a cakewalk, stored in the same place as the meshes/animations and are standard TIMs. The textures for NPCs and other objects are abit trickier though, have to read other chunks (0x0A I think, but unsure) when working with them.
-
So, rooms indeed contain the anims for their occupants, but some also contain anims that don't match any of their contents. I guess those are for scripted scenes where the player characters must move in specific ways.[/URL]
That seems likely, I think similar things were done in 7 and 8 as well as parts of 10 possibly... I can't think of what it was but I know there was another game I looked into that Chrono Cross NPC's had animation data stored that way as well IIRC...
I recognize Steiner and Garnet with her hood down in those last two frames... This is getting exciting... Oh, and even if you don't have an export function, if it's programmed in Direct X 9/10 I should be able to rip... though exporting would be preferable...
Getting very exciting....
-
Getting very exciting....
No kidding! Great work here! It's taken quite a long time, but it seems like some real breakthroughs are on the way concerning FF9
-
Well, for now I'm really just applying all the wonderful work others have accomplished. There's nothing my program does that hasn't been written in the wiki :-)
-
As you can see I've got textures now. The trick is they're not stored in the exact same cluster instead they're in another cluster that's got the same parent cluster as the one the models are in. I used the material indices to choose what texture to use, because I couldn't be bothered to figure out chunk 0x12 (contains CLUT and tpage for models, says the wiki), and it worked fine with monsters and party members but it'll probably be a problem for NPCs, so next phase will be to understand chunk 0x12.
(http://img22.imageshack.us/img22/6802/texsteiner.th.png) (http://img22.imageshack.us/i/texsteiner.png/) (http://img522.imageshack.us/img522/7586/texvivi.th.png) (http://img522.imageshack.us/i/texvivi.png/) (http://img208.imageshack.us/img208/8929/texfreya.th.png) (http://img208.imageshack.us/i/texfreya.png/) (http://img22.imageshack.us/img22/541/texgoblin.th.png) (http://img22.imageshack.us/i/texgoblin.png/) (http://img208.imageshack.us/img208/2909/texhedge.th.png) (http://img208.imageshack.us/i/texhedge.png/)
-
As you can see I've got textures now. The trick is they're not stored in the exact same cluster instead they're in another cluster that's got the same parent cluster as the one the models are in. I used the material indices to choose what texture to use, because I couldn't be bothered to figure out chunk 0x12 (contains CLUT and tpage for models, says the wiki), and it worked fine with monsters and party members but it'll probably be a problem for NPCs, so next phase will be to understand chunk 0x12.
(http://img22.imageshack.us/img22/6802/texsteiner.th.png) (http://img22.imageshack.us/i/texsteiner.png/) (http://img522.imageshack.us/img522/7586/texvivi.th.png) (http://img522.imageshack.us/i/texvivi.png/) (http://img208.imageshack.us/img208/8929/texfreya.th.png) (http://img208.imageshack.us/i/texfreya.png/) (http://img22.imageshack.us/img22/541/texgoblin.th.png) (http://img22.imageshack.us/i/texgoblin.png/) (http://img208.imageshack.us/img208/2909/texhedge.th.png) (http://img208.imageshack.us/i/texhedge.png/)
Do I see my lovely face-sake in there? I have to have a copy of this when you finish it. Keep it up, you're fulfilling one of my dreams here...
-
Got good news and bad news
Good news are, I can see all weapons too, with textures. Only wireframe for NPCs and map models though, can't find the textures. Also the directory supposedly containing the summons seems to have a different structure, can't read it yet.
Bad news is, my PC was taken out by a trojan. I haven't lost everything, but the project will have to be on hold for a while.
-
Got good news and bad news
Good news are, I can see all weapons too, with textures. Only wireframe for NPCs and map models though, can't find the textures. Also the directory supposedly containing the summons seems to have a different structure, can't read it yet.
Bad news is, my PC was taken out by a trojan. I haven't lost everything, but the project will have to be on hold for a while.
This makes me very sad...and a little bit angry... what purpose do viruses serve!? Why do people make them!!? Do they have any idea how long it can take to restore a computer to it's original state? BASTARDS!
-
Also the directory supposedly containing the summons seems to have a different structure, can't read it yet.
It's a bit different, although the actual data is in the same format (just doesn't have the DB chunk headers). It's padded with 0xFF instead of zero also.
For the 3D model data, you'll have to work with the B and C polygon types, they are only used for summon models. I'll see if I can dig out some old code with a somewhat decent description of them.
Edit:
I was bored... (http://the-world-is.mine.nu/eidolons02.png)
...so bored (http://the-world-is.mine.nu/eidolons01.png)
-
Oh duh! Now you mention it I remember reading about that around here before! well, that'll be the next step once I get my new computer then
EDIT: Ok, up and running again. But I have to confess I can't wrap my mind around the summon directory. It's one of those type 0x03 hierarchical directories, right? So that means what's listed inside is subdirectories? But how do you know how many files each subdirectory contains? Are there headers for them?
EDITEDIT: it's now been remade to use shaders for animation, which is much better (despite being visually identical). Also finally realized what I'd gotten wrong with the lower bits of angles: they're optional, some anims don't contain them, signalled by a zero offset. I can kinda detect which anims match which models now - sorta. The method gives false positives so you can get some aberrant anims, but it's better than nothing. I still don't get how to go through the summons (well, effects, really) directory or find the character textures in the room/worldmap clusters.
I was wondering about exporting though. I wanted to use .OBJ format but it doesn't allow animations...
-
Oh duh! Now you mention it I remember reading about that around here before! well, that'll be the next step once I get my new computer then
EDIT: Ok, up and running again. But I have to confess I can't wrap my mind around the summon directory. It's one of those type 0x03 hierarchical directories, right? So that means what's listed inside is subdirectories? But how do you know how many files each subdirectory contains? Are there headers for them?
EDITEDIT: it's now been remade to use shaders for animation, which is much better (despite being visually identical). Also finally realized what I'd gotten wrong with the lower bits of angles: they're optional, some anims don't contain them, signalled by a zero offset. I can kinda detect which anims match which models now - sorta. The method gives false positives so you can get some aberrant anims, but it's better than nothing. I still don't get how to go through the summons (well, effects, really) directory or find the character textures in the room/worldmap clusters.
I was wondering about exporting though. I wanted to use .OBJ format but it doesn't allow animations...
Well... OBJ is becoming the de facto standard for 3D files recently.... everything can open it... animations though... I never expected you'd export those at all really...
Check 3DS it may have nonskeletal animation capabilities....
You could always export animations as a separate file and let people work on an import/export script to open them in various apps... but you'd need a format that supports skeletons still...
*while typing, I am reminded of morrowind's Nif / Kf(sp?) combo...*
-
Nice work. Sucks about the machine.
Animation usually sucks since there really isn't an good general format. 3DS is probably the best format widely adopted that allows for animation its but not really that good for storing multiple animations in my experience.
BVH is probably the closest to OBJ in simplicity but good luck trying to use that in max on a skeleton. I might recommend using another game format like Half Life's smd or Unreal's psk/psa formats. Both games have a number of importers/exporters for a variety of modeling tools and have reasonably simple text formats as I recall. Gamebyro's NIF/KF are binary and very complicated so no go there but the (optional) separate files is nice when you have need to manage several animations for playback.
I might suggest FBX or COLLADA if you are a masochist. Autodesk which owns nearly every modeling tool anymore supports both in some form for max, maya and xsi. In practice both are too specific and difficult to work with in multiple tools probably ok if you stick to only one tool chain.
-
I'd seriously be happy with just a bind-pose export really... Animations would be EPIC... but a basic pose like FFX/FFVIICC/KH/KHII/etc have for their viewers would still be awesome in my opinion....
-
The thing is, unlike KH there's no bind pose, it's implicit since the vertices are directly stored in bone space. What that means is the characters look like origami, or maybe like they fell into a metal compacter, unless you assign an anim to them
-
I see... that is the oddest thing I've ever heard... I suppose it works though...
-
It actually makes a lot of sense. If you don't store the vertices in bone space you have to transform them by the inverse of the bind pose before you can put them in another pose, which means at minimum one more matrix operation per bone, and that's assuming you've memorized all the inverse transforms. It may be okay for modern machines with memory and power to spare but for FF9 they probably valued such an optimization.
And once you store vertices in bone space there's really no point in storing the bind pose, since it's the one pose you never use in-game.
-
I see... so in order to export at all, you still need a way to store the animations... or at least the first frame of one...
Couldn't you use the first frame of the standing animation for more models?
-
well, each model has its animations in its own order so choosing an appropriate pose can't be automated. But really, my interrogations revolve around which format to choose to keep the hassle to a minimum, not whether to include the anims or not.
-
well, each model has its animations in its own order so choosing an appropriate pose can't be automated. But really, my interrogations revolve around which format to choose to keep the hassle to a minimum, not whether to include the anims or not.
Ok, well... if you're using DX9... 3D Ripper will work on it... so we could capture any frame from any animation...
Or, you could make a manual animation scrub-through option and use whatever frame is currently showing as the bone position, thus taking away the automation issue...
As for formats... if you can transcribe the vertex positions relative to the bone positions into absolute positions then OBJ would be fine. But you wouldn't, obviously, have animations themselves...
-
Collada (.DAE file) Might be a good choice. the FFX/FFX-2 model viewers use it to export model and animation data together.
-
I know collada is supported by maya and I believe it is with max as well... but I have no idea how flexible it is for programming...
-
The bind pose issue certainly is an interesting one though. Makes sense when you explain it.
My problem with COLLADA is I've not been able to take any scene in max export as a DAE and reimport it and get back to effectively the same scene or vice-versa. And usually its not even close. I've tried 2 different max plugins and get different results when using one vs the other. Skin and animation are frequently corrupted in my attempts to use it. Then again this is true with a lot of formats. Last time I looked at it there were not a lot of decent helper libraries. The official one was C++ only.
I've tried transferring assets from Maya to Max with it and ultimately it didn't go that badly but lost a lot of detail like IK Chains and what not. But it was the only thing I could use to make it work (FBX was also tried with similar results but its probably because its basically the same code base).
Anyway, this is just my limited experience. Take a look at and see if you like it. Its probably is one of the better open standards because it seems to supports everything and xml makes it easier to write tools for it. I didn't like the support tools for it on the programming side when I tried it a few years ago. Flexibility and support for a lot of different objects makes learning it more difficult as well but there is pretty good documentation for the file format on the official website.
Overall COLLADA is ok in my opinion. Its basically XML so its really flexible. In practice, it works well for me in cases where OBJ also works well. Ultimately my complaints aren't with the format but with the modeling tools support around it and how most dont seem to fully support the format.
I've been sort of planning to use collada or rather the FBX library from autodesk (which supports fbx, collada, obj, and 3ds for format conversion) for a new project this fall but not really in a position to recommend it. Its Visual C++ only and binary libraries. My hope would be using that would give better max/maya/blender/xsi support without having to completely write something from scratch. Given its from Autodesk more people would likely be used to working around its quirks in the way it exports its files.
-
Blender uses Python...
-
That's right! You could technically just export all the data as a random filetype of your own design or whatever, and use a blender python script as an (re)import tool.
it wouldn't *cough* be the first time I've seen it done...
Which is all moot if you don't know python...
-
Only problem with that is, people with anything other than Blender wouldn't be able to import it. i think there's something called "Maxscript" for 3dsmax, but it looks complicated O.o
-
Well, maxscript has the problem it only works in Max. If I turn to blender, since it's free anybody can install it if only to import the model through a custom script and then re-export it in their fave format, sicne Blender exports to a lot of format IIRC.
Collada's also a good idea since most programs can import it.
-
Blender, much as I loathe it, seems like the best option in terms of simplicity so far...
-
dunno. but since i very interesting :-o with this post, i would try the experiment myselft. perhaps, i will post my project in this forums too. :roll:
-
Well, I'm still on it but there's a lotta work still to do. I've learned python and can make custom Blender exporters fine, but not yet importers, plus I'll also need to rework the interface of my program to something user friendly, still gonna take some time. I also have to add some stuff to my data chunk methods so I can start figuring out how the summons directory works, because it's still a complete mystery to me.
-
Well, it all depends on your affinities, they're totally different things. HTML is a markup language, ie a way to write documents with annotations, and Python is a programming language, ie a way to write computer programs. For me it was easy to learn python because I'm a programmer by trade and already knew quite a few other programming languages (most of them being what we call object-oriented, a specific approach to writing programs, and python is too, which makes them all quite similar) but for someone who isn't familiar with object-oriented programming it's gonna take a bit longer.
if you wanna have a go at it you can go there and download "a byte of python", a very good python tutorial aimed at both newbies and experienced programmers:
http://www.swaroopch.com/notes/Python
Also worth noting I'm only using python for importing/exporting from Blender, the actual FF9 viewer is written in C# and using XNA (http://creators.xna.com/en-US/)
-
So, the viewer is working, just not "user friendly"? Is it directX based?
3D's Via and Ripper DX can steal from DX9 applications....
Via can supposedly rip from Open GL as well...
And, while that isn't as preferable as a full file conversion with animations and all... It would be nice to just be able to get the models at all at this point...
Why not make a sort of WIP release... something so we can at least look at the models for now...
-
It's XNA-based (so yeah, DX9 behind it), grafted on another application of mine which isn't quite meant for distribution because it had a lot of things useful for a model viewer, with a custom WIP GUI system, has to be recompiled when you want to change the active model directory and will crash on any machine but mine because the FF9 resource file location is hardcoded. That's what I call user-unfriendly. It's all small details stemming from the fact it was originally a hack job.
Not really hard to fix apart from the fact it needs a decent UI (a lot of the necessary functions are there already, implemented for other parts of the program), but those fixes take time, and I got in somewhat of a slump while searching for the missing textures and turned my attention to other subjects for a bit.
-
ok, I see what you're saying. Rather than actual "user-friendly" stuff like pretty GUI buttons telling you what you can do, you meant it doesn't have the flexibility to run without the exact setup you are running it from.
That would be a real issue. Well... I can see that it won't be done in the next few seconds (or probably days, likely weeks, considering the whole unfunded programmers having personal lives thing)...
Regardless of how long it takes, I am eager to get an up close look at the models.
I'll be waiting with high hopes for progression speed, but low expectations thereof.
(In truth I generally just slap a bunch of random-ass buttons on things that each do another process that I've already built... but I don't know if XNA is quite as forgiving as standard C#...)
In any case good luck with whatever you're currently working on.
-
XNA is standard C#. I reiterate, all that stuff is a consequence of the way I did things, not the way XNA makes you do things.
-
XNA is standard C#. I reiterate, all that stuff is a consequence of the way I did things, not the way XNA makes you do things.
XNA is C#, yes, but as I haven't messed with it before, I am unsure of the special methods and revisions that exist that make it separate from normal C#.
I'm assuming XNA uses an SDK of some sort and isn't built directly into C#. After all, it wouldn't need a name like XNA if it were a standard part of existing C#.
Invariably when using an SDK of some kind, there are changes in the way certain tasks need to be handled, some are more drastic than others. I was simply stating that I don't know enough about XNA to know how drastic the changes in handling of basic windows forms is.
Quoth the Satoh: Nothing more.
Again, good luck on whatever your current project is.
-
Yeah but I mean, it's like saying using DirectX necessitates some changes. Sure, it does, but they're unrelated to ease of use of the final application. Here the only practical difference is people need to get the XNA runtime to run XNA apps, just like they need the directX runtime to run DirectX apps. Every problem here stems from the fact I've been experimenting more than turning the experiment into a final product.