Author Topic: [FF8] .x battle fields (Read 98112 times)

Halfer · « **on:** 2015-01-13 16:55:51 »

Full documentation of battlefields done by MaKiPL found here: http://wiki.qhimm.com/view/FF8/FileFormat_X

Tool to extract battle stages found here: http://forums.qhimm.com/index.php?topic=16017.0

Hello!

I've started to study a bit of Final Fantasy 8 files and I happened to see that there was no documentation of .x battle fields in Qhimm's forum. I didn't find any threads about this file format or I completely missed it so I decided to make this thread.

After searching about this format I came to conclusion that this is DirectX file. However they are compressed and they are not lzs compressed (maybe a modified version?). Since I don't have any experience in decompressing, I wanted to ask if someone has already worked around this or if it's relatively easy to work on based on success of decompressing other files?

Accessing these files would be the next big leap after we finally got the textures working.

Also on side note (off-topic), I found something interesting from wmx.obj (world map object?) while opening it with NuGraf. NuGraf reported that the password was wrong so maybe it's encrypted?

Maki · « **Reply #1 on:** 2015-01-25 13:39:13 »

Completed WIKI page: http://wiki.qhimm.com/view/FF8/FileFormat_X

Maki · « **Reply #2 on:** 2015-01-25 15:23:30 »

I took a0stg105, and HEX'd fun it.

Code: [Select]

#OFFSET IN HEX                                 #DEFINITION
0x48-49                                              Changing this does nothing
0x5C-5D                                              Before map load, there is a cursor that blink once, after that map loads. Sometimes
0x588-58E                                          Completely nothing.
0x5DA-5DB                                          Still nothing
0x5D6-9A5                                        ##CAMERA HERE! - I reversed bytes from end to start by this offset, and camera is fixed no matter what. Also some enemies can be off-screen.

Summary:

Whole code from 000-5F1 is probably the same. I'll check with other maps.

NULL'ing header makes no effect... really... Game doesn't care what it is.

0-23A can be NULL'ified and the game doesn't care. WTF?
Huh...
NULL'ied from 0x0 to 0x5d3 and guess what... NOTHING

EDIT2:
Finally, so te textures are like:

LINE1
0101 01 01 01 01 01 01 01 01 01 01 01 01 01 01 01 01
LINE 2
etc...
It's pallete colored. This is detailed somewhere here in Texture section. So, I nulled from beggining to 0x998 and still no vertex glitches or anything.
I nulled one line covered by those 0101010101, and here it is:

EDIT3:
Yea, found something, probably reversed the "pallete" or UV. More like both. it is 44CE to 4905

Summary:
File is divided in some parts, where:

Code: [Select]

STAGE 1 - from START to 0x??? - ???
STAGE 2 - from 0x5D6 to 0x9A5 - Camera *THIS NEEDS TO BE NARROWED! Narrowed by 1/2
STAGE 3 - from 0x??? to 0x??? - Vertex data
STAGE 4 - Search for first 0x/10,0x/00,0x/00,0x/00 - Texture

Maki · « **Reply #3 on:** 2015-01-27 11:49:30 »

Okay. The texture is no more than just a TIM file, starting with 10 00 00 00. Thanks to paul and _Tom_ for TIM header infos.

DEPRECATED:
*Updated "summary" post above. I finally found where the texture starts. RGBA pixel format is probably 555, Didn't test it yet, but that's what wiki says here for FF7 textures.

The texture section, clearly starts at repeating 01 01 01... bytes. After each line it has a tendency to drop to small numbers. For example the first one header for top-one pixel line is 178 bytes long, where the next one header for second pixel has 176 as for a0stg105.x

So, to find the texture, and re-export it, or EVEN replace, just find the first appearance of 01 01 01 01 01 01 bytes starting from 0x0 offset, and you should get it in all normal BG's.
See below:

Ragna · « **Reply #4 on:** 2015-01-27 14:14:37 »

.X files?

In FF7 those are gzip compressed files like BATTLE.X and others in the same folder.
If you could post the first 16 bytes... maybe is the same or similar?

Maki · « **Reply #5 on:** 2015-01-27 14:21:50 »

I tried to decode it using GZIP deflate, but it thrown exceptions. (GZIP string decompress and GZIP archive unpacking using built-in .NET System.Compression).
Also tried with RFC1950, this one used Knightshift for saves encoding, I had to write my own software for that, but when tried with FF8 files, it just thrown another exception for bad file.

The header for .X files in every stage file is the same, and the first 16 bytes are:

Code: [Select]

E8 FF BD 27 01 00 02 24 14 00 BF AF 11 00 82 10
The same

Code: [Select]

E8 FF BD 27 are used for Final Fantasy Tactics as in battle file there. That's what I read from someone's "Battle hacking" post.

However, I NULL'ed this, and 2560 (sic!) next bytes, and game doesn't care, nothing changed and everything is 100% functional. So, 2560 (currently, I'm going to nullify next bytes later to see what I get) non-needed data?

Ragna · « **Reply #6 on:** 2015-01-27 17:04:10 »

Ah, I expected to see a

Code: [Select]

1F 8B 08 there in the middle but seems like I was wrong.

Four first bytes are usually just a useless tag for format identification so you should try nulling further bytes in the 8-16 first bytes.
Perhaps there's data like compression type, offsets, or uncompressed file size or something? (that should affect the game...)

EDIT: Or maybe the file is not compressed at all? Or only parts if it is a multi-file compressed directory with offsets to files and flags for compressed/not-compressed?

paul · « **Reply #7 on:** 2015-01-27 19:24:17 »

you can just call me paul, not "paulwork"

are you interested in hacking world map model format too? looks like you're making good progress with this.

Edit: P.S I would help you out but I don't own a copy of the game.

Zervox · « **Reply #8 on:** 2015-01-28 00:04:46 »

Just wanted to say, even if a file has a .something extension doesn't mean it is the format you found.
Just like some Engines can use .obj doesn't mean it is necessarily wavefront .obj but rather an object in its entirety which could be
name of object
AI module link
Mesh info
Texture info
other engine's object properties for use in game like, stats or similar.
maybe it is keeping records of where in the world it was called from so that a player can be returned to that location.

and in this case .x seems to include other data than what would need to be in a normal directx .x file.

Maki · « **Reply #9 on:** 2015-01-28 11:37:48 »

Quote from: paul on 2015-01-27 19:24:17

are you interested in hacking world map model format too? looks like you're making good progress with this.

Edit: P.S I would help you out but I don't own a copy of the game.

Sure I am. I still however have no idea if it's just a big 3D Model, or this is a package containing maps, that are loaded when needed in parts. Just like in MMO games.

I could send you some samples with stages and this world.

@Zervox - of course it does. It's not normal DirectX file, also this one contains uncompressed .TIM hardcoded. That file contains camera data. I'm an Unreal Engine developer, and as far as I know OBJ's nor X files does NOT contain textures and camera. Things like that are available to FBX files (Autodesk), but obj has always contained mtl file and refferenced to external texture, so does DirectX file.

UPDATE: Narrowed the camera, looks like the camera X Y Z fixed is somewhere 0x5D5 and 7B8, as I FF'd it and camera was outside of battlefield.
UPDATE2: Uhum... FFVIII finally is going into infinite error loop (No exception, but still something wrong, so it's not going to next stage... God, wish to have a log option to see what it does generate... Looks like I need to copy original file again, and start from beginning knowing where to stop null'ying data. But I think camera is somewhere near. I can smell it.

Look at this:

0x5d4-0x5d7 is responsible for OPENING battle camera. It is possible reverse byte. I'm trying to find how's it going, but every next byte is responsible for X/Y for key in camera animation, so changing one byte can make camera in some places jump, so it's clearly camera key movement edit.

Kaldarasha · « **Reply #10 on:** 2015-01-28 12:29:21 »

Quote

Sure I am. I still however have no idea if it's just a big 3D Model, or this is a package containing maps, that are loaded when needed in parts. Just like in MMO games.

Maybe it would be good to extract the wm from the PSX game if possible and generate out of it a new wm with correct UV coordinates. At last this would be better as fixing the whole thing by hand.

Mcindus · « **Reply #11 on:** 2015-01-29 01:59:22 »

Quote from: Kaldarasha on 2015-01-28 12:29:21

Maybe it would be good to extract the wm from the PSX game if possible and generate out of it a new wm with correct UV coordinates. At last this would be better as fixing the whole thing by hand.

Yes, mainly because it looks like the walkmesh is doubled over itself in the PC version (Kaldarasha has gone over this many times). Essentially, if we could just find out the differences between the PC wm and the PSX wm, we should be able to fix the PC wm.

Maki · « **Reply #12 on:** 2015-01-29 13:56:09 »

@Updated my first post here with the newest info

GAH! Finally! I found the camera "header", or at least something, that makes the camera go crazy if changed one of the parameter. This pseudo-header is:

Code: [Select]

02 00 08 00 20Whether stage it is, it's always fixed.

Changing third byte from 08 to 07 makes camera freeze at the battle no matter what. Conclusion? Changing this makes the game not detect global camera (probably), so when it does no see 02 00 08 00 20, then the battle camera freeze.

What do we got now?

*Stage file contains normal .TIM texture starting with header

Code: [Select]

01 00 00 00*Camera data starts with header

Code: [Select]

02 00 08 00 20 Does this sound familiar to anyone of you?

EDIT: Yup, tested it on many stages, and looks like changing 08 to 07 will always freeze the camera at the end spot of opening scene.

Maki · « **Reply #13 on:** 2015-01-30 11:49:08 »

Click below for video

This video shows many aspects of camera edit, also reverse bytes of UV+vertices probably at the end.

Halfer · « **Reply #14 on:** 2015-01-30 11:58:31 »

I'm not quite sure about this but correct me if i'm wrong. Isn't it a bit hard to analyze the camera settings or any other variables if you don't know if the file is compressed. I mean, can't variables change a bit randomly if it turns out it is compressed? Although I don't see any reason for it to be compressed unless the whole camera scene is included in the file which feels pretty dumb thing to do by a developer.

Maki · « **Reply #15 on:** 2015-01-30 12:02:49 »

This file does not look like compressed. Also, the camera data is having the same bytes in some characteristic places in all stages. Now I want just to get X Y Z of vertices, and just search for them inside stage file.

Maki · « **Reply #16 on:** 2015-01-30 14:33:00 »

MILESTONE! Updated first post, check it for summary, newest updates, and everything

Found geometry data!

It is as I can see, on one battlefield, there is more than one or two objects here.
Now, I'm going to reproduce and see how's made, find the starting header offset, see where is the length byte, and try to reproduce it manually from this data part, to see if it's reverse bytes and etc.

Yup. Wiped out one geometry object, that starts with specific header I'm still looking for.

See? MaKiPL stole the sky! Oh no!

EDIT3:
Sky appears to be the very first geometry object here. Searching now for trees.

EDIT4:
Oops

Nailed it.

EDIT5:
That's funny

EDIT6:
HAHA!

Model set information:

Code: [Select]

Rotation rate for forest model REVERSE-BYTES!  0x7282-0x7285 (4 bytes). 
EXAMPLE:
00 38 00 02 - is for forest model group, to be in correct way, changing last byte makes rotation go 90o, changing third byte makes rotation go up to:
00-0
FF-45o (probably)
and so on...

Extreme milestone!
I probably found out how single verticle is made, also how they're connected to faces. It needs some testing though.

Single verticle:

Code: [Select]

20 00 35 00 21 00 36 00 54 9c 80 3c 54 68 b2 00 7e 9c 7e 68 80 80 80 2c (24Bytes)

That's what I found is:
0x0 (1B) - Stands for ID, with every verticle it raises up by 1.
0x2 (1B) - Same as above (?)
0x3 (17B) - Not yet researched
0x21 (3B) - It does repeat in every single object
0x24 (1B) - End of object.

Way how they're connected into faces is TODO. Still there's a lot of work.

A debug tip!
FFVIII Steam release produce crash log file .DMP. This one can be really fast opened by Visual Studio, also with detailed exception name. All I got before was "Unknown exception", here in VS you can get detailed exception, also debug software with VS. Gonna check it later, I'm tired ATM.

Halfer · « **Reply #17 on:** 2015-01-30 19:36:56 »

Okay here's something I pulled off.

Found at offset 0x3d90. this byte seems operate with ground mesh somehow

Code: [Select]

01 00 00 00Nullifing the first byte vanishes the ground, not sure if the object itself is removed, parts of it, or the texture.

Maki · « **Reply #18 on:** 2015-01-31 12:26:31 »

Okay. I tried to find some dependence on it, but no luck. Hm, by the way great find Halfer. Looks like there's still a lot to do.

EDIT2: Found again some vertices, this time for ground, it's just a some bytes after your found Halfer, nice job! Still working. If I find the good place I could change the values of specific vertices, to see how it's manipulated, and if it's X Y Z normal.

Also this 01 00 00 00 looks just like a hide option. It's starting parameter for this first geometry object. So I was wrong. The very first model here is ground, then forest and sky the 3rd one.

EDIT3: Hey, it appears, that the next bytes are the face connectors!
EDIT4: Mhm... Wait a second! I got something!
EDIT5: There're FOUR objects in this model.

Every objects starts with specific header:

Code: [Select]

01 00 00 00 08 00 00 00 01 00 00 38 00 02 00 00 03 00 00 00 14 00 00 00 84*Thanks Halfer again, this just opened my eyes for new aspect. Now I'm 90% sure I found every object start.
01 00 00 00 - Changing this either makes whole object ignore to load or crash
08- Changing this crashes level
01 00 - Changing first byte to 0 always hides the mesh, no matter what.
00 38 00 02 - rotation rate, calculated backward, so it should be read by: 02 00 38 00
00 00 03 00 - Currently finding out
and etc.

EDIT6: WAIT... OMG... What a strawberries... ;/ I'm off... It's just too much..

Halfer · « **Reply #19 on:** 2015-01-31 13:23:14 »

Yeah no problem, I found out also that the next parameters does something to the faces/vertices when bytes are reversed, seems like you also found it so keep us updated if you find the structure!

Maki · « **Reply #20 on:** 2015-01-31 18:57:24 »

I just tried using some of the headers I found at a0stg105.x, to a0stg012, and... It doesn't work. So, it's compressed? Changing the parameter I had to show/hide objects is working here like a camera freeze... Ugh... I'm losing it... I'm off for today, maybe someone find something out. Could it be really gzip'ed with moved header? I don't know.

EDIT: Nope. I'm just tired and do mistakes. It works the same way as on other stages:

Also there's 1981 vertices, so it does look like it only hides the mesh or craps it's UV map. Or maybe it was 3000 vertices before ground dissapearal? Nah, I'll leave it for the next time... Too tired now.

Also thanks to.. rcxrdx, for patch, to display more debug strings for debugger.

Halfer · « **Reply #21 on:** 2015-01-31 19:04:54 »

That may be the case, I hope someone with lot of knowledge in compressions could have a look at this. I did find out that after the header in 0x3d90 the variables are for vertices/faces for the ground part. However I thought about the structure for a long time but it just didn't make sense. Maybe more experiments would do the work but yeah.... Compression would still be a big limiting factor in cracking the format.

Maki · « **Reply #22 on:** 2015-02-01 13:30:26 »

Used Hex2OBJ, to visualize point cloud, as I was refereed to on XentaX forum.

*This ARE in fact vertices starting at this 01 00 00 00 08 ...
0x3D90 - 119 vertices
Object 1:

paul · « **Reply #23 on:** 2015-02-01 19:38:51 »

So that must be the ground plane?

Maki · « **Reply #24 on:** 2015-02-01 20:14:11 »

Yep. Two other objects are like spheres, but with some details. I'll take a photo of it tomorrow. Still searching for a faces info. I'm having a huge support on Xentax, so I think we'll find it eventually. I'll make you updated

Still however no one could identify this data from 0x0 to 0x5D4. On 5D4 is camera starting, but before that? Nothing. Completely nothing. If anyone of us could get the breakpoint when this portion of data is accessed would be awesome, as I'm having almost no experience in process debugging and etc.