Author Topic: [FF8] world map and objects (world.fs)  (Read 69250 times)

Halfer

  • *
  • Posts: 142
    • View Profile
[FF8] world map and objects (world.fs)
« on: 2015-03-01 18:15:28 »


First off, huge thanks to MakiPL for inspiring me and hopefully others with his research on ff8 battle fields. Since he's making a very good and much faster progress than i managed to make I decided to hop on to world map data. This concerns wmx.obj for now since I haven't looked on to the other files yet.

Remember that this info may be inaccurate in some parts since it is a huge file (33MB) and it is a pain in the neck to scroll through. However I'll talk about how to handle this file quicker at the end.

Insight:

For now (not sure), the world map seems to consist of 768 sections. Each section contains 16 blocks and each block can have different amount of triangles. Luckily, the sea mesh seems to have 32 triangles  (formation told later) so we can do tests on that more easily. Here are pictures with 1 section/16 blocks and 1 block with 32 triangles:



Header:

Header consists of 68 bytes. Each section contains the same header structure.
  • First 4 bytes are still unknown since changing these don't seem to do anything.
  • Rest 64 bytes indicates the offset of blocks after the header


Data:

Possible render/collision options:
This is very complicated.
After the header, the first 4 bytes seems to remove the whole corresponding block when set to "00 00 00 00", also it creates a collision over it so it can't be fled over. Setting it something else messes with uv's as well as geometry and may cause the game to crash or freeze. Setting it to "30 30 30 30" also affects the next block for some reason. The collision is also changed and in this particular example you can go under the world (something like in ff7 :D).



Triangle data:

The data for 1 triangle is 16 bytes long starting after possible render/collision options.

The next 4 bytes are for triangle vertices. zeroing these gives the following results for one triangle:


Here are the marked triangle vertices:                                                                        And here's how they are formed:





Unknown still:

The next 8 bytes are still unknown, feel free to look at it!
Also the data between in-game blocks in a section is still unknown.

Ground type?

The last 4 bytes for one triangle data seems to contain UV data reference for the corresponding triangle. This is also how I made the example photos for this thread. Changing the value for this changes the triangles type. The marked bytes, when set to "88 88 88 88" gives the ground the type as in in-game example photos:

Also landing on this ground is possible, but as I tested I found out that you can reference to any type of ground with this and I managed to get probably a cliff type where I were unable to land with ragnarok.

Padding:

Following the data for one section comes a bunch of zeros. This is called padding. The data and padding are 36864 or 0x9000 bytes for every section.

How to manage the file:

Since this file is huge I found out a little solution for managing it. One reason for the size of this file is padding. The header, data and padding seems to be fixed amount of 36864 bytes or 0x9000 bytes for easier reading in hex editor. This helps a lot since each section is therefore 0x9000 bytes long (including padding) and for next section you just add 0x9000 bytes more from the beginning of previous section to get for next section.

How to know which section you are editing in game:

This is still TODO. However I think there is 768 sections, but i'm not sure if one section can be larger or smaller than I showcased. For what I know, the first section in the file is top left corner section in-game and the next sections goes horizontally to the top right corner of the in-game map. This way of thinking isn't true in the vertical way however:

If there was 768 sections in the world map and the aspect ratio would be 4:3 like the world map picture, it would mean that the sections would be 32x24. Here's a picture of first block from 12th section of file which possible render/collision options are nullified:


You can see that it is pretty accurately in place.
If we want 12th section vertically it would be (12x32=) 384th section in file, however it isn't correct so the 768 section hypothesis isn't correct.

Although it isn't correct we can use the first horizontal line and maybe the second one for testing purposes with this hypothesis.

Lastly:

I hope this helps someone with their research on this and someone manages to do something huge with this!

Also the very last section includes the model for great salt lake, its train station and little part of the bridge. Just remember if you replace the first section or any other section with this that there is 36864 or 0x9000 bytes in the section including padding.
« Last Edit: 2015-05-22 08:28:07 by Halfer »

paul

  • *
  • Posts: 179
    • View Profile
Re: [FF8] world map object (wmx.obj)
« Reply #1 on: 2015-03-01 18:22:09 »
Nice work, once this is cracked FF7-9 WM's will be reversed so can come up with sane design to support them all in QGears :)

Softtm17

  • *
  • Posts: 33
    • View Profile
Re: [FF8] world map object (wmx.obj)
« Reply #2 on: 2015-03-03 12:20:54 »
Halfer can you give us an help?
Check on this page and you can understand: http://forums.qhimm.com/index.php?topic=15906.msg224982#msg224982
Let me know and great work, keep it up.

Maki

  • 0xBAADF00D
  • *
  • Posts: 621
  • 0xCCCCCCCC
    • View Profile
Re: [FF8] world map object (wmx.obj)
« Reply #3 on: 2015-05-16 16:18:59 »
I wrote a documentation for world map camera translation:
http://wiki.qhimm.com/view/FF8/Engine/WorldMapCamera

I'm going now for world map objects- towns and vehicles.

Ripped Balamb Garden thanks to http://forums.qhimm.com/index.php?topic=13799.msg193791#msg193791 notes.
Converter update is coming. ;)
« Last Edit: 2015-05-16 17:47:56 by MaKiPL »

kaspar01

  • *
  • Posts: 118
  • FFVIII Fan & Collector , 3D Artist , FF8RP-WIP
    • View Profile
Re: [FF8] world map object (wmx.obj)
« Reply #4 on: 2015-05-16 18:41:14 »
I wrote a documentation for world map camera translation:
http://wiki.qhimm.com/view/FF8/Engine/WorldMapCamera

I'm going now for world map objects- towns and vehicles.

Ripped Balamb Garden thanks to http://forums.qhimm.com/index.php?topic=13799.msg193791#msg193791 notes.
Converter update is coming. ;)

This is a damn cool thing to read!

Can't wait to see it  ;D

Maki

  • 0xBAADF00D
  • *
  • Posts: 621
  • 0xCCCCCCCC
    • View Profile
Re: [FF8] world map object (wmx.obj)
« Reply #5 on: 2015-05-17 16:12:35 »

*HEY! This shot is a bit old and uses 1/2 of texture. :D

Phew.

World map objects VT calculation:

Code: [Select]
                const float newSzerU = 256.0f; //CHANGE height * 2
                const float newWysok = 256.0f; //CHANGE width * 2

                float UU1 = (float)U1v / newSzerU;
                float VV1 = 1.0f - (V1v / newWysok);
                float UU2 = (float)U2v / newSzerU;
                float VV2 = 1.0f - (V2v / newWysok);
                float UU3 = (float)U3v / newSzerU;
                float VV3 = 1.0f - (V3v / newWysok);
                float UU4 = (float)U4v / newSzerU;
                float VV4 = 1.0f - (V4v / newWysok);

After model there's of course well known: MODULO(Offset)+4
Balamb halo (or ring, whatever you call it) is second model just after Balamb. Ah, can't wait to see what's next. :)

Train. :3
« Last Edit: 2015-05-17 16:48:57 by MaKiPL »

kaspar01

  • *
  • Posts: 118
  • FFVIII Fan & Collector , 3D Artist , FF8RP-WIP
    • View Profile
Re: [FF8] world map object (wmx.obj)
« Reply #6 on: 2015-05-18 09:05:50 »
AWWWWWW :3

That's amazing to see finally!

Anyway..am I wrong or there are some triangular holes?..maybe some triangular mesh are missing?

Thanks so much for what you're doing :)

Maki

  • 0xBAADF00D
  • *
  • Posts: 621
  • 0xCCCCCCCC
    • View Profile
Re: [FF8] world map object (wmx.obj)
« Reply #7 on: 2015-05-18 16:33:36 »
Yea. Just like the stage meshes, these are divided on quads and triangles. It's only textured quad mesh on picture above. :3
I'm just laaaazy. :D I'm lending you train mesh and need to go for now. :C



Currently examined:
FFVIII_Steam/data/lang-en/world.fs/wmset.obj/
offset;description;TIM_texturePosition;TextureWidth/Height;
("0x3cfc", "Balamb Garden"); #51 (128/64)
("0x4a24", "Balamb Ring/Halo"); #51 (128/64)
("0x4bb0", "Train 1"); #52 (32/64)
("0x51cc", "Train 2"); #53 (16/64)

kaspar01

  • *
  • Posts: 118
  • FFVIII Fan & Collector , 3D Artist , FF8RP-WIP
    • View Profile
Re: [FF8] world map object (wmx.obj)
« Reply #8 on: 2015-05-18 17:40:04 »
I love trains!

:D

Actually from the preview I cant understand if this is more or less detailed than the background model.. (I guess "less" but I could be wrong).


Halfer

  • *
  • Posts: 142
    • View Profile
Re: [FF8] world map object (wmx.obj)
« Reply #9 on: 2015-05-18 18:55:11 »
Glad to see you again Maki :D and great work!

I feel pretty bad for not being able to contribute frequently (lately at all) on cracking the files. I have so much work in school and right now I'm in middle of moving places and well the change of PC has knocked me a little back, but once I get the files and FF8 installed again and settle down in my new apartment, I'm good to get back in FF8 :D

Maki

  • 0xBAADF00D
  • *
  • Posts: 621
  • 0xCCCCCCCC
    • View Profile
Re: [FF8] world map object (wmx.obj)
« Reply #10 on: 2015-05-20 14:06:26 »
I coded an analyzer for ripper. This is complete list of models and it's offsets:

wmset.obj (Steam 2013) SHA1: 31516E8E2452FF65EF6009BB86AB0D7CC58D0E22
Offsets are int.
Code: [Select]
1. 15612 - Balamb Garden
2. 18980 - Balamb Garden Ring
3. 19376 - Train 1
4. 20940 - Train 2
5. 21256
6. 22820
7. 23136
8. 24100
9. 24336
10. 24492
11. 26088
12. 27700
13. 29288
14. 29444
15. 31140
16. 33172
17. 36484
18. 36880
19. 38320
20. 39712
21. 41144
22. 42604
23. 43904
24. 44884
25. 45968
26. 49360
27. 55100
28. 60808
29. 65588
30. 66640
31. 67036
32. 67896
33. 68388

First texture is at: 413048 (EDIT: Was 429980, but Balamb uses two times the same texture, and due to file specific it needs to be repeated two times. 413048 is correct entry.)

Texture offset list:
Where: 52 is 1 in index, and 83 is 31 in index.


Could someone please calculate SHA1 from wmset.obj file from 2000PC release? (lang_en).
« Last Edit: 2015-05-22 17:12:22 by MaKiPL »

Halfer

  • *
  • Posts: 142
    • View Profile
Re: [FF8] world map object (wmx.obj)
« Reply #11 on: 2015-05-20 14:22:11 »
This is SHA1 for wmset.obj old PC release: 31516E8E2452FF65EF6009BB86AB0D7CC58D0E22 according to this site http://onlinemd5.com/

Seems to be same as steam release. I think the files are untouched, only executable is modified in steam release.
« Last Edit: 2015-05-20 14:28:00 by Halfer »

Maki

  • 0xBAADF00D
  • *
  • Posts: 621
  • 0xCCCCCCCC
    • View Profile
Re: [FF8] world map object (wmx.obj)
« Reply #12 on: 2015-05-20 14:48:40 »
Thanks Halfer!
That's cool files are the same! :)
Now I can do a entry table for each model.

Halfer

  • *
  • Posts: 142
    • View Profile
Re: [FF8] world map and objects (world.fs)
« Reply #13 on: 2015-05-22 08:29:46 »
Title updated to not only deal with world map but the whole world container.

Maki

  • 0xBAADF00D
  • *
  • Posts: 621
  • 0xCCCCCCCC
    • View Profile
Re: [FF8] world map and objects (world.fs)
« Reply #14 on: 2015-05-22 16:59:28 »
@up - Okay. :)

---

Wmset.obj is using 8BPP and 4BPP textures, with various resolution and CLUT size +  Clut sets. The problem is some 4BPP textures ARE animated / or just are changing textures, or just work as battle stages do. No TIM documentation says about animated sets. I found out, that it's saved as big-endian two bytes before CLUT. Like every byte there is little-endian, but this one is big-endian and should be reordered in code. Really, really weird. This should be calculated like this then:

Code: [Select]
            //Getting # set
            String CLUT_sets = filebuffer[lookuped+16].ToString("X2");
            CLUT_sets = CLUT_sets.Substring(0, 1);

            if (BPP == 8)
            {
                if(int.Parse(CLUT_sets)>1)
                    tempres = tempres + 20 + (int.Parse(CLUT_sets) * 32);
                else
                    tempres = tempres + 20 + (CLUTs * 32);
            }
            else
            {
                if (int.Parse(CLUT_sets) > 1)
                    tempres = tempres + 20 + (int.Parse(CLUT_sets) * 512);
                else
                    tempres = tempres + 20 + (CLUTs * 512);
            }

            tempres += 8; //Padding + origin rot
            UInt16 width = BitConverter.ToUInt16(filebuffer, tempres);
            UInt16 height = BitConverter.ToUInt16(filebuffer, tempres + 2);
« Last Edit: 2015-05-22 17:06:03 by MaKiPL »

Vehek

  • *
  • Posts: 215
    • View Profile
Re: [FF8] world map and objects (world.fs)
« Reply #15 on: 2015-05-23 02:43:11 »
I'm not sure whether you're already aware of this (especially since I don't know exactly how you're using the textures after loading them), but you have to be careful with UV calculation. I believe the UVs are given for the texture pages, rather than the individual textures, and not all textures have coordinates perfectly aligned at the corners of the texture pages. When working with an extracted texture, you have to account for that difference.

Maki

  • 0xBAADF00D
  • *
  • Posts: 621
  • 0xCCCCCCCC
    • View Profile
Re: [FF8] world map and objects (world.fs)
« Reply #16 on: 2015-05-23 05:59:29 »
I'm currently working on it. Sometimes I need to divide bytes by 256, sometimes by 64 or even 32. I need to find some dependency on it.  This is currently problem number 1.

Maki

  • 0xBAADF00D
  • *
  • Posts: 621
  • 0xCCCCCCCC
    • View Profile
Re: [FF8] world map and objects (world.fs)
« Reply #17 on: 2015-05-24 16:58:29 »
Oops:


List of files that the game can run without:

Code: [Select]
music0.obj - ??
music1.obj - ??
music2.obj - ??
music3.obj - ??
music4.obj - ??
music5.obj - ??
rail.obj - ??
texl.obj - Destroys whole world textures. See image above.
wmset.obj - Contains towns, objects etc. but is NOT used by the game!

Crash:
Code: [Select]
wmsetXX.obj - where XX is language code. This one is actually used in-game.
wmx.obj - Well... :D

I updated wmsetus on wiki:
http://wiki.qhimm.com/view/FF8/WorldMap_wmsetxx

It appears, that wmset does not containg towns. It's not confirmed though!
« Last Edit: 2015-05-24 19:18:06 by MaKiPL »

Halfer

  • *
  • Posts: 142
    • View Profile
Re: [FF8] world map and objects (world.fs)
« Reply #18 on: 2015-05-26 13:48:29 »
Okay, I had spare time to look on the files again and I found an interesting thing in the geometry forming.

In the files it seems that some parts of geometries are duplicated over itself meaning that there is 2 layers overlapping with each other. This led me to remember that in some thread there was discussion about this in some form, but I don't seem to recall/find it anymore. If somebody remembers which thread it was, I would appreciate to know too. I'm pretty sure it was here on Qhimm at least.

EDIT: Good news, I now know how the geometry is read from the file and I'll write an update tomorrow regarding on it.
« Last Edit: 2015-05-26 23:04:45 by Halfer »

Halfer

  • *
  • Posts: 142
    • View Profile
Re: [FF8] world map and objects (world.fs)
« Reply #19 on: 2015-05-27 14:19:00 »
As mentioned in the first post, each section is size of 0x9000h with padding and it represents small are on map (see first post).

Sections start with offsets of blocks which are formed inside section. On each offset the block starts with 4 bytes long header.

Code: [Select]
struct
{

        u32 unknown (sections always starts with 4 bytes which affect nothing)
        u32 block_offset[amount of blocks]; // (first offset value/4-1) = amount of blocks
        header headerData ;
        face faceData[amount of faces];
        vertex verticeData[amount of vertices];
        shadows data[size differs]; // possible shadowing, on continents this sometimes match the amount of vertices.
        padding; // to next 0x9000h offset

} Section
       

Code: [Select]
struct
{
        u8 face_count; (mostly triangles or quads)
        u8 vertex_count;
        u8 shadow_count;  // shaders or something related to make shadows
        u8 unknown, never used??

}header, size = 4 bytes

Code: [Select]
struct
{

        u8 vertexIndices[6];
        u8 tex_coordX for v 0;
        u8 tex_coordY for v 0;
        u8 tex_coordX for v 1;
        u8 tex_coordY for v 1;
        u8 tex_coordX for v 2;
        u8 tex_coordY for v 2;
        u8 face type related [3]    // (texture page, type, etc...)
        s8 collision; //needs a bit research

} Face
size = 16 bytes

Code: [Select]
struct
{

        s16 coordinates[3]; // in form of XZY or YZX
        s16 unknown, unused (optional W dimension probably)

} Vertex

Code: [Select]
struct
{

        s16 or u16 unknown data [3]  // shadow possibly on XYZ (XZY) coordinate.
        u16 always zeros ??

} Shadows, under inspection still.

EDIT: texture related values may not be correct since they are not yet inspected properly.
« Last Edit: 2015-07-11 17:52:56 by Halfer »

Maki

  • 0xBAADF00D
  • *
  • Posts: 621
  • 0xCCCCCCCC
    • View Profile
Re: [FF8] world map and objects (world.fs)
« Reply #20 on: 2015-05-27 18:40:45 »
You're a God Halfer! :)

It works indeed.


Balamb part is at 1d13000-1d198a0.
Towns and garden is made from triangles and is part of world map, so yes, towns are in wmx.obj.

Also, I'm like 70% sure rail.obj is for trains movement. Deleting file makes train blink sometimes and go crazy on world map. Like show for 1/2 second on center of sea.

@EDIT: I don't know if it's in fact vertex data, probably not, but one byte operates shadow. Probably that unknown data is about shadowing.
« Last Edit: 2015-05-27 18:47:54 by MaKiPL »

Halfer

  • *
  • Posts: 142
    • View Profile
Re: [FF8] world map and objects (world.fs)
« Reply #21 on: 2015-05-27 19:25:17 »
Yeah, that would probably make sense. In the water sections where there is only water blocks the "shadow" part is only 0x01 long which is 8 bytes right after vertex data. That is probably why I had trouble looking at that data because the sea doesn't really need shadowing. However it looks like in continents and other areas where "ground" is present the "shadow" data matches the amount of vertices which may indeed be for shadowing. I'll edit this to structures.

So towns are in wmx.obj? That's interesting. I wonder where the sections are set to match the exact position in world. I looked in FF7 wiki for the world info and in FF7 files it seemed to be separate file which handled the positioning of loaded sections. I'm starting to feel that they are coded in .exe, hopefully not.

EDIT: Also worth to mention that the vertices can be positioned outside it's block and section but the collision settings don't follow the boundaries, but are stitched to its block or section boundaries. In other words the section which is subject to world has priority on its own area. Here's an example:



Here the vertices are stretched on top of other section and while I can walk on the current area, I can't go over the area which is on top of water.

I'm not 100% if this is how it really works or if there are exceptions, but right now this seems logical.
« Last Edit: 2015-05-27 19:53:58 by Halfer »

Maki

  • 0xBAADF00D
  • *
  • Posts: 621
  • 0xCCCCCCCC
    • View Profile
Re: [FF8] world map and objects (world.fs)
« Reply #22 on: 2015-05-27 20:51:31 »
Ran a script and generated complete offset list for every world segment. There are total of 835 segments.
http://pastebin.com/WLRjSDFj

I can't help more today. Too tired.

Halfer

  • *
  • Posts: 142
    • View Profile
Re: [FF8] world map and objects (world.fs)
« Reply #23 on: 2015-05-27 21:02:35 »
Cool!

In the first post I stated that there was 768 segments in the world map and I based that in that 835 segments value since I found out that the last segments include great salt lake, the bridge and other areas. Seems like in reality it may be even less, maybe half of the size or something, since the sections don't need to be the same size unless they are preset somewhere.

Aavock

  • *
  • Posts: 303
    • View Profile
Re: [FF8] world map and objects (world.fs)
« Reply #24 on: 2015-05-28 02:30:22 »
Great breakthrough! Since you're both working on this, is it possible to use the hi-res worldmap textures and the UV coordinates from the PSX version?

Don't know if someone noticed before but PSX textures are better than PC as for the worldmap;  I've been looking in the world.fs file and there is 2 kinds of worldmap textures, one is low res and the other is 2 times bigger; the road doesn't even have borders... is the PC version uses the low res textures only ?