Qhimm.com Forums

Miscellaneous Forums => Scripting and Reverse Engineering => Topic started by: Halfer on 2015-03-01 18:15:28

Title: [FF8] world map and objects (world.fs)
Post by: Halfer on 2015-03-01 18:15:28
(http://i.gyazo.com/a8b92ad4d55c9d77a1136e3e27287f8a.png)

First off, huge thanks to MakiPL for inspiring me and hopefully others with his research on ff8 battle fields. Since he's making a very good and much faster progress than i managed to make I decided to hop on to world map data. This concerns wmx.obj for now since I haven't looked on to the other files yet.

Remember that this info may be inaccurate in some parts since it is a huge file (33MB) and it is a pain in the neck to scroll through. However I'll talk about how to handle this file quicker at the end.

Insight:

For now (not sure), the world map seems to consist of 768 sections. Each section contains 16 blocks and each block can have different amount of triangles. Luckily, the sea mesh seems to have 32 triangles  (formation told later) so we can do tests on that more easily. Here are pictures with 1 section/16 blocks and 1 block with 32 triangles:
(http://i.gyazo.com/08fab4522ad4da3433558e5bf1801cca.png)(http://i.gyazo.com/3933d32ff14ac88a58c62589045ab4b9.png)


Header:

Header consists of 68 bytes. Each section contains the same header structure.
(http://i.gyazo.com/9196f85953c06068de6c85ff17f32047.png)

Data:

Possible render/collision options:
This is very complicated.
After the header, the first 4 bytes seems to remove the whole corresponding block when set to "00 00 00 00", also it creates a collision over it so it can't be fled over. Setting it something else messes with uv's as well as geometry and may cause the game to crash or freeze. Setting it to "30 30 30 30" also affects the next block for some reason. The collision is also changed and in this particular example you can go under the world (something like in ff7 :D).



Triangle data:

The data for 1 triangle is 16 bytes long starting after possible render/collision options.

The next 4 bytes are for triangle vertices. zeroing these gives the following results for one triangle:
(http://i.gyazo.com/693f5c73cfecaa814c2f1fd011e84124.png)

Here are the marked triangle vertices:                                                                        And here's how they are formed:

(http://i.gyazo.com/48189adec5ca94522f4a47d8bbd48fb0.png)(http://i.gyazo.com/92ee9bc9731cd4280217803737fd3797.png)



Unknown still:

The next 8 bytes are still unknown, feel free to look at it!
Also the data between in-game blocks in a section is still unknown.

Ground type?

The last 4 bytes for one triangle data seems to contain UV data reference for the corresponding triangle. This is also how I made the example photos for this thread. Changing the value for this changes the triangles type. The marked bytes, when set to "88 88 88 88" gives the ground the type as in in-game example photos:
(http://i.gyazo.com/2dbef39bac5c29232491fbc40ebded00.png)
Also landing on this ground is possible, but as I tested I found out that you can reference to any type of ground with this and I managed to get probably a cliff type where I were unable to land with ragnarok.

Padding:

Following the data for one section comes a bunch of zeros. This is called padding. The data and padding are 36864 or 0x9000 bytes for every section.

How to manage the file:

Since this file is huge I found out a little solution for managing it. One reason for the size of this file is padding. The header, data and padding seems to be fixed amount of 36864 bytes or 0x9000 bytes for easier reading in hex editor. This helps a lot since each section is therefore 0x9000 bytes long (including padding) and for next section you just add 0x9000 bytes more from the beginning of previous section to get for next section.

How to know which section you are editing in game:

This is still TODO. However I think there is 768 sections, but i'm not sure if one section can be larger or smaller than I showcased. For what I know, the first section in the file is top left corner section in-game and the next sections goes horizontally to the top right corner of the in-game map. This way of thinking isn't true in the vertical way however:

If there was 768 sections in the world map and the aspect ratio would be 4:3 like the world map picture, it would mean that the sections would be 32x24. Here's a picture of first block from 12th section of file which possible render/collision options are nullified:
(http://i.gyazo.com/130ae765eb7138d9603ddf97d3444a10.png)

You can see that it is pretty accurately in place.
If we want 12th section vertically it would be (12x32=) 384th section in file, however it isn't correct so the 768 section hypothesis isn't correct.

Although it isn't correct we can use the first horizontal line and maybe the second one for testing purposes with this hypothesis.

Lastly:

I hope this helps someone with their research on this and someone manages to do something huge with this!

Also the very last section includes the model for great salt lake, its train station and little part of the bridge. Just remember if you replace the first section or any other section with this that there is 36864 or 0x9000 bytes in the section including padding.
Title: Re: [FF8] world map object (wmx.obj)
Post by: paul on 2015-03-01 18:22:09
Nice work, once this is cracked FF7-9 WM's will be reversed so can come up with sane design to support them all in QGears :)
Title: Re: [FF8] world map object (wmx.obj)
Post by: Softtm17 on 2015-03-03 12:20:54
Halfer can you give us an help?
Check on this page and you can understand: http://forums.qhimm.com/index.php?topic=15906.msg224982#msg224982
Let me know and great work, keep it up.
Title: Re: [FF8] world map object (wmx.obj)
Post by: Maki on 2015-05-16 16:18:59
I wrote a documentation for world map camera translation:
http://wiki.qhimm.com/view/FF8/Engine/WorldMapCamera

I'm going now for world map objects- towns and vehicles.

Ripped Balamb Garden thanks to http://forums.qhimm.com/index.php?topic=13799.msg193791#msg193791 notes.
Converter update is coming. ;)
Title: Re: [FF8] world map object (wmx.obj)
Post by: kaspar01 on 2015-05-16 18:41:14
I wrote a documentation for world map camera translation:
http://wiki.qhimm.com/view/FF8/Engine/WorldMapCamera

I'm going now for world map objects- towns and vehicles.

Ripped Balamb Garden thanks to http://forums.qhimm.com/index.php?topic=13799.msg193791#msg193791 notes.
Converter update is coming. ;)

This is a damn cool thing to read!

Can't wait to see it  ;D
Title: Re: [FF8] world map object (wmx.obj)
Post by: Maki on 2015-05-17 16:12:35
(http://i.imgur.com/OdBu1vW.jpg?1)
*HEY! This shot is a bit old and uses 1/2 of texture. :D

Phew.

World map objects VT calculation:

Code: [Select]
                const float newSzerU = 256.0f; //CHANGE height * 2
                const float newWysok = 256.0f; //CHANGE width * 2

                float UU1 = (float)U1v / newSzerU;
                float VV1 = 1.0f - (V1v / newWysok);
                float UU2 = (float)U2v / newSzerU;
                float VV2 = 1.0f - (V2v / newWysok);
                float UU3 = (float)U3v / newSzerU;
                float VV3 = 1.0f - (V3v / newWysok);
                float UU4 = (float)U4v / newSzerU;
                float VV4 = 1.0f - (V4v / newWysok);

After model there's of course well known: MODULO(Offset)+4
Balamb halo (or ring, whatever you call it) is second model just after Balamb. Ah, can't wait to see what's next. :)

Train. :3
Title: Re: [FF8] world map object (wmx.obj)
Post by: kaspar01 on 2015-05-18 09:05:50
AWWWWWW :3

That's amazing to see finally!

Anyway..am I wrong or there are some triangular holes?..maybe some triangular mesh are missing?

Thanks so much for what you're doing :)
Title: Re: [FF8] world map object (wmx.obj)
Post by: Maki on 2015-05-18 16:33:36
Yea. Just like the stage meshes, these are divided on quads and triangles. It's only textured quad mesh on picture above. :3
I'm just laaaazy. :D I'm lending you train mesh and need to go for now. :C

(http://i.imgur.com/K0epbMg.jpg)

Currently examined:
FFVIII_Steam/data/lang-en/world.fs/wmset.obj/
offset;description;TIM_texturePosition;TextureWidth/Height;
("0x3cfc", "Balamb Garden"); #51 (128/64)
("0x4a24", "Balamb Ring/Halo"); #51 (128/64)
("0x4bb0", "Train 1"); #52 (32/64)
("0x51cc", "Train 2"); #53 (16/64)
Title: Re: [FF8] world map object (wmx.obj)
Post by: kaspar01 on 2015-05-18 17:40:04
I love trains!

:D

Actually from the preview I cant understand if this is more or less detailed than the background model.. (I guess "less" but I could be wrong).

Title: Re: [FF8] world map object (wmx.obj)
Post by: Halfer on 2015-05-18 18:55:11
Glad to see you again Maki :D and great work!

I feel pretty bad for not being able to contribute frequently (lately at all) on cracking the files. I have so much work in school and right now I'm in middle of moving places and well the change of PC has knocked me a little back, but once I get the files and FF8 installed again and settle down in my new apartment, I'm good to get back in FF8 :D
Title: Re: [FF8] world map object (wmx.obj)
Post by: Maki on 2015-05-20 14:06:26
I coded an analyzer for ripper. This is complete list of models and it's offsets:

wmset.obj (Steam 2013) SHA1: 31516E8E2452FF65EF6009BB86AB0D7CC58D0E22
Offsets are int.
Code: [Select]
1. 15612 - Balamb Garden
2. 18980 - Balamb Garden Ring
3. 19376 - Train 1
4. 20940 - Train 2
5. 21256
6. 22820
7. 23136
8. 24100
9. 24336
10. 24492
11. 26088
12. 27700
13. 29288
14. 29444
15. 31140
16. 33172
17. 36484
18. 36880
19. 38320
20. 39712
21. 41144
22. 42604
23. 43904
24. 44884
25. 45968
26. 49360
27. 55100
28. 60808
29. 65588
30. 66640
31. 67036
32. 67896
33. 68388

First texture is at: 413048 (EDIT: Was 429980, but Balamb uses two times the same texture, and due to file specific it needs to be repeated two times. 413048 is correct entry.)

Texture offset list:
Where: 52 is 1 in index, and 83 is 31 in index.
(http://i.imgur.com/IRIkmsZ.jpg)

Could someone please calculate SHA1 from wmset.obj file from 2000PC release? (lang_en).
Title: Re: [FF8] world map object (wmx.obj)
Post by: Halfer on 2015-05-20 14:22:11
This is SHA1 for wmset.obj old PC release: 31516E8E2452FF65EF6009BB86AB0D7CC58D0E22 according to this site http://onlinemd5.com/

Seems to be same as steam release. I think the files are untouched, only executable is modified in steam release.
Title: Re: [FF8] world map object (wmx.obj)
Post by: Maki on 2015-05-20 14:48:40
Thanks Halfer!
That's cool files are the same! :)
Now I can do a entry table for each model.
Title: Re: [FF8] world map and objects (world.fs)
Post by: Halfer on 2015-05-22 08:29:46
Title updated to not only deal with world map but the whole world container.
Title: Re: [FF8] world map and objects (world.fs)
Post by: Maki on 2015-05-22 16:59:28
@up - Okay. :)

---

Wmset.obj is using 8BPP and 4BPP textures, with various resolution and CLUT size +  Clut sets. The problem is some 4BPP textures ARE animated / or just are changing textures, or just work as battle stages do. No TIM documentation says about animated sets. I found out, that it's saved as big-endian two bytes before CLUT. Like every byte there is little-endian, but this one is big-endian and should be reordered in code. Really, really weird. This should be calculated like this then:

Code: [Select]
            //Getting # set
            String CLUT_sets = filebuffer[lookuped+16].ToString("X2");
            CLUT_sets = CLUT_sets.Substring(0, 1);

            if (BPP == 8)
            {
                if(int.Parse(CLUT_sets)>1)
                    tempres = tempres + 20 + (int.Parse(CLUT_sets) * 32);
                else
                    tempres = tempres + 20 + (CLUTs * 32);
            }
            else
            {
                if (int.Parse(CLUT_sets) > 1)
                    tempres = tempres + 20 + (int.Parse(CLUT_sets) * 512);
                else
                    tempres = tempres + 20 + (CLUTs * 512);
            }

            tempres += 8; //Padding + origin rot
            UInt16 width = BitConverter.ToUInt16(filebuffer, tempres);
            UInt16 height = BitConverter.ToUInt16(filebuffer, tempres + 2);
Title: Re: [FF8] world map and objects (world.fs)
Post by: Vehek on 2015-05-23 02:43:11
I'm not sure whether you're already aware of this (especially since I don't know exactly how you're using the textures after loading them), but you have to be careful with UV calculation. I believe the UVs are given for the texture pages, rather than the individual textures, and not all textures have coordinates perfectly aligned at the corners of the texture pages. When working with an extracted texture, you have to account for that difference.
Title: Re: [FF8] world map and objects (world.fs)
Post by: Maki on 2015-05-23 05:59:29
I'm currently working on it. Sometimes I need to divide bytes by 256, sometimes by 64 or even 32. I need to find some dependency on it.  This is currently problem number 1.
Title: Re: [FF8] world map and objects (world.fs)
Post by: Maki on 2015-05-24 16:58:29
Oops:
(http://i.imgur.com/wKimjpQ.jpg)

List of files that the game can run without:

Code: [Select]
music0.obj - ??
music1.obj - ??
music2.obj - ??
music3.obj - ??
music4.obj - ??
music5.obj - ??
rail.obj - ??
texl.obj - Destroys whole world textures. See image above.
wmset.obj - Contains towns, objects etc. but is NOT used by the game!

Crash:
Code: [Select]
wmsetXX.obj - where XX is language code. This one is actually used in-game.
wmx.obj - Well... :D

I updated wmsetus on wiki:
http://wiki.qhimm.com/view/FF8/WorldMap_wmsetxx

It appears, that wmset does not containg towns. It's not confirmed though!
Title: Re: [FF8] world map and objects (world.fs)
Post by: Halfer on 2015-05-26 13:48:29
Okay, I had spare time to look on the files again and I found an interesting thing in the geometry forming.

In the files it seems that some parts of geometries are duplicated over itself meaning that there is 2 layers overlapping with each other. This led me to remember that in some thread there was discussion about this in some form, but I don't seem to recall/find it anymore. If somebody remembers which thread it was, I would appreciate to know too. I'm pretty sure it was here on Qhimm at least.

EDIT: Good news, I now know how the geometry is read from the file and I'll write an update tomorrow regarding on it.
Title: Re: [FF8] world map and objects (world.fs)
Post by: Halfer on 2015-05-27 14:19:00
As mentioned in the first post, each section is size of 0x9000h with padding and it represents small are on map (see first post).

Sections start with offsets of blocks which are formed inside section. On each offset the block starts with 4 bytes long header.

Code: [Select]
struct
{

        u32 unknown (sections always starts with 4 bytes which affect nothing)
        u32 block_offset[amount of blocks]; // (first offset value/4-1) = amount of blocks
        header headerData ;
        face faceData[amount of faces];
        vertex verticeData[amount of vertices];
        shadows data[size differs]; // possible shadowing, on continents this sometimes match the amount of vertices.
        padding; // to next 0x9000h offset

} Section
       

Code: [Select]
struct
{
        u8 face_count; (mostly triangles or quads)
        u8 vertex_count;
        u8 shadow_count;  // shaders or something related to make shadows
        u8 unknown, never used??

}header, size = 4 bytes

Code: [Select]
struct
{

        u8 vertexIndices[6];
        u8 tex_coordX for v 0;
        u8 tex_coordY for v 0;
        u8 tex_coordX for v 1;
        u8 tex_coordY for v 1;
        u8 tex_coordX for v 2;
        u8 tex_coordY for v 2;
        u8 face type related [3]    // (texture page, type, etc...)
        s8 collision; //needs a bit research

} Face
size = 16 bytes

Code: [Select]
struct
{

        s16 coordinates[3]; // in form of XZY or YZX
        s16 unknown, unused (optional W dimension probably)

} Vertex

Code: [Select]
struct
{

        s16 or u16 unknown data [3]  // shadow possibly on XYZ (XZY) coordinate.
        u16 always zeros ??

} Shadows, under inspection still.

EDIT: texture related values may not be correct since they are not yet inspected properly.
Title: Re: [FF8] world map and objects (world.fs)
Post by: Maki on 2015-05-27 18:40:45
You're a God Halfer! :)

It works indeed.
(http://puu.sh/i2HEB/e12c0f4adc.jpg)

Balamb part is at 1d13000-1d198a0.
Towns and garden is made from triangles and is part of world map, so yes, towns are in wmx.obj.

Also, I'm like 70% sure rail.obj is for trains movement. Deleting file makes train blink sometimes and go crazy on world map. Like show for 1/2 second on center of sea.

@EDIT: I don't know if it's in fact vertex data, probably not, but one byte operates shadow. Probably that unknown data is about shadowing.
Title: Re: [FF8] world map and objects (world.fs)
Post by: Halfer on 2015-05-27 19:25:17
Yeah, that would probably make sense. In the water sections where there is only water blocks the "shadow" part is only 0x01 long which is 8 bytes right after vertex data. That is probably why I had trouble looking at that data because the sea doesn't really need shadowing. However it looks like in continents and other areas where "ground" is present the "shadow" data matches the amount of vertices which may indeed be for shadowing. I'll edit this to structures.

So towns are in wmx.obj? That's interesting. I wonder where the sections are set to match the exact position in world. I looked in FF7 wiki for the world info and in FF7 files it seemed to be separate file which handled the positioning of loaded sections. I'm starting to feel that they are coded in .exe, hopefully not.

EDIT: Also worth to mention that the vertices can be positioned outside it's block and section but the collision settings don't follow the boundaries, but are stitched to its block or section boundaries. In other words the section which is subject to world has priority on its own area. Here's an example:

(http://i.gyazo.com/1fc4c3ed05ccf7750859ca8e29a61acf.png)

Here the vertices are stretched on top of other section and while I can walk on the current area, I can't go over the area which is on top of water.

I'm not 100% if this is how it really works or if there are exceptions, but right now this seems logical.
Title: Re: [FF8] world map and objects (world.fs)
Post by: Maki on 2015-05-27 20:51:31
Ran a script and generated complete offset list for every world segment. There are total of 835 segments.
http://pastebin.com/WLRjSDFj

I can't help more today. Too tired.
Title: Re: [FF8] world map and objects (world.fs)
Post by: Halfer on 2015-05-27 21:02:35
Cool!

In the first post I stated that there was 768 segments in the world map and I based that in that 835 segments value since I found out that the last segments include great salt lake, the bridge and other areas. Seems like in reality it may be even less, maybe half of the size or something, since the sections don't need to be the same size unless they are preset somewhere.
Title: Re: [FF8] world map and objects (world.fs)
Post by: Aavock on 2015-05-28 02:30:22
Great breakthrough! Since you're both working on this, is it possible to use the hi-res worldmap textures and the UV coordinates from the PSX version?

Don't know if someone noticed before but PSX textures are better than PC as for the worldmap;  I've been looking in the world.fs file and there is 2 kinds of worldmap textures, one is low res and the other is 2 times bigger; the road doesn't even have borders... is the PC version uses the low res textures only ?

(http://image.noelshack.com/fichiers/2014/51/1419045508-worldmap.png)
Title: Re: [FF8] world map and objects (world.fs)
Post by: Halfer on 2015-05-28 08:30:37
Great breakthrough! Since you're both working on this, is it possible to use the hi-res worldmap textures and the UV coordinates from the PSX version?


This is an interesting question. I still haven't looked on the textures for much but what I've been taking to note is that it should be possible to give faces a bigger resolution texture, so hi-res textures may be possible in the future. As for PSX version, there isn't any reason for PC to be any different when speaking of textures since they are similar. However it seems like the UV calculation for road isn't correct thus making the white lines disappear. It can also be that PSX version uses some polished version of PC world map, one way to confirm this is to get the wmx.obj from PSX files and import it to PC version assuming that the file hierarchy is same.
Title: Re: [FF8] world map and objects (world.fs)
Post by: Maki on 2015-05-28 10:10:21
I could run a BinDiff on those two files, but I only have a PSN version of FF8, so I would have to "download" one.
Title: Re: [FF8] world map and objects (world.fs)
Post by: Halfer on 2015-05-28 10:21:57
Well I have PSX version on my parents house, but I can't get it for a while :P. But I hardly doubt that the files differ from each other, it may be instruction related problem with PSX GPU and computer GPU. But if that's not the case then the PC is indeed an older version of world map which is pretty hard to believe because it was released later than PSX.
Title: Re: [FF8] world map and objects (world.fs)
Post by: Maki on 2015-05-28 17:26:52
I found it little hard to browse FF8DISC1.img, as almost every extractor for PSOne files is dead. I had to manually extract it. My HexEditor just exploded, so had to work with buffer blockcopy. Thank you Visual Studio. <3

wmx.obj:
FFVIII Steam 2013: A594059C0D6D16F5DBEB8A82C4BE696FEB8A0E3D
FF8DISC1.IMG 0x108800 (size: 0x1D5B000): A594059C0D6D16F5DBEB8A82C4BE696FEB8A0E3D
FF8DISC1.IMG 0x19C8800 (size: 0x1D5B000): 500A79C34243AE19A49078BF36AB156C361F1043
FF8DISC1.IMG 0x388B800 (size: 0x333000): 950BCB3B5C8A3FE7F236C248044EC44653781329

This is a bit confusing. Offset 108800 is the same file that is present in Steam release. I'll run a BinDiff on 0x19C8800 and Steam one.

Update: There is a massive difference in 0x108800 and 0x19C8800 file. See the example: (http://puu.sh/i3PVf/351adf35d9.png)
 
I'll replace Steam wmx.obj with 0x19C8800 dump.

Well. It works. xD But I'm stuck inside some mountain and I can't move. Wait, I'll force ragnarok.

What the...

(http://puu.sh/i3R25/621e97982c.jpg)
Title: Re: [FF8] world map and objects (world.fs)
Post by: Halfer on 2015-05-28 18:26:27
haha :D, seems like there is 2 wmx.obj in the psx image file? That's odd.
Title: Re: [FF8] world map and objects (world.fs)
Post by: paul on 2015-05-29 21:56:15
Probably done like FF7 where this is grid map and optimized for cd loading map?
Title: Re: [FF8] world map and objects (world.fs)
Post by: Halfer on 2015-05-29 23:01:33
If by grid map you mean the placement of sections and how they are loaded to match the map, it isn't like FF7. In FF7 they are separate files and in FF8 there is only wmx.obj container which contains the sections and other files which contain textures. However there may be the placements also but I think they are coded in .exe for optimization.

Edit: Well, to be honest, I don't even know that yet. wmx.obj is 30MB big and I'm 99% sure that it contains only the geometry and texture calls. Other files mostly contains textures, but there're still lot to inspect. If these files contains the geometry placements, then it's packed very well compared to what FF7 respective data takes.

Edit 2: As I suspected, I found water geometry and texture calls from wmsetus.obj, but it's only size of a block when comparing to wmx.obj. It does contain much more data than the models itself, probably metadata mostly. I'll take closer look tomorrow for wmsetus.obj since I can't manage to get any information from wmx.obj for now because the texture calls don't make sense.
Title: Re: [FF8] world map and objects (world.fs)
Post by: Halfer on 2015-05-30 21:42:02
Okay, little analysis I did today. Here's what I documented:

Tested by nullifying these offsets from the beginning of wmsetus.obj and little inspection from the offset also.
wmsetus.obj:

0xB38 contains texture calls for rails and streets. There are 13 different pieces which have own texture. Each call is 4 bytes long and is somehow connected to their respective texture.

0xB70 contains data for the transition from world to fields. If offset indicator from the beginning of file is nullified you can't enter towns, except ragnarok cockpit (probably garden also).

0x18D0 has something to do with visibility of world models, especially ragnarok.

0x1CE0 has something to do with lunatic Pandora on tear’s point, nullifying offset indicator disables whole entry.

0x21B8 contains world map dialogues, probably all the texts that are shown in world map.

0x1AB04 contains color and lighting options for world map.


Title: Re: [FF8] world map and objects (world.fs)
Post by: Mcindus on 2015-05-30 22:37:58
Okay, little analysis I did today. Here's what I documented:

Tested by nullifying these offsets from the beginning of wmsetus.obj and little inspection from the offset also.
wmsetus.obj:

0xB38 contains texture calls for rails and streets. There are 13 different pieces which have own texture. Each call is 4 bytes long and is somehow connected to their respective texture.

0xB70 contains data for the transition from world to fields. If offset indicator from the beginning of file is nullified you can't enter towns, except ragnarok cockpit (probably garden also).

0x18D0 has something to do with visibility of world models, especially ragnarok.

0x1CE0 has something to do with lunatic Pandora on tear’s point, nullifying offset indicator disables whole entry.

0x21B8 contains world map dialogues, probably all the texts that are shown in world map.

0x1AB04 contains color and lighting options for world map.




I learned a lot by modding the textures using Tonberry for the world map... here are a few notes:

1:  UV's are incorrect on the PC worldmap - textures are not 'seamless' and strange artifacts happen over the textures where the walkmesh is either 'doubled' over itself, causing strange wrapping issues, or some kind of glitch is present.

2:  The Ocean, Roads, and a couple other 'animation' effects that exist in the world map all 'rotate' to the same cycle of sprites.  If you change a 'road' texture from intercepting the textures at ram with hash-codes, unless you use a code that 'overrides' the entire animation, the road blinks between animation sprites.  As for the rails/streets Halfer was talking about - the textures for them are all on one texture page.

3:  When you are using the English version and are on the ground, the world textures are hi-res -- if you're flying around the world in the Ragnarok, textures 'pop' back down to lower resolution, so this behavior might help you guys figure out what's going on.

Thanks for all your hard work!
Title: Re: [FF8] world map and objects (world.fs)
Post by: Halfer on 2015-05-30 22:46:04
I learned a lot by modding the textures using Tonberry for the world map... here are a few notes:

1:  UV's are incorrect on the PC worldmap - textures are not 'seamless' and strange artifacts happen over the textures where the walkmesh is either 'doubled' over itself, causing strange wrapping issues, or some kind of glitch is present.

2:  The Ocean, Roads, and a couple other 'animation' effects that exist in the world map all 'rotate' to the same cycle of sprites.  If you change a 'road' texture from intercepting the textures at ram with hash-codes, unless you use a code that 'overrides' the entire animation, the road blinks between animation sprites.  As for the rails/streets Halfer was talking about - the textures for them are all on one texture page.

3:  When you are using the English version and are on the ground, the world textures are hi-res -- if you're flying around the world in the Ragnarok, textures 'pop' back down to lower resolution, so this behavior might help you guys figure out what's going on.

Thanks for all your hard work!

Really? That's useful, thanks!
Couple questions though. You are saying that rails/streets are on one texture page. Do you know this for sure because tim viewer separates them as own images. Though if they are on same page then the hex values would make sense also.

And the hi-res textures. I don't seem to see any kind of difference in textures while on ragnarok or ground. Is there a setting which operates this behavior in ff8config?

Also, 0x15CC contains placement to world from field. Here's ending replicated in world map :P
(http://i.gyazo.com/c7b2b6ba1572979b57198cb35bd1c4e3.gif)

Title: Re: [FF8] world map and objects (world.fs)
Post by: Mcindus on 2015-05-31 02:03:00
Really? That's useful, thanks!
Couple questions though. You are saying that rails/streets are on one texture page. Do you know this for sure because tim viewer separates them as own images. Though if they are on same page then the hex values would make sense also.

And the hi-res textures. I don't seem to see any kind of difference in textures while on ragnarok or ground. Is there a setting which operates this behavior in ff8config?


It's hard to tell because the images are so pixelated anyway, but have you used tonberry?  If you want to run a test, you can install Tonberry (1.6 or later), then install my 'horizonpack' mod and disable the low res png's by removing wm_texpg_low1, _low2, and _low3 from the 'wm' textures folder.  Then hop on ragnarok in Balamb and then dismount.  You will see how the textures are being swapped back and forth.

When I get a debug dump from Tonberry, I get a single bitmap dump with all 16 road/textures in one image - and Tonberry -normally- doesn't dump single textures, but the entire page the texture came from (sometimes this is estimated, since all blocks come into Tonberry as 256px images, even though many texture pages are 128x256, 128x128, 256x128, etc.  These textures fit in a 256x128 bitmap, I believe.
Title: Re: [FF8] world map and objects (world.fs)
Post by: Maki on 2015-05-31 13:55:37
This is World map with wmsetus.obj with null'ied all textures. 0x1cbd8-0x86cf8 (First file which textures are in center of file, not at the ending)
(http://puu.sh/i79hL/5ce84ba4ef.jpg)

NOTE: wmsetxx.obj also is responsible for forest walking particle. I have some theory about that ragnarok texture change. Give me a sec, gonna test it.

As I thoughŁ
(http://puu.sh/i79sr/2e3f9d7482.jpg)

The textures in wmsetus(sp/it...).obj of world map are for Ragnarok. When you ride ragnarok, the game loads 1/2 sized textures from wmsetus.obj. When you land, the game loads texl.obj file.

TOWNS textures are gone too!
Title: Re: [FF8] world map and objects (world.fs)
Post by: Halfer on 2015-05-31 14:36:02
I see.... I can't replicate this for some reason. I nullified the textures from the file like this but I still see all the textures??:

Notice that the first texture is the water textures!
(http://i.gyazo.com/0556a5626da0495c2b447de69817fcf4.png)

(http://i.gyazo.com/0d9c33c026eb54f8dcc460b3a3d847c8.png)

(http://i.gyazo.com/b465726310cf8e1acfcf8d9ce1f48889.png)
Title: Re: [FF8] world map and objects (world.fs)
Post by: Maki on 2015-05-31 15:13:14
I see... Then steam re-release is different than the original one. My first texture is not a water and is at 117720. Hm... I have an idea. Let's see what happens if we mix files between releases.

Here: https://mega.co.nz/#!qRQVSAqC!Pkw2dyeOxnt235spPwdM34D2bhNRaMLybIGPvIfj-QA

BTW> I'm having fun with rail.obj :D

Aaaaand... Done. TRAAAAAINS!
http://wiki.qhimm.com/view/FF8/WorldMap_rail
Title: Re: [FF8] world map and objects (world.fs)
Post by: Halfer on 2015-05-31 15:32:33
Oh, I mean that is how timviewer shows my wmsetus.obj when I nullified the header of tim textures. I nullified 6 or 7 textures from the beginning and 117720 is the first texture offset for me also. I just tried to demonstrate that for me the textures didn't change for some reason while and while not in ragnarok. I wonder if i have some settings differently because I use dxwnd for windowed screen
Title: Re: [FF8] world map and objects (world.fs)
Post by: Halfer on 2015-05-31 15:38:37
Oh I think it is because I have software rendering on from ff8config. I'm using laptop with gt 840m and HD 4400 and for some reason with both graphic cards I can't use direct3d acceleration because it does this:

(http://i.gyazo.com/436bf9783397426bfdabffce749e5a55.png)

(http://i.gyazo.com/ec31a651434ace2208e502d9afa6e3e3.png)

Any solutions for this?
Title: Re: [FF8] world map and objects (world.fs)
Post by: Halfer on 2015-05-31 17:04:54
So, I tried to analyze a bit more wmx.obj and this is what I got.

Near the end are the sections that are placed to world map according to game progression:
(http://i.gyazo.com/013397864e4a1658858d9f4f0e9f65b2.png)
(http://i.gyazo.com/4f04e6b95d74169ee6cc46c0a056a638.png)(http://i.gyazo.com/0642dc09aba5e91f6173e672456e8ca9.png)
And also there is the whole area which is showed in place of esthar.

I also found that the crater done by Lunar Cry is there but for some reason the area is split in two and the parts are next to each other like this:
(http://i.gyazo.com/dbacef8939591c198bb1013049b711ae.png)(http://i.gyazo.com/51fdb11084b280e2c6a744b46cfd8321.png)

You can see that they are horizontally next to each other while in game they should be vertically on top of each other.

This led me to think that most likely there isn't a simple algorithm to read this file in the engine, but somehow it's described which sections corresponds which area. So it's either somewhere in wmset files or described in engine which would be unfortunate for us. I'll try to search more info for the placements.

EDIT: Can someone confirm if the wmset.obj contains japanese texts? There seems to be text differences between these files.
Title: Re: [FF8] world map and objects (world.fs)
Post by: Mcindus on 2015-06-03 04:16:06
Will there be any way to change the UV of the textures, since they aren't seamless in the PC version?  It looks like the UV is about 15% too large.
Title: Re: [FF8] world map and objects (world.fs)
Post by: Maki on 2015-06-03 05:37:56
Sure, but this can take some time, Halfer is making amazing progress, and after we know how texturing works I could convert data to UV mask only, fix it up and re-parse to FF8.

Edit: I tried to trick game to think I am ragnarok and can pass mountains, sea etc. but no luck. I'll try with coding sea and mountains as grass, thanks to Halfer I know it's possible to do, and where to search.

While coding ultimate hack found this:
(http://puu.sh/iaV19/71e2895288.jpg)
Probably GPU related, but it's fun. :D
Title: Re: [FF8] world map and objects (world.fs)
Post by: Halfer on 2015-06-04 13:24:32
Good news guys. I finally managed to crack the UV placements and no wonder it was so hard because I didn't realize they are picked from psx emulated framebuffer. Well I'm gonna do a little more research and write here at evening how it is read from the file. Get ready! :D
Title: Re: [FF8] world map and objects (world.fs)
Post by: Halfer on 2015-06-04 17:59:30
UPDATE!

wmsetusXX.obj contains world map textures. The object textures used for grounds are size of 128x128 pixels. Here's an example:

(http://i.gyazo.com/cba83486c113ed7dcf814002ba415f1e.png)

Here is 1 block which includes the first 4 textures from wmsetusXX.obj. There are 16 polygons and/or 32 triangles in one block. Each polygon has been made to show 4 32x32 texture area which are the small square areas in the image above.

As I described geometry before I wrote UV coordinates also while I hadn't researched it much yet. Turns out I was correct, but here's better struct with explanation after:

Code: [Select]
struct
{

        u8    face indices [6];         // Describes which indices are connected to each other. See first post, there is an example of face indices for blocks used in sea.
        u8    U vertex 1;
        u8    V vertex 1:
        u8    U vertex 2;
        u8    V vertex 2;
        u8    U vertex 3;
        u8    V vertex 3;
        u4    Texture page [4-bits];    // This changes the texture page according to value (0-F)
        u4    CLUT_ID [4-bits];          // The current palettes CLUT info. This needs to point for the texture you are using (first row 0-3, second row 4-7, and so on...).
        u8    Material type;             // Defines if the geometry is mountain, forest, sea or so on.... (00 seems to be forest)
        u8    Unknown;        // I'm not quite sure what this operates, but it seems to change texture pages too. Currently got only the sea textures while changing the value.
        u8    Unknown;       // Also unknown, however the last 4-bits seems to operate collision somehow

}Triangle

Now for the UV's. Each UV must be set according to triangle's face indices. UV values are from 00 to FF. They are set to indicate a rectangular area where the texture is present in current texture page. For example, if you have 128x128 texture and you want to fill a block with 16 polygon, you have to give each U and V coordinate a value in 2D axis where (0,0) indicates top left corner and (FF,FF) bottom left corner. The textures used in wmsetusXX.obj are 128x128 and it contains 16 32x32 textures so if you want to give a specific texture, you set each face indices UV's to the corresponding area by scaling UV's according to texture location on the page.

Here's an example how I did the image above (and great demonstration of my paint skills  :evil:):

(http://i.gyazo.com/1bc8fba600a7c86059da8dba48481337.png)

Blue dots are face indices. In this example the top left dots values are (U = 0x00, V = 0x00).
Middle dots values are (U = 0x7F, V = 0x7F).
Bottom right dots values are (U = 0xFF, V = 0xFF).

This indicates that the UV area contains the whole area. While there is now 4 polygons used for showing one texture, you can set the area inside 1 polygon also. It's also possible to enlarge it, but it needs more "waypoints" (dots) to determine in between the desired area and making it manually is frustrating.

So, this means that in theory it is possible to use large textures in world, but since the memory space is limited it may be difficult. I'll look on this later.


If you have questions please ask. I'm pretty tired and noticing my own mistakes and the "shortness" of this post makes me think I may have forgotten to explain something so ask if there's something you don't understand or needs a little explanation.
Title: Re: [FF8] world map and objects (world.fs)
Post by: Kaldarasha on 2015-06-04 18:26:13
But the tile system has also information for low res textures. How does the game handle these? Are they also stored in wmsetusXX.obj?
Title: Re: [FF8] world map and objects (world.fs)
Post by: Halfer on 2015-06-04 18:35:17
Actually these are the low-res textures. The higher resolution textures are stored in texl.obj and as far as I have been informed they are used when not in ragnarok (or any other vehicle?). I'm using Aali's drivers to run the game in direct3d mode, but for some reason my game isn't using these higher resolution textures like it doesn't if running with software rendering on.

Oh and here's a picture with all the texture pages showcased on the 2 top rows:
(http://i.gyazo.com/691668b0d4b00675166799895f329b21.png)
Title: Re: [FF8] world map and objects (world.fs)
Post by: Maki on 2015-06-04 18:50:25
Halfer, excelent work. Will try this on my own.
Oh, by the way- try to completely delete texl.obj file and see what would happen.
Title: Re: [FF8] world map and objects (world.fs)
Post by: Kaldarasha on 2015-06-04 18:52:15
The Steam version shows the high res textures but if you open and close the menu the game falls back to the low res ones.

Halfer, excelent work. Will try this on my own.
Oh, by the way- try to completely delete texl.obj file and see what would happen.

Maybe re-name texl.obj to wmsetusXX.obj do would work to force the game to use the high res textures.  ???
Title: Re: [FF8] world map and objects (world.fs)
Post by: Halfer on 2015-06-04 18:55:29
Tried it before, no changes :( . That's interesting, I wonder if it was accidentally left out in the old release.
Title: Re: [FF8] world map and objects (world.fs)
Post by: Kaldarasha on 2015-06-04 19:21:01
I guess some one with the steam version here could send you these files. I'm currently not on my notebook so I can't.

By the way doesn't Aali's driver run the game trough OpenGL?
Title: Re: [FF8] world map and objects (world.fs)
Post by: Halfer on 2015-06-04 19:24:41
Yeah it does, that's why I'm guessing that this problem occurs on my PC. I posted pictures couple posts back where you can see what happens when i run the game in direct3d. Actually now that I remember it does this only when I set it on windowed mode. Haven't tried it in full screen.
Title: Re: [FF8] world map and objects (world.fs)
Post by: Aali on 2015-06-05 19:29:01
The high-res textures are only used in the Steam version. The original PC version will never use them.
Title: Re: [FF8] world map and objects (world.fs)
Post by: Mcindus on 2015-06-06 19:59:20
The high-res textures are only used in the Steam version. The original PC version will never use them.

Were they just never implemented into the PC game, even though the files exist?

Sounds like they had to rush it, considering the UV/walkmesh in the PC version is also incorrect.
Title: Re: [FF8] world map and objects (world.fs)
Post by: Aali on 2015-06-07 01:26:46
It seems more likely the high-res textures were cut out due to performance issues. The worldmap module in FF8 is extremely inefficient and I doubt it ran very well even on mid-range hardware at the time it was released.
Title: Re: [FF8] world map and objects (world.fs)
Post by: Maki on 2015-06-09 18:40:45
I added a nice feature to my ultimate mem/asm hack, this loads map portions by offset to specific memory blocks, to make the map tile by this specific uploaded chunk from wmx.obj (real-time).

Take a look at this video: https://www.youtube.com/watch?v=2ZKFb4x5bl8&feature=youtu.be

It really looks like it goes by role like from top left to bottom right with horizontal lines. Just like you wrote Halfer. :)
I'm going to do a test by changing camera root position*, leaving Squall safe on sea, so he can't catch exception, and auto-rotating camera tangent X for something like "showcase" of WM chunks. :D

*See more cool camera features available to hack at: http://wiki.qhimm.com/view/FF8/Engine/WorldMapCamera
Title: Re: [FF8] world map and objects (world.fs)
Post by: kaspar01 on 2015-06-09 20:16:11
The zoom looks cool!

Anyway I'm really happy to see you're making big steps here!

Are you planning to create an exporter for the world map mesh like the one you did for battle stage?

Title: Re: [FF8] world map and objects (world.fs)
Post by: Maki on 2015-06-09 20:36:11
All thanks goes to Halfer. He is a master here. ;)

WMX is structured the other way than battle stages. Stitching 837 chunks can be painful to code. I ought to think about completely fresh solution to perform conversion.
Title: Re: [FF8] world map and objects (world.fs)
Post by: Halfer on 2015-06-09 22:21:54
Great! :D Happy to be helpful here!

Looking forward to your tools, seems like you have put much effort for these tools! Keep it up :D.

I think that exporter could be a good thing but leaving it as an exporter/importer wouldn't make much sense. The problem would be the way FF8 engine handles the file, like Aali said earlier. It needs a bit more studying how to implement more complex geometries to FF8 without crashing the game/slowing it down significantly. Speaking of geometries, textures are the underlying problem right now. Maybe for starters the exporter/importer could work as a tester for engine capabilities. Maybe in the future the program could be a proper editor with 3d view.

By the way, MaKiPL, have you encountered to crash on world map with camera zoom (under 300) when flying ragnarok on top of esthar and flying back to bridge? For me it seems to mess up the textures from the bridge and after that crashes.
Title: Re: [FF8] world map and objects (world.fs)
Post by: Maki on 2015-06-11 16:16:05
Halfer, could you upload a flat walkable ground you posted in first screenshot here? Working with smaller data is always better. Thanks.

And ya. Just tested, and setting zoom under 300 and flew from bridge to Esthar. Crashed. Weird.

I attached Visual Studio debugger to FF8. At first time the game gone crazy, the bridge just dissapeared (no rail texture, just black) and I just teleported to Balamb (Sic!). However, my party was empty (no visible characters), but still I managed to get out of town and just appeared as Squall in front of Balamb on world map. My ragnarok of course was flying in sky in last position... Weird.
Second time I finally ran to exception.

Code: [Select]
First-chance exception at 0x00545F36 in FF8_EN.exe: 0xC0000005: Access violation reading location 0x36382834.
I now have memory locations I can investigate.

0x00545F36 is MOV opcode.
NOP'ing freezes game.
Title: Re: [FF8] world map and objects (world.fs)
Post by: Halfer on 2015-06-11 17:46:07
http://pastebin.com/GZVNkcL5 (http://pastebin.com/GZVNkcL5)

Here you go, one block, remember that there are 16 blocks in one section. I guess this was what you meant? The texture section (last 4 bytes of each triangle) has been changed to "88 88 88 88", makes it look like in the very first post and is landable and walkable. Hopefully you can insert the values as in text. At least HxD editor can do this.

Yeah the crash was pretty odd. It seemed like the memory couldn't keep up with the data or something like that.
Title: Re: [FF8] world map and objects (world.fs)
Post by: Maki on 2015-06-17 11:58:00
I finally made it! I tricked engine to think I am ragnarok, so I can walk on everything I want (sea, mountains). I've been hunting this for days!

Here, in FF8_EN.EXE (Steam) ASM opcode:
FF8_EN.exe+13E370 - mov ebx,[FF8_EN.exe+1C409E0]   (0x0053E370)

Halfer, your memory theory is correct. Game just can't keep up so much memory (because the variables are static) and overwrites some core data that is in fact pointer and orders engine to read memory at ~0x7... (but it's unallocated, so it's null, and access violation exception is fired)

Also this:
0x020409E0 in memory tracks your current vehicle status. Here's the list:
#HEX!
Code: [Select]
00 - Squall on feet
20 - invisible low fuel car
30 - Balamb Garden
31 - *Crash*
32 - Ragnarok
Some variables makes Squall go maximally undermap, that's because vehicle doesn't exist on worldmap. For example if you rent a car, then some variable instead of digging you up to the ground just warps you to this car. Unfortunately, I lost "Blue sky car" from Balamb variable. Also warping should work on whole map (Vehicle is warping to your location). You can place Balamb next to ragnarok in seconds.
Title: Re: [FF8] world map and objects (world.fs)
Post by: kaspar01 on 2015-06-17 12:15:28
Code: [Select]
00 - Squall on feet
20 - invisible low fuel car
30 - Balamb Garden
31 - *Crash*
32 - Ragnarok

Good job maki!

The list isn't supposed to contain something for chocobo as well, isn't? (maybe 31 is that or there is something missing)

I wonder what happen when you're not driving the vehicles .. (happen whith garden car , boat during dollet mission and trains)..

Actually driving the trains or the assault boat would be damn cool XD
Title: Re: [FF8] world map and objects (world.fs)
Post by: Maki on 2015-06-17 12:46:19
Ye, it warps you to actually available on map cars and vehicles. There must be variables for chocobo, cars, trains, even boat or "vessel" as they call it in-game. I'll try to find them, gather in one place, make a documentation and save. :)

@UPDATE - Aaaand, You we're right! 31 is for chocobo! Also 10 is for train, but it's completely uncontrollable and problematic.
22 for blue sky car
I could memory hack to swap models. (Just like I did with world map chunks)
Saves does not contain defiinition for Chocobo. Unfortunately.


Today's video:
https://youtu.be/dosLK8DiuU8

I'm driving train and flying with Lunatic Pandora. :D
Title: Re: [FF8] world map and objects (world.fs)
Post by: Halfer on 2015-06-18 08:53:42
Great job! :D

That blue sky car is probably the model for the thing that is blocking access to towns in disc 4 :P
Title: Re: [FF8] world map and objects (world.fs)
Post by: Blue on 2015-07-06 11:54:44
I created a tool that converts the world map geometry into known Wavefront .obj format. It's available for download in the Tools section (http://forums.qhimm.com/index.php?topic=16230).
Title: Re: [FF8] world map and objects (world.fs)
Post by: Maki on 2015-07-06 22:12:37
Wow! First post and releases full tool for world map conversion. You are good. :3
Thank you!
Title: Re: [FF8] world map and objects (world.fs)
Post by: Halfer on 2015-07-08 15:50:07
I've done some progress with my texture exporting using Blue's program as a base:

(http://i.gyazo.com/188b4f7d8a2356369f9f89a5ac1ed1a0.jpg)

Texture coordinates should be correct right now.
I still need to figure out the process of changing texture page and CLUT info on the fly while exporting. It may or may not take a while, but after that I'll release an updated exporter ;).

EDIT: Some eye candy without CLUT info:

(http://i.gyazo.com/c9d3d172bc94ed1327566bfc2752c9ec.png)
(http://i.gyazo.com/4eec5ddaaa9eb775b34e2324fe232d0d.jpg)

The program now automatically switches texture pages, however water textures don't still work because they are presented a bit differently due to their animation data, but i'm gonna add it too (without animation of course). aaand they are done.

And of course the next thing is to take CLUT info into account. I'm still not entirely sure how to make this happen :(. The image should be somehow exported as 128x128 whereas every tile has own  CLUT already corrected.
Title: Re: [FF8] world map and objects (world.fs)
Post by: kaspar01 on 2015-07-09 13:30:45
And off course the next thing is to take CLUT info into account. I'm still not entirely sure how to make this happen :(. The image should be somehow exported as 128x128 whereas every tile has own  CLUT already corrected.

Don't worry for that.. I'm not expert about how to automatically do it but get the correct texture for these maps is pretty simple (it just take some time) because the textures are always in a 4x4 grid for the hi-res in textL file or 4x8 grid (for the low res version.. in wmsetus? not sure)

Beside that luckly  the "right color clut position" always follow the same order for these textures... I mean the first clut is always the top-left piec of the grid and so on..

You can see some sample of the "already correct" textures I made some time ago on this ol post of mine:

http://forums.qhimm.com/index.php?topic=13787.msg193708#msg193708

If you need them for testing purpose I can send you the final textures already processed I made  :)

P.S.

this new version of the tool is already avaiable somewhere for download?

I'd like to try it :)
Title: Re: [FF8] world map and objects (world.fs)
Post by: Halfer on 2015-07-09 14:38:02
Yeah, I'm aware of the positions, thanks to timviewer, but I'm pretty sure that it is possible to do it automatically, though with some work. Well I'll have to look on that later on.

Yeah I'd like to test the processed textures if you could send them somehow to me. I'll release the program probably today after making sure that I haven't made any UV calculation errors so the corrected textures would be nice :).
Title: Re: [FF8] world map and objects (world.fs)
Post by: kaspar01 on 2015-07-09 15:39:44
Yeah, I'm aware of the positions, thanks to timviewer, but I'm pretty sure that it is possible to do it automatically, though with some work. Well I'll have to look on that later on.
It's possible for sure but I have no idea about how to write a tool that automatically process tim files..at the time I created a simple script/action sequence in photoshop to make the work faster.

Yeah I'd like to test the processed textures if you could send them somehow to me. I'll release the program probably today after making sure that I haven't made any UV calculation errors so the corrected textures would be nice :).
I'll upload them somewhere in a zip and send you a link as soon as I get back home :)

Title: Re: [FF8] world map and objects (world.fs)
Post by: Halfer on 2015-07-09 21:09:36
Seems like this will take at least a day still, it is mostly working correctly like this:
(http://i.gyazo.com/f19936a03f92d912f3fd30fbef30209c.jpg)

I still do have some problems with galbadia area and some textures for some reason. Have to think about them tomorrow

Title: Re: [FF8] world map and objects (world.fs)
Post by: kaspar01 on 2015-07-09 21:36:04
Glad to see this :)

At the moment I just wonder about Balamb Garden Mesh.. I can see only the "hole".. and there is a copy of the same zone but whith the garden in the right place...
so the map has some "interchangable" chunks.. interesting! :D

Can't wait to see the tool complete to place every "alternative" mesh to the right place just to see what changes from one to the other  :P
Title: Re: [FF8] world map and objects (world.fs)
Post by: Halfer on 2015-07-09 21:39:19
Well you can view them later with textures once I get this fixed :P they are found under the world map.

Can't wait to see the tool complete to place every "alternative" mesh to the right place just to see what changes from one to the other  :P

Haven't really given a thought about this yet but I can see it happening with future updates ;)
Title: Re: [FF8] world map and objects (world.fs)
Post by: kaspar01 on 2015-07-09 22:15:56
Haven't really given a thought about this yet but I can see it happening with future updates ;)

Oh.. actually in my head I was going to do it "manually" on 3ds max.. but  of course a function in the tool would be more handy! ;D


Since I still see the untextured version right now I can just guess anyway..

Among these interchangeable zones I can recognize:

-balamb Garden  (before/after lift-off)
-galbadia Garden (before/after lift-off)
-desert prison (inground/exposed)
-missile base (before/after destruction)
-trabia garden (??)
-start of F.H. Bridge (??)
Title: Re: [FF8] world map and objects (world.fs)
Post by: falkTX on 2015-07-10 12:13:45
I haven't looked at those files, but what about esthar before and after entering the area?
Title: Re: [FF8] world map and objects (world.fs)
Post by: Maki on 2015-07-10 19:32:57
So, in future we would do a complete model replacement. Unfortunately, we'd still care about limits like 0x9000h for world map segment or uint16 count of tris. I'll complete wmset.obj tool, and start with more advanced memory hack to do realtime geometry edit.
Title: Re: [FF8] world map and objects (world.fs)
Post by: Halfer on 2015-07-10 19:43:08
Well actually it is uint8 for triangles, right? But, I don't know,  that's 256 vertices for 1 block which is pretty small area by itself, but I can see what you mean.

It would be a dream come true if we get to the point where we could edit those restraints like memory caps and so on. Thinking about the PSX memory limitations which are present in the PC version also. If those can be bypassed then the true graphic overhaul can start :P.

I haven't looked at those files, but what about esthar before and after entering the area?

Yes the whole esthar continent is a part of interchangeable sections. That Trabia Garden which kaspar01 mentioned is probably a leftover from early builds replaced with the same section. It probably contained the model and textures before the missiles, but that's never seen in the game so dumping that out seems to be logical answer.
Title: Re: [FF8] world map and objects (world.fs)
Post by: kaspar01 on 2015-07-12 11:54:37
What about add a check box to make the .mtl file optional (after first export it become useless I guess and I think that choose 2 times for destination is not a good thing)?

Moreover I get an "invalid texture index" error whith some  mesh (not sure but it seems to happen when I export map not starting from 1).

Edit :

my theory about when the texture error happen was wrong.. here some example of export that gave me error:

268-268
328-328
150-151


but for example i don't get error whith 275-276 ... this is weird..
Title: Re: [FF8] world map and objects (world.fs)
Post by: Halfer on 2015-07-12 12:02:44
Yeah I'll fix that .mtl file to not be mandatory to export everytime.

About invalid texture index. That's odd, I maybe encountered it once or twice when I first made the code but never after that. Do you know specific sections which launches this bug? Would help me to fix is ASAP. Actually if it's not severe issue I'll fix it with next update since the optimization should fix those kind of problems and also would make much less lag while editing the mesh.
Title: Re: [FF8] world map and objects (world.fs)
Post by: kaspar01 on 2015-07-12 13:40:02
Yeah I'll fix that .mtl file to not be mandatory to export everytime.

About invalid texture index. That's odd, I maybe encountered it once or twice when I first made the code but never after that. Do you know specific sections which launches this bug? Would help me to fix is ASAP. Actually if it's not severe issue I'll fix it with next update since the optimization should fix those kind of problems and also would make much less lag while editing the mesh.

I edited my previous post and put a little list in it.

Most of the sections I tried gave me the error.

Anyway I manually fixed transparency and fixed garden just to make see this:

(http://s4.postimg.org/d8267we4t/Full_HD_Balamb_Isle.jpg)

defintely worth it  8)
Title: Re: [FF8] world map and objects (world.fs)
Post by: Halfer on 2015-07-12 13:59:25
That's really odd, I don't seem to get the same errors for some reason:
(http://i.gyazo.com/f3aea9ea3ae71f44c74ab1f6e0aa6695.png)

Well anyway, all the texture related problems should be corrected after next update except transparency which has to always be done manually. I'll release the updated version once I get normals working, textures corrected and the segment swapping done, may take a while  :cry:
Title: Re: [FF8] world map and objects (world.fs)
Post by: kaspar01 on 2015-07-12 15:45:29
That's really odd, I don't seem to get the same errors for some reason:
Hum... maybe it's a 3ds max problem?
I could try to send you one of my files or you could send me one of yours to verify.. or both :D

Update: I try to export segment 268 againg in a new file and now it works..tried to import the old one and still get the error.. maybe an error during export?

Update 2 : I think I understood when the error happens! The first file I export is fine but every following file gives me the error (apparently I have to close and open again the tool to avoid the problem)

Well anyway, all the texture related problems should be corrected after next update except transparency which has to always be done manually. I'll release the updated version once I get normals working, textures corrected and the segment swapping done, may take a while  :cry:
If I may help you somehow let me know :)
Title: Re: [FF8] world map and objects (world.fs)
Post by: Halfer on 2015-07-14 12:32:13
Oh btw, one thing about high resolution textures occurred to me sometime ago.

If we assume that the engine can handle larger size texture pages like 512x512, since it can manage 256x256 over 128x128, it could be possible to replace these files in wmsetXX.obj (and why not in texl.obj but it's not used in old release). The only thing I can see limiting this is that the headers are pointing to specific location in the file, so by changing those it could be possible to replace the old ones in the file. That will not work on texl.obj however because it doesn't have any headers.

Just a thought but should be worth a try, unless the wmsetXX.obj needs to be certain size.
Title: Re: [FF8] world map and objects (world.fs)
Post by: kaspar01 on 2015-07-14 13:05:05
Oh btw, one thing about high resolution textures occurred to me sometime ago.

If we assume that the engine can handle larger size texture pages like 512x512, since it can manage 256x256 over 128x128, it could be possible to replace these files in wmsetXX.obj (and why not in texl.obj but it's not used in old release). The only thing I can see limiting this is that the headers are pointing to specific location in the file, so by changing those it could be possible to replace the old ones in the file. That will not work on texl.obj however because it doesn't have any headers.

Just a thought but should be worth a try, unless the wmsetXX.obj needs to be certain size.

I'm quite sure that pc version of the game had an option for using hi-res texture (I guess it referred to textl usage since I remember it affected world map only.. )

I guess it could be possibile to exploit that option somehow..

Edit:

I also wonder.. Maki has been able to switch a piece of the world map whith another existing one..
Maybe modify geometry of existing piece making them more detailed would result in a crash but what about "add new pieces" after the last one and try to swhitch them? I mean.. there is a size limit? it applies to the whole file or to the single chunks?we can add new chunks?
Title: Re: [FF8] world map and objects (world.fs)
Post by: Halfer on 2015-07-14 18:53:36
Well I've been thinking about the subdividing of the existing polygons but there is that performance downside. However once I get the program working I can do some testing with normals affecting the performance. If somehow the game is utilizing gourard shading, then we can maybe limit the usage of normals so it wouldn't look worse but give some performance boost. Also it would save space from already limited segments which could be filled with subdivided blocks for better detail. And yes, Maki's program looked very nice. I'm wondering if you are (MakiPL) focused in the engine side of the game because of the application(s)? If you are then I think it's okay that we don't overlap too much, but of course, I wouldn't mind that.

Speaking of the textures itself. I don't think that texl.obj file leads us to anywhere with higher quality textures since it doesn't contain any header except for the textures itself. That would mean the headers have been written somewhere (maybe to .exe) and that makes it impossible to replace the textures with bigger ones. However wmsetXX.obj contains headers for these textures so utilizing those the texture size can maybe be increased.

While it is pretty interesting and definitely worth of a try, I don't plan to do it myself at least yet. The converter itself gives me some hard times since I've low amount of experience with coding still. However I won't let it stop me from doing the program because how else would I ever learn to code if I never do it. I don't want to make you think that there is a chance that I would drop out of this because I do study information and communication technologies and in fall I attend software engineering studies with main focus of game applications.
Title: Re: [FF8] world map and objects (world.fs)
Post by: Mcindus on 2015-07-14 22:33:14
Hey guys!  I'm not sure if I'm following the issues with the road/rail textures properly, but this is the image that Tonberry dumps in the debug folder:
(http://www.13tomidnight.com/FF8/Debugstuff/roads_tb.bmp)

The image is always 256x256 in tonberry, because it takes a snapshot of the vram, but it looks like the TIM is actually 128x64 in a set of anywhere from 13-18 palettes - I haven't found the .TIM in tim viewer.
Title: Re: [FF8] world map and objects (world.fs)
Post by: kaspar01 on 2015-07-14 22:51:49
Hey guys!  I'm not sure if I'm following the issues with the road/rail textures properly, but this is the image that Tonberry dumps in the debug folder:
(http://www.13tomidnight.com/FF8/Debugstuff/roads_tb.bmp)

The image is always 256x256 in tonberry, because it takes a snapshot of the vram, but it looks like the TIM is actually 128x64 in a set of anywhere from 13-18 palettes - I haven't found the .TIM in tim viewer.

Thanks.

This confirm the texture I built is almost 90% correct.

and It's interesting to see the top right corner.. If I'm right those texture should be used for ultimecia castle portals.. that's interesting...

Actually I wonder where those models are stored..
Title: Re: [FF8] world map and objects (world.fs)
Post by: Halfer on 2015-07-14 23:06:18
@Mcindus
That is really interesting! I haven't done anything with tonberry, but I would like to know how to take those VRAM snaps. Would help a lot to understand how the game utilizes the memory. Also is it possible to take snap of the whole VRAM with it because at least the psx vram is size of 1MB and it is presented as 1024x512 area. This is probably how it's in the PC also.

@kaspar01
Those castle portals are stored in wmsetXX.obj, I did a lot of studies on that before and I'm 90% sure that it contains the portals because it does contain lunatic pandora and other models used for world map.
Title: Re: [FF8] world map and objects (world.fs)
Post by: kaspar01 on 2015-07-14 23:26:32

@kaspar01
Those castle portals are stored in wmsetXX.obj, I did a lot of studies on that before and I'm 90% sure that it contains the portals because it does contain lunatic pandora and other models used for world map.

Yes I was thinking the same thing.. MakiPL was working on that side and found some models if I'm right.

A tool that extract them would be cool also.. I'm not sure but I think that no one wrote it yet.
Title: Re: [FF8] world map and objects (world.fs)
Post by: Halfer on 2015-07-15 00:04:18
I think Maki did that already but didn't make any release yet. I guess he's keeping us waiting on purpose so he can surprise us big time ;). Well anyway, the models have been documented by Vehek so it's only matter of programming.
Title: Re: [FF8] world map and objects (world.fs)
Post by: Maki on 2015-07-16 13:58:32
Hi! I'm back. :D

Quote
I think Maki did that already but didn't make any release yet
Yes, I did, but the software works highly code side, so that'd require changing offsets and etc in code and recompiling. I'm currently working on AIO toolset, so expect fully working wmsetxx.obj converter next week.

About playing with memory...
FF8 memory is static. The game uses hardcoded addresses to work with objects, so if you add more detailed object to the game, you will probably overwrite other stuff or headers or any sensitive variables. Let me explain this further:

World map segments are 0x9000h stored in memory. There are 5 segments (which I found, but I'm sure I missed at least one). So... the smallest world map segment is the one with water. One water only segment is ~0x2dbb leaving 0x6244 bytes of free space. This space CAN be modified, but 0x9000h cannot be exceed. If you exceed the 0x9000 limit and write like x12300 memory to segment 1 address start, then you'd probably crash the game, or just touch the second segment. Changing this without source would be extremely hard.
However!
You can allocate memory at the end of process, change by assembler the engine to read bigger variables, and call segments from new hardcoded address, which is the one you allocated.

Wmx.obj file can be normally edited, and it's size can be freely changed. Why? The engine reads only specific segments. If you add new, custom built segment at the end of file, and keep 0x9000h size of it with padding if needed, then nothing would happen. I didn't find a opcode, the engine calls for segment load with specific index, but I'll see if the knowledge of where specific segments are will help me with this. If I found it, then I could recall it to use custom built segments from the end of file. FF8 engine reads files by stream, so they can be real-time edited, repacked and etc. For example, you can extract and edit wmx.obj file, even when on worldmap in-game, then re import file, save it, get back to already running game and moving forward, so it can read next segment at specific position. It's the same as any other stream works. It makes the engine don't load whole file, but only it's part. This way, a programmer can get for example a 2MB data, that is stored in the semi center of 2GB file in matter of milliseconds (with current CPUs).

For wmsetus.obj file, it's more complicated. World map objects like cars, garden, ragnarok does not contain padding, and are loaded and allocated just as the game needs. That means, if you wish to replace ragnarok with much more data, then real-time memory editing would affect other vehicles data (however, this didn't appear in my tests, even when I wrote memory that overwritten other objects data loaded in memory, they still were working. I didn't know which one I was overwriting, so probably that's why I didn't notice the change).
If memory editing in this case is not recommended, then normal wmsetus.obj hex'edit should work. The engine would just push the other vehicle data for your new object. (Or the engine loads all vehicles, and keeps them in the memory? I don't know, need to test it someday).

Still, if the wmsetus declared size will be exceed (which I don't know at the moment how much does it takes space in the memory) then you still be able to overwrite some other data.

Another example to clarify this:
Just after world map segments in memory, a core Squall data is stored. This means, if the last segment in memory would be bigger than 0x9000h, then the game would read for example one triangle's vertex as Squall position, instead of this triangle (as we intended to use it). Still, modifying engine to read more than 9000h data and calling it from the allocated by user data at the end of file is the safest and best way. Still, due to x86-86 (32 bits) compilation, you'd need to make sure you doesn't allocate more than 4GB memory in total.

Why does changing camera logical zoom and flying from esthar to bridge would crash the game?
This is the glitch that is made because of "size exceeding" and static memory "bad sides".

This is more detailed exception with memory addresses and related FF8.EXE opcode (thanks to debugger):
Code: [Select]
First-chance exception at 0x00545F36 in FF8_EN.exe: 0xC0000005: Access violation reading location 0x36382834.
This means, that opcode at 0x00545F36 (FF8_EN.exe+145F36 - mov ax,[edi]) tried to read memory at 36382834, but this memory address didn't exist.
Why this happened? The memory exceed it's size limit (probably engine tried to load another texture, but in fact loading this made the engine to overwrite some sensitive parts of memory and in fact ordered the engine to read 36382834 address (which could be just part of texture bytes, or for example changed header, to read for example 255 vertex instead of 10 and completely make the game go insane).

Battle stages - I didn't test memory addresses and sizes yet. Can't write much here.


@Edit:
Just a little back to memory allocation. Making the game allocate memory ADD's null byte at the end of process. This way you can store everything you want, even assembly (sic!). You can still order a process to call specific instruction. You can even add your own code to the game.
Title: Re: [FF8] world map and objects (world.fs)
Post by: Halfer on 2015-07-16 17:02:12
That's a lot of information! You've been rocking it :D

Why does changing camera logical zoom and flying from esthar to bridge would crash the game?
This is the glitch that is made because of "size exceeding" and static memory "bad sides".

This is more detailed exception with memory addresses and related FF8.EXE opcode (thanks to debugger):
Code: [Select]
First-chance exception at 0x00545F36 in FF8_EN.exe: 0xC0000005: Access violation reading location 0x36382834.

This means, that opcode at 0x00545F36 (FF8_EN.exe+145F36 - mov ax,[edi]) tried to read memory at 36382834, but this memory address didn't exist.
Why this happened? The memory exceed it's size limit (probably engine tried to load another texture, but in fact loading this made the engine to overwrite some sensitive parts of memory and in fact ordered the engine to read 36382834 address (which could be just part of texture bytes, or for example changed header, to read for example 255 vertex instead of 10 and completely make the game go insane).

I understand that the engine overwrites wrong memory addresses if the limit is exceeded, but I don't understand why changing the logical zoom of camera cause this kind of crash? As seen the engine always loads fixed amount of segments depending on the camera rotation so shouldn't all the textures and geometry be pushed on the memory also? So does that mean that it is VRAM related issue?
Title: Re: [FF8] world map and objects (world.fs)
Post by: Maki on 2015-07-16 17:34:03
Yes, that's weird. It's logical zoom, not the real distance-to camera, so it's indeed weird a bit. I'll take a closer look what's happening there, but my theory is the game additionally stores info "what's really visible". I mean when you fly ragnarok, the world map is loaded by blocks. What if the game stores currently visible data, and changing camera zoom makes the game logic to redraw additional blocks and overflow memory?

I'm going to test this now.

The block that is causing the crash is part of wmsetus.obj file.

@UPDATE: 0x00545F36 MOV appears to access big amount of data. Ticked every milisecond. Works with addresses in range:
01EB8D4C-01EBA758
@UPDATE2: Accessed addresses are 4 byte long.
@UPDATE3: Altering accessed variables makes extremely weird things. For example spams with message "This shouldn't go now...", warping to Balamb, hidden emergency capsule (after coming from space), calling battle with UFO encounter... Weird.
@UPDATE4: Hello human... What a lovely tune.
@UPDATE5: "There is a stone pillar. If you look closely, there's something written on it"
Title: Re: [FF8] world map and objects (world.fs)
Post by: kaspar01 on 2015-07-18 23:01:06
@UPDATE3: Altering accessed variables makes extremely weird things. For example spams with message "This shouldn't go now...", warping to Balamb, hidden emergency capsule (after coming from space), calling battle with UFO encounter... Weird.
@UPDATE4: Hello human... What a lovely tune.
@UPDATE5: "There is a stone pillar. If you look closely, there's something written on it"[/i]

Those are supposed to be side quest "events" which can be triggered while on the map (Obel Lake side quest).

http://finalfantasy.wikia.com/wiki/Obel_Lake

I guess you found a way to manually trigger some "quest stuff"..
Title: Re: [FF8] world map and objects (world.fs)
Post by: Halfer on 2015-07-20 11:38:12
While looking in the memory I found out the addresses used for the sections. I'm sure that Maki found them already but something interesting I noticed in that is that the engine adds or subtracts 2 from the value indicated in the wmx.obj for example if U1 = 40 and V1 = 60 the engine does some calculations on them and in the memory they are U1 = 42, V1 = 62 .

Sometimes it subtracts instead of adding. That is the reason why the UV's are misplaced in the PC version and the road does not show the white lines.
Title: Re: [FF8] world map and objects (world.fs)
Post by: kaspar01 on 2015-07-20 11:49:58
That is the reason why the UV's are misplaced in the PC version and the road does not show the white lines.

Ok... that's why they're always flat grey colored..

Hey guys!  I'm not sure if I'm following the issues with the road/rail textures properly, but this is the image that Tonberry dumps in the debug folder:
(http://www.13tomidnight.com/FF8/Debugstuff/roads_tb.bmp)

The image is always 256x256 in tonberry, because it takes a snapshot of the vram, but it looks like the TIM is actually 128x64 in a set of anywhere from 13-18 palettes - I haven't found the .TIM in tim viewer.

Yesterday I edited my texture (added portals piece) and it perfectly match but I Wonder about the bottom of your dumped one.. what's all that noise in the lower half of the texture?

It's supposed to be empty or not? why does it looks that way?
I'm not sure but..any chance that is an animated material texture?
I should check but if I remember right that kind of noise on the image happen on "face texture" where it's supposed to change (eyes close - open) and maybe that part of the texture is for waves on the sea or something like that..
Title: Re: [FF8] world map and objects (world.fs)
Post by: Maki on 2015-07-20 14:24:35
The noise usually is the data, that cannot be image, but is forced to display as image. In that case, that looks, that this noise place probably may operate animation, like how to handle it and etc.
Title: Re: [FF8] world map and objects (world.fs)
Post by: Mcindus on 2015-07-20 16:43:58
Yesterday I edited my texture (added portals piece) and it perfectly match but I Wonder about the bottom of your dumped one.. what's all that noise in the lower half of the texture?

It's supposed to be empty or not? why does it looks that way?
I'm not sure but..any chance that is an animated material texture?
I should check but if I remember right that kind of noise on the image happen on "face texture" where it's supposed to change (eyes close - open) and maybe that part of the texture is for waves on the sea or something like that..

What MaKiPL said.  The image is only 128x64 but when tonberry dumps the vram, it has to dump a 256x256 texture, so the rest of the image is remnants of other textures that are loaded.  I have different things that populate in the top right/bottom of the image if  i'm on different saves, etc.  The only information that matters in that image is the top left block.
Title: Re: [FF8] world map and objects (world.fs)
Post by: Maki on 2015-07-26 13:52:41
I probably found a way to create 32BPP (ARGB) textures using multi generated textures with CLUT palettes, block UV and texture mixing. I'm finishing slowly this original algorithm on battle stages. I have correctly working palette 8BPP indexed texture drawing, UV bounding boxes locations and auto multi CLUT generator. Only mixer and CLUT resolver is TODO. Let's see how will it work. I'll edit this post if I make it to finish texture generator today...
Title: Re: [FF8] world map and objects (world.fs)
Post by: kaspar01 on 2015-07-26 14:48:47
That would be awesome! is it supposed to work whith every tim file?
Title: Re: [FF8] world map and objects (world.fs)
Post by: Maki on 2015-07-26 14:54:20
Only if I got bounding box of segment's UV. I have some problems with CLUT ID. Looks like it works strictly bit style...

CLUT ID for battle stage is probably like this:

#1:00000000 00111100
#2:01000000 00111100
#3:10000000 00111100
#4:11000000 00111100
#5:00000000 00111101
#6:01000000 00111101
#7:10000000 00111101
#8:11000000 00111101

This comes a little familiar as I was coding palette.
The palette was example:
ABCD> 10101011 11001101
And spec was: A, B, G, R

So AB should be moved behind CD so it's:
11001101 10101011  and should be read from left.

In this way:
#1:00000000 00111100 > 00111100 00000000
#8:11000000 00111101 > 00111101 11000000

So #9 will be probably:
00111111 00000000

Maybe... Need to test it.

Random tests:
Pavement is #5, and #5 clut is gray like so it pass. a0stg027.x
Specifical blue element on a0stg012 is 403c so  00111100 01000000  and is #2. Passed!
Okay. Made it! Max CLUT count is 1111 (with 0, so 16)

Complete CLUT reference:
Code: [Select]

#1: 00000000 00111100 (00 3C)
#2: 01000000 00111100 (40 3C)
#3: 10000000 00111100 (80 3C)
#4: 11000000 00111100 (C0 3C)
#5: 00000000 00111101 (00 3D)
#6: 01000000 00111101 (40 3D)
#7: 10000000 00111101 (80 3D)
#8: 11000000 00111101 (C0 3D)
#9: 00000000 00111110 (00 3E)
#10: 01000000 00111110 (40 3E)
#11: 10000000 00111110 (80 3E)
#12: 11000000 00111110 (C0 3E)
#13: 00000000 00111111 (00 3F)
#14: 01000000 00111111 (40 3F)
#15: 10000000 00111111 (80 3F)
#16: 11000000 00111111 (C0 3F)


Dirty sample code #updated (Forgot about reversing I mentioned above):
Code: [Select]
                       byte[] bt = new byte[2];
            Buffer.BlockCopy(buffer, 1, bt, 0, 1); //00
            Buffer.BlockCopy(buffer, 0, bt, 1, 1); //3C
            BitArray ba = new BitArray(bt);
            BitArray CLUTbit = new BitArray(4);
            CLUTbit[3] = ba[15]; CLUTbit[2] = ba[14];
            CLUTbit[1] = ba[1]; CLUTbit[0] = ba[0];

            int[] ClutArray = new int[1];
            CLUTbit.CopyTo(ClutArray, 0);
            return ClutArray[0];

Only texture mixing left... Like damn, I was coding today for whole day... Just this, and I'm taking a break.

Nah, reading image by image and transferring pixels algorithm would take a lot of time. I'll leave it for tomorrow or something...
Title: Re: [FF8] world map and objects (world.fs)
Post by: kaspar01 on 2015-07-27 07:14:24
Good job Maki!

Can't wait to see it finished.

I always had to fix textures manually but I was not always sure if I choose the right palette for every clut (some of them were really similar)..

And it's been a pain in the ass every time XD
Title: Re: [FF8] world map and objects (world.fs)
Post by: Maki on 2015-07-27 15:16:46
Done!
(http://i.imgur.com/cryU5mw.jpg)
(http://i.imgur.com/5pPpuME.jpg)
(http://i.imgur.com/8LRl3rU.jpg)

Also corrected code from yesterday's post, as it had wrong indexes:
Code: [Select]
            byte[] bt = new byte[2];
            Buffer.BlockCopy(buffer, 1, bt, 0, 1); //00
            Buffer.BlockCopy(buffer, 0, bt, 1, 1); //3C
            BitArray ba = new BitArray(bt);
            BitArray CLUTbit = new BitArray(4);
            CLUTbit[3] = ba[1]; CLUTbit[2] = ba[0];
            CLUTbit[1] = ba[15]; CLUTbit[0] = ba[14];

            int[] ClutArray = new int[1];
            CLUTbit.CopyTo(ClutArray, 0);
            return ClutArray[0];

All done. Only wmsetus object remaining to do and maybe world map.
Title: Re: [FF8] world map and objects (world.fs)
Post by: kaspar01 on 2015-07-27 15:49:10
That's so cool!

I'm really happy to finally see so many progress in ff8 tools development  :)

Title: Re: [FF8] world map and objects (world.fs)
Post by: Halfer on 2015-07-27 17:07:48
Oh damn, that's incredible!! :D

I guess I'll have to up the pace with the wmx tool since you are making this much progress with the textues :S
Title: Re: [FF8] world map and objects (world.fs)
Post by: Halfer on 2015-10-19 21:27:01
Little update now!

Since I have a week off from school I started to make the importer finally. I feel kinda sorry that I haven't contributed as I planned but the fact is, that now the project is back and running.

I've built a simple algorithm that can import a model to ff8 world map format. This was the result:
(https://i.gyazo.com/8d54f18f4bf2adb7b6def5338746fda4.png)

So it's a cube (little stretched) with material properties that I manually set because the importer currently generates only vertices and faces.

Just a reminder though, although importing is possible, it's probably necessary to make many options for user because for example if user wants to add object over segment, it has to iterate through blocks. And same goes for multiple segments. Importing to one block however is rather easy as done in picture above.

So that concludes to my question, what options people would like to have in the importer? Also if you don't know what can be done through world map file, feel free to ask.
Title: Re: [FF8] world map and objects (world.fs)
Post by: Maki on 2015-10-19 22:00:55
Oh Halfer, you're a God! That's awesome! :3

There should have an option to select faces to block player movement. Also CLUT selection and texture selection. I keep my fingers crossed and wish you best. We want to see this finished! :3
Take care!
Title: Re: [FF8] world map and objects (world.fs)
Post by: Halfer on 2015-10-19 23:02:23
Oh Halfer, you're a God! That's awesome! :3

There should have an option to select faces to block player movement. Also CLUT selection and texture selection. I keep my fingers crossed and wish you best. We want to see this finished! :3
Take care!

Definitely there eventually will be those options! I have couple of ways to implement those features but i'm open to suggestions!
Title: Re: [FF8] world map and objects (world.fs)
Post by: Halfer on 2015-10-20 19:46:49
Update.

Texture coordinates are being worked on, need to replace the algorithm though, but working so far. Also I manually set texture indices to FF8 model since I haven't done that algorithm yet at all.

(https://i.gyazo.com/3c779a319018e711e891922a5e41df9a.jpg)
(https://i.gyazo.com/f842c2b005a17f9dce14bd9f3cce166e.jpg)

Seems like some combination of material property values are yet to discover because I just found about this:

(https://i.gyazo.com/e123afbe885810a012e5388bab6ad380.png)
Title: Re: [FF8] world map and objects (world.fs)
Post by: Halfer on 2015-10-21 20:40:11
Update, Current phase.

Texture coordinates and indices are now working. Here's a segment that I filled with water blocks. Water blocks were corrected so that 1 quad shows the texture instead of 4 quads as in original file, so it's 4x upscale actually.
(https://i.gyazo.com/0cc4b5c0bd3e4defec5505d3237d2121.png)

I tried to subdivide it to upscale it more but seems like the developers were aware that the blocks including only water can have only 32 faces (32 triangles or 16 quads). So subdividing gave 128 faces which goes beyond the limit (0x900h) for one block (16 blocks in segment).

The importer currently creates a block from an imported wavefront object with:
(minimum information to form the block)

Features the importer doesn't do yet, but are planned to be added:
Title: Re: [FF8] world map and objects (world.fs)
Post by: Halfer on 2015-10-29 18:32:32
I can't believe how I didn't think this sooner but Squaresoft was very lazy or stupid or even both with their world map formats.

The segment includes 16 blocks with their positions in segments header indicating where the block is in file. Well off course you can change the offset, but what I didn't think until now is that why the heck do blocks like the sea/water or any other block that repeats itself need to be separated? Well THEY DON'T!
You can have only 1 instance of that block and indicate all the 16 block headers to that offset. This way the block don't have to have maximum size of 0x900h, in theory it can be 0x9000 long. Only thing that limits it, is block header with the count of faces, vertices and normals which can only have maximum of 256 value.

I noticed that if I put all the block headers to indicate 1 block, texture coordinates changed drastically. However I think I learned how the engine tries to "fix" texture coordinates and I may need to reformat some things in my algorithm for that.

But this sure is really important information since any block that needs to be repeated inside segment can be referenced multiple times while having one instance of it in segment.

*I also did some experiment with the offsets and off course you can set the offset to be over the segment itself. This however doesn't work by itself, because  FF8 engine is changing the location of segments in memory each time you rotate camera. In other words, making it to work needs a hell lot modification on engine algorithms or memory managment.
Title: Re: [FF8] world map and objects (world.fs)
Post by: Yagami Light on 2015-11-04 19:15:12
Great progress Halfer, has anyone delved into the world map chara.one file ? I think it contains geometry for ragnarok/garden/lunatic pandora/etc I also reversed these bytes and found possible beach animation data? Not sure, if anyone wants to have a look

(http://i.imgur.com/K6H9Ns1.png)

(http://i.imgur.com/9n8aNAr.png)
Title: Re: [FF8] world map and objects (world.fs)
Post by: Maki on 2015-11-04 19:57:22
No, you affected only GPU operation opcode. Maybe I'm calling this wrong, but in many cases by different researches thing like this was just called "GPU related", nothing more. Changing this makes you extremely crash sensitive. Ragnarok indeed is part of chara.one file as well as Squall on chocobo, Squall on feet, Zell on feet and etc. but not lunatic Pandora nor wave animation. These are objects from wmsetus/fr/it... file.

Check the updated wiki about wmset files:
http://wiki.qhimm.com/view/FF8/WorldMap_wmsetxx
Check section 16 (textures are in section 42), and for wave animations see section 38/39. It's all I researched at the moment. 

Cheers! :3
Title: Re: [FF8] world map and objects (world.fs)
Post by: Halfer on 2015-11-05 13:39:01
Great progress Halfer, has anyone delved into the world map chara.one file ? I think it contains geometry for ragnarok/garden/lunatic pandora/etc I also reversed these bytes and found possible beach animation data? Not sure, if anyone wants to have a look

Took a quick look and seems like you reversed bytes from one the 3 TIM textures for ragnarok. As Maki said, by doing this, values can get insane and may cause crashes or affect other elements in bad way which in theory shouldn't be affected by the values.

By a quick note about the structure I can say that Chara.one first 4 bytes indicates the size of the whole file. There are 15 textures and it seems like that after every corresponded texture (for example 3 textures for ragnarok) starts a new section with some kind of data.

Edit: Well, just looked again at the file for curiosity and the structure seems to go like this:

Code: [Select]
u32 Chara.one (whole file) size;    // only once in the beginning of the file
Tim texture(s)           // Every texture for one object comes before .mch structure
.MCH file                   

http://wiki.qhimm.com/view/FF8/FileFormat_MCH
Title: Re: [FF8] world map and objects (world.fs)
Post by: Vehek on 2015-11-07 18:14:42
I briefly  wrote about that (http://forums.qhimm.com/index.php?topic=15070.msg211428#msg211428) before. That type of chara.one has a a footer (I mean a structure at the end of the file), much like the regular chara.one header, but with the order of the entries reversed, where the final 4 bytes of the file are the model count, the 4 bytes before that are the first model's first texture offset, and so on. Also, no EE EE EE EE separator.

http://wiki.qhimm.com/view/FF8/FileFormat_ONE
Title: Re: [FF8] world map and objects (world.fs)
Post by: Halfer on 2015-11-07 20:26:46
I briefly  wrote about that (http://forums.qhimm.com/index.php?topic=15070.msg211428#msg211428) before. That type of chara.one has a a footer (I mean a structure at the end of the file), much like the regular chara.one header, but with the order of the entries reversed, where the final 4 bytes of the file are the model count, the 4 bytes before that are the first model's first texture offset, and so on. Also, no EE EE EE EE separator.

http://wiki.qhimm.com/view/FF8/FileFormat_ONE

Seems like you are right, didn't cross my mind to look from the end. Also it looks that EE EE EE EE separators are FF FF FF FF in this file.
Title: Re: [FF8] world map and objects (world.fs)
Post by: Blue on 2015-11-17 11:56:24
Just dropping a quick note to this topic, for someone might not read the wmx2obj thread as actively: I refactored the source code of wmx2obj for readability and efficiency purposes. Known bugs and inefficiencies should finally be removed.

wmx2obj (http://forums.qhimm.com/index.php?topic=16230)

Those interested in using parts of the wmx2obj source code in other software or upgrading wmx2obj, feel free to do so. I released the program under a copyleft license, so you are allowed to do pretty much anything you want with the source code. Sharing is caring.
Title: Re: [FF8] world map and objects (world.fs)
Post by: Quantize on 2015-12-05 13:34:53
Awesome work!
As a long time FF8 fan and a tinkerer I love these creative projects.

I have a question regarding the content this tool can export. I am currently working on a project to recreate the Balamb Garden in 3D and figure every inconsistency out to be able to make it a full building with a 3D/VR walkthrough.
So although i have a lot of reference material, getting original 3D models and textures could help out a lot for me to make it as accurate as possible, is this project capable of exporting this kind of data at of yet?

Thanks!
Title: Re: [FF8] world map and objects (world.fs)
Post by: Halfer on 2015-12-05 13:58:57
This program is intended to export world map vertex, texture and normal data. Creating 3D Balamb Garden from world map visualization of Balamb Garden isn't enough for your kind of project:

(http://vignette1.wikia.nocookie.net/finalfantasy/images/1/13/Balamb_Garden_(stationary).jpg/revision/latest?cb=20090614115328)

This is the Balamb Garden that this program exports/will export.

If you are referencing to the model that is used as a vehicle in the game, then this program is out of bounds on that.
(http://i.imgur.com/OdBu1vW.jpg?1)

However MakiPL did a script/program that exports those models from the files, but I think he hasn't released it yet (probably part of his bigger program ;) ).
Title: Re: [FF8] world map and objects (world.fs)
Post by: Maki on 2015-12-05 19:40:45
However MakiPL did a script/program that exports those models from the files, but I think he hasn't released it yet (probably part of his bigger program ;) ).

Sure. :D Yet, still I hardly can find a time to do this, only GFs enviro and BS reimport is missing.

Quantize, here's the Balamb Garden + Halo/Ring/Floating thing (that even rhymes! xD )
https://www.dropbox.com/s/7o8yvqjcfa5z3ri/Baaalamb.7z?dl=0

(http://snap.ashampoo.com/uploads/2015-12-05/FvrHcdju.png)
Title: Re: [FF8] world map and objects (world.fs)
Post by: LeonhartGR on 2015-12-14 05:02:56
Some gr8 progress! Yeah!
Title: Re: [FF8] world map and objects (world.fs)
Post by: Halfer on 2016-05-28 14:55:54
Hello all!

I'll do a little update and plans for the future on this subject.

First of all, I'd like to say that unfortunately the development has been slow and I'll admit that it's been too slow what I originally planned, however due to my school projects and school in general, I've been pretty busy with it (I'm a student in IT industry majoring in game developing). Well, maybe more on to that later.

Right now I've started to work again on the wmx2obj started by Blue. I've modified it pretty heavily and moved from Java to C#. I'm pretty happy with the current stage it is in, but I'm eager to make it a full-fledged world map editor.

Here's a short brief what's up (please don't mind the messy buttons and texts obj2wmx tab, they are just placeholders  :-X):

(https://i.gyazo.com/ef5f5c28dac3f5183e1ee2f5ac341081.png)

As you can see there's some cool features I've been planning.


So, that's pretty much it for now. I'm happy to hear what you think and more happy to hear what YOU would want as a user to add to the program for better workflow for FF8 world editing. I will edit this post if suggestions do pop up.

Oh, and what do you think about the name Ragnarok? :) ya or na?  ;D