Author Topic: [FF8] world map and objects (world.fs)  (Read 49741 times)

kaspar01

  • *
  • Posts: 118
  • FFVIII Fan & Collector , 3D Artist , FF8RP-WIP
    • View Profile
Re: [FF8] world map and objects (world.fs)
« Reply #75 on: 2015-07-09 22:15:56 »
Haven't really given a thought about this yet but I can see it happening with future updates ;)

Oh.. actually in my head I was going to do it "manually" on 3ds max.. but  of course a function in the tool would be more handy! ;D


Since I still see the untextured version right now I can just guess anyway..

Among these interchangeable zones I can recognize:

-balamb Garden  (before/after lift-off)
-galbadia Garden (before/after lift-off)
-desert prison (inground/exposed)
-missile base (before/after destruction)
-trabia garden (??)
-start of F.H. Bridge (??)
« Last Edit: 2015-07-09 22:42:43 by kaspar01 »

falkTX

  • *
  • Posts: 92
    • View Profile
Re: [FF8] world map and objects (world.fs)
« Reply #76 on: 2015-07-10 12:13:45 »
I haven't looked at those files, but what about esthar before and after entering the area?

Maki

  • 0xBAADF00D
  • *
  • Posts: 602
  • 0xCCCCCCCC
    • View Profile
Re: [FF8] world map and objects (world.fs)
« Reply #77 on: 2015-07-10 19:32:57 »
So, in future we would do a complete model replacement. Unfortunately, we'd still care about limits like 0x9000h for world map segment or uint16 count of tris. I'll complete wmset.obj tool, and start with more advanced memory hack to do realtime geometry edit.

Halfer

  • *
  • Posts: 142
    • View Profile
Re: [FF8] world map and objects (world.fs)
« Reply #78 on: 2015-07-10 19:43:08 »
Well actually it is uint8 for triangles, right? But, I don't know,  that's 256 vertices for 1 block which is pretty small area by itself, but I can see what you mean.

It would be a dream come true if we get to the point where we could edit those restraints like memory caps and so on. Thinking about the PSX memory limitations which are present in the PC version also. If those can be bypassed then the true graphic overhaul can start :P.

I haven't looked at those files, but what about esthar before and after entering the area?

Yes the whole esthar continent is a part of interchangeable sections. That Trabia Garden which kaspar01 mentioned is probably a leftover from early builds replaced with the same section. It probably contained the model and textures before the missiles, but that's never seen in the game so dumping that out seems to be logical answer.
« Last Edit: 2015-07-10 19:46:55 by Halfer »

kaspar01

  • *
  • Posts: 118
  • FFVIII Fan & Collector , 3D Artist , FF8RP-WIP
    • View Profile
Re: [FF8] world map and objects (world.fs)
« Reply #79 on: 2015-07-12 11:54:37 »
What about add a check box to make the .mtl file optional (after first export it become useless I guess and I think that choose 2 times for destination is not a good thing)?

Moreover I get an "invalid texture index" error whith some  mesh (not sure but it seems to happen when I export map not starting from 1).

Edit :

my theory about when the texture error happen was wrong.. here some example of export that gave me error:

268-268
328-328
150-151


but for example i don't get error whith 275-276 ... this is weird..
« Last Edit: 2015-07-12 12:13:36 by kaspar01 »

Halfer

  • *
  • Posts: 142
    • View Profile
Re: [FF8] world map and objects (world.fs)
« Reply #80 on: 2015-07-12 12:02:44 »
Yeah I'll fix that .mtl file to not be mandatory to export everytime.

About invalid texture index. That's odd, I maybe encountered it once or twice when I first made the code but never after that. Do you know specific sections which launches this bug? Would help me to fix is ASAP. Actually if it's not severe issue I'll fix it with next update since the optimization should fix those kind of problems and also would make much less lag while editing the mesh.
« Last Edit: 2015-07-12 12:06:54 by Halfer »

kaspar01

  • *
  • Posts: 118
  • FFVIII Fan & Collector , 3D Artist , FF8RP-WIP
    • View Profile
Re: [FF8] world map and objects (world.fs)
« Reply #81 on: 2015-07-12 13:40:02 »
Yeah I'll fix that .mtl file to not be mandatory to export everytime.

About invalid texture index. That's odd, I maybe encountered it once or twice when I first made the code but never after that. Do you know specific sections which launches this bug? Would help me to fix is ASAP. Actually if it's not severe issue I'll fix it with next update since the optimization should fix those kind of problems and also would make much less lag while editing the mesh.

I edited my previous post and put a little list in it.

Most of the sections I tried gave me the error.

Anyway I manually fixed transparency and fixed garden just to make see this:



defintely worth it  8)
« Last Edit: 2015-07-12 13:42:07 by kaspar01 »

Halfer

  • *
  • Posts: 142
    • View Profile
Re: [FF8] world map and objects (world.fs)
« Reply #82 on: 2015-07-12 13:59:25 »
That's really odd, I don't seem to get the same errors for some reason:


Well anyway, all the texture related problems should be corrected after next update except transparency which has to always be done manually. I'll release the updated version once I get normals working, textures corrected and the segment swapping done, may take a while  :cry:

kaspar01

  • *
  • Posts: 118
  • FFVIII Fan & Collector , 3D Artist , FF8RP-WIP
    • View Profile
Re: [FF8] world map and objects (world.fs)
« Reply #83 on: 2015-07-12 15:45:29 »
That's really odd, I don't seem to get the same errors for some reason:
Hum... maybe it's a 3ds max problem?
I could try to send you one of my files or you could send me one of yours to verify.. or both :D

Update: I try to export segment 268 againg in a new file and now it works..tried to import the old one and still get the error.. maybe an error during export?

Update 2 : I think I understood when the error happens! The first file I export is fine but every following file gives me the error (apparently I have to close and open again the tool to avoid the problem)

Well anyway, all the texture related problems should be corrected after next update except transparency which has to always be done manually. I'll release the updated version once I get normals working, textures corrected and the segment swapping done, may take a while  :cry:
If I may help you somehow let me know :)
« Last Edit: 2015-07-12 16:17:04 by kaspar01 »

Halfer

  • *
  • Posts: 142
    • View Profile
Re: [FF8] world map and objects (world.fs)
« Reply #84 on: 2015-07-14 12:32:13 »
Oh btw, one thing about high resolution textures occurred to me sometime ago.

If we assume that the engine can handle larger size texture pages like 512x512, since it can manage 256x256 over 128x128, it could be possible to replace these files in wmsetXX.obj (and why not in texl.obj but it's not used in old release). The only thing I can see limiting this is that the headers are pointing to specific location in the file, so by changing those it could be possible to replace the old ones in the file. That will not work on texl.obj however because it doesn't have any headers.

Just a thought but should be worth a try, unless the wmsetXX.obj needs to be certain size.

kaspar01

  • *
  • Posts: 118
  • FFVIII Fan & Collector , 3D Artist , FF8RP-WIP
    • View Profile
Re: [FF8] world map and objects (world.fs)
« Reply #85 on: 2015-07-14 13:05:05 »
Oh btw, one thing about high resolution textures occurred to me sometime ago.

If we assume that the engine can handle larger size texture pages like 512x512, since it can manage 256x256 over 128x128, it could be possible to replace these files in wmsetXX.obj (and why not in texl.obj but it's not used in old release). The only thing I can see limiting this is that the headers are pointing to specific location in the file, so by changing those it could be possible to replace the old ones in the file. That will not work on texl.obj however because it doesn't have any headers.

Just a thought but should be worth a try, unless the wmsetXX.obj needs to be certain size.

I'm quite sure that pc version of the game had an option for using hi-res texture (I guess it referred to textl usage since I remember it affected world map only.. )

I guess it could be possibile to exploit that option somehow..

Edit:

I also wonder.. Maki has been able to switch a piece of the world map whith another existing one..
Maybe modify geometry of existing piece making them more detailed would result in a crash but what about "add new pieces" after the last one and try to swhitch them? I mean.. there is a size limit? it applies to the whole file or to the single chunks?we can add new chunks?
« Last Edit: 2015-07-14 14:36:12 by kaspar01 »

Halfer

  • *
  • Posts: 142
    • View Profile
Re: [FF8] world map and objects (world.fs)
« Reply #86 on: 2015-07-14 18:53:36 »
Well I've been thinking about the subdividing of the existing polygons but there is that performance downside. However once I get the program working I can do some testing with normals affecting the performance. If somehow the game is utilizing gourard shading, then we can maybe limit the usage of normals so it wouldn't look worse but give some performance boost. Also it would save space from already limited segments which could be filled with subdivided blocks for better detail. And yes, Maki's program looked very nice. I'm wondering if you are (MakiPL) focused in the engine side of the game because of the application(s)? If you are then I think it's okay that we don't overlap too much, but of course, I wouldn't mind that.

Speaking of the textures itself. I don't think that texl.obj file leads us to anywhere with higher quality textures since it doesn't contain any header except for the textures itself. That would mean the headers have been written somewhere (maybe to .exe) and that makes it impossible to replace the textures with bigger ones. However wmsetXX.obj contains headers for these textures so utilizing those the texture size can maybe be increased.

While it is pretty interesting and definitely worth of a try, I don't plan to do it myself at least yet. The converter itself gives me some hard times since I've low amount of experience with coding still. However I won't let it stop me from doing the program because how else would I ever learn to code if I never do it. I don't want to make you think that there is a chance that I would drop out of this because I do study information and communication technologies and in fall I attend software engineering studies with main focus of game applications.
« Last Edit: 2015-07-14 18:57:48 by Halfer »

Mcindus

  • *
  • Posts: 917
  • Artist, Modder, Musician.
    • View Profile
    • Lunatic Pandora
Re: [FF8] world map and objects (world.fs)
« Reply #87 on: 2015-07-14 22:33:14 »
Hey guys!  I'm not sure if I'm following the issues with the road/rail textures properly, but this is the image that Tonberry dumps in the debug folder:


The image is always 256x256 in tonberry, because it takes a snapshot of the vram, but it looks like the TIM is actually 128x64 in a set of anywhere from 13-18 palettes - I haven't found the .TIM in tim viewer.

kaspar01

  • *
  • Posts: 118
  • FFVIII Fan & Collector , 3D Artist , FF8RP-WIP
    • View Profile
Re: [FF8] world map and objects (world.fs)
« Reply #88 on: 2015-07-14 22:51:49 »
Hey guys!  I'm not sure if I'm following the issues with the road/rail textures properly, but this is the image that Tonberry dumps in the debug folder:


The image is always 256x256 in tonberry, because it takes a snapshot of the vram, but it looks like the TIM is actually 128x64 in a set of anywhere from 13-18 palettes - I haven't found the .TIM in tim viewer.

Thanks.

This confirm the texture I built is almost 90% correct.

and It's interesting to see the top right corner.. If I'm right those texture should be used for ultimecia castle portals.. that's interesting...

Actually I wonder where those models are stored..

Halfer

  • *
  • Posts: 142
    • View Profile
Re: [FF8] world map and objects (world.fs)
« Reply #89 on: 2015-07-14 23:06:18 »
@Mcindus
That is really interesting! I haven't done anything with tonberry, but I would like to know how to take those VRAM snaps. Would help a lot to understand how the game utilizes the memory. Also is it possible to take snap of the whole VRAM with it because at least the psx vram is size of 1MB and it is presented as 1024x512 area. This is probably how it's in the PC also.

@kaspar01
Those castle portals are stored in wmsetXX.obj, I did a lot of studies on that before and I'm 90% sure that it contains the portals because it does contain lunatic pandora and other models used for world map.

kaspar01

  • *
  • Posts: 118
  • FFVIII Fan & Collector , 3D Artist , FF8RP-WIP
    • View Profile
Re: [FF8] world map and objects (world.fs)
« Reply #90 on: 2015-07-14 23:26:32 »

@kaspar01
Those castle portals are stored in wmsetXX.obj, I did a lot of studies on that before and I'm 90% sure that it contains the portals because it does contain lunatic pandora and other models used for world map.

Yes I was thinking the same thing.. MakiPL was working on that side and found some models if I'm right.

A tool that extract them would be cool also.. I'm not sure but I think that no one wrote it yet.

Halfer

  • *
  • Posts: 142
    • View Profile
Re: [FF8] world map and objects (world.fs)
« Reply #91 on: 2015-07-15 00:04:18 »
I think Maki did that already but didn't make any release yet. I guess he's keeping us waiting on purpose so he can surprise us big time ;). Well anyway, the models have been documented by Vehek so it's only matter of programming.

Maki

  • 0xBAADF00D
  • *
  • Posts: 602
  • 0xCCCCCCCC
    • View Profile
Re: [FF8] world map and objects (world.fs)
« Reply #92 on: 2015-07-16 13:58:32 »
Hi! I'm back. :D

Quote
I think Maki did that already but didn't make any release yet
Yes, I did, but the software works highly code side, so that'd require changing offsets and etc in code and recompiling. I'm currently working on AIO toolset, so expect fully working wmsetxx.obj converter next week.

About playing with memory...
FF8 memory is static. The game uses hardcoded addresses to work with objects, so if you add more detailed object to the game, you will probably overwrite other stuff or headers or any sensitive variables. Let me explain this further:

World map segments are 0x9000h stored in memory. There are 5 segments (which I found, but I'm sure I missed at least one). So... the smallest world map segment is the one with water. One water only segment is ~0x2dbb leaving 0x6244 bytes of free space. This space CAN be modified, but 0x9000h cannot be exceed. If you exceed the 0x9000 limit and write like x12300 memory to segment 1 address start, then you'd probably crash the game, or just touch the second segment. Changing this without source would be extremely hard.
However!
You can allocate memory at the end of process, change by assembler the engine to read bigger variables, and call segments from new hardcoded address, which is the one you allocated.

Wmx.obj file can be normally edited, and it's size can be freely changed. Why? The engine reads only specific segments. If you add new, custom built segment at the end of file, and keep 0x9000h size of it with padding if needed, then nothing would happen. I didn't find a opcode, the engine calls for segment load with specific index, but I'll see if the knowledge of where specific segments are will help me with this. If I found it, then I could recall it to use custom built segments from the end of file. FF8 engine reads files by stream, so they can be real-time edited, repacked and etc. For example, you can extract and edit wmx.obj file, even when on worldmap in-game, then re import file, save it, get back to already running game and moving forward, so it can read next segment at specific position. It's the same as any other stream works. It makes the engine don't load whole file, but only it's part. This way, a programmer can get for example a 2MB data, that is stored in the semi center of 2GB file in matter of milliseconds (with current CPUs).

For wmsetus.obj file, it's more complicated. World map objects like cars, garden, ragnarok does not contain padding, and are loaded and allocated just as the game needs. That means, if you wish to replace ragnarok with much more data, then real-time memory editing would affect other vehicles data (however, this didn't appear in my tests, even when I wrote memory that overwritten other objects data loaded in memory, they still were working. I didn't know which one I was overwriting, so probably that's why I didn't notice the change).
If memory editing in this case is not recommended, then normal wmsetus.obj hex'edit should work. The engine would just push the other vehicle data for your new object. (Or the engine loads all vehicles, and keeps them in the memory? I don't know, need to test it someday).

Still, if the wmsetus declared size will be exceed (which I don't know at the moment how much does it takes space in the memory) then you still be able to overwrite some other data.

Another example to clarify this:
Just after world map segments in memory, a core Squall data is stored. This means, if the last segment in memory would be bigger than 0x9000h, then the game would read for example one triangle's vertex as Squall position, instead of this triangle (as we intended to use it). Still, modifying engine to read more than 9000h data and calling it from the allocated by user data at the end of file is the safest and best way. Still, due to x86-86 (32 bits) compilation, you'd need to make sure you doesn't allocate more than 4GB memory in total.

Why does changing camera logical zoom and flying from esthar to bridge would crash the game?
This is the glitch that is made because of "size exceeding" and static memory "bad sides".

This is more detailed exception with memory addresses and related FF8.EXE opcode (thanks to debugger):
Code: [Select]
First-chance exception at 0x00545F36 in FF8_EN.exe: 0xC0000005: Access violation reading location 0x36382834.
This means, that opcode at 0x00545F36 (FF8_EN.exe+145F36 - mov ax,[edi]) tried to read memory at 36382834, but this memory address didn't exist.
Why this happened? The memory exceed it's size limit (probably engine tried to load another texture, but in fact loading this made the engine to overwrite some sensitive parts of memory and in fact ordered the engine to read 36382834 address (which could be just part of texture bytes, or for example changed header, to read for example 255 vertex instead of 10 and completely make the game go insane).

Battle stages - I didn't test memory addresses and sizes yet. Can't write much here.


@Edit:
Just a little back to memory allocation. Making the game allocate memory ADD's null byte at the end of process. This way you can store everything you want, even assembly (sic!). You can still order a process to call specific instruction. You can even add your own code to the game.
« Last Edit: 2015-07-16 14:01:37 by MaKiPL »

Halfer

  • *
  • Posts: 142
    • View Profile
Re: [FF8] world map and objects (world.fs)
« Reply #93 on: 2015-07-16 17:02:12 »
That's a lot of information! You've been rocking it :D

Why does changing camera logical zoom and flying from esthar to bridge would crash the game?
This is the glitch that is made because of "size exceeding" and static memory "bad sides".

This is more detailed exception with memory addresses and related FF8.EXE opcode (thanks to debugger):
Code: [Select]
First-chance exception at 0x00545F36 in FF8_EN.exe: 0xC0000005: Access violation reading location 0x36382834.

This means, that opcode at 0x00545F36 (FF8_EN.exe+145F36 - mov ax,[edi]) tried to read memory at 36382834, but this memory address didn't exist.
Why this happened? The memory exceed it's size limit (probably engine tried to load another texture, but in fact loading this made the engine to overwrite some sensitive parts of memory and in fact ordered the engine to read 36382834 address (which could be just part of texture bytes, or for example changed header, to read for example 255 vertex instead of 10 and completely make the game go insane).

I understand that the engine overwrites wrong memory addresses if the limit is exceeded, but I don't understand why changing the logical zoom of camera cause this kind of crash? As seen the engine always loads fixed amount of segments depending on the camera rotation so shouldn't all the textures and geometry be pushed on the memory also? So does that mean that it is VRAM related issue?

Maki

  • 0xBAADF00D
  • *
  • Posts: 602
  • 0xCCCCCCCC
    • View Profile
Re: [FF8] world map and objects (world.fs)
« Reply #94 on: 2015-07-16 17:34:03 »
Yes, that's weird. It's logical zoom, not the real distance-to camera, so it's indeed weird a bit. I'll take a closer look what's happening there, but my theory is the game additionally stores info "what's really visible". I mean when you fly ragnarok, the world map is loaded by blocks. What if the game stores currently visible data, and changing camera zoom makes the game logic to redraw additional blocks and overflow memory?

I'm going to test this now.

The block that is causing the crash is part of wmsetus.obj file.

@UPDATE: 0x00545F36 MOV appears to access big amount of data. Ticked every milisecond. Works with addresses in range:
01EB8D4C-01EBA758
@UPDATE2: Accessed addresses are 4 byte long.
@UPDATE3: Altering accessed variables makes extremely weird things. For example spams with message "This shouldn't go now...", warping to Balamb, hidden emergency capsule (after coming from space), calling battle with UFO encounter... Weird.
@UPDATE4: Hello human... What a lovely tune.
@UPDATE5: "There is a stone pillar. If you look closely, there's something written on it"
« Last Edit: 2015-07-17 11:28:14 by MaKiPL »

kaspar01

  • *
  • Posts: 118
  • FFVIII Fan & Collector , 3D Artist , FF8RP-WIP
    • View Profile
Re: [FF8] world map and objects (world.fs)
« Reply #95 on: 2015-07-18 23:01:06 »
@UPDATE3: Altering accessed variables makes extremely weird things. For example spams with message "This shouldn't go now...", warping to Balamb, hidden emergency capsule (after coming from space), calling battle with UFO encounter... Weird.
@UPDATE4: Hello human... What a lovely tune.
@UPDATE5: "There is a stone pillar. If you look closely, there's something written on it"[/i]

Those are supposed to be side quest "events" which can be triggered while on the map (Obel Lake side quest).

http://finalfantasy.wikia.com/wiki/Obel_Lake

I guess you found a way to manually trigger some "quest stuff"..
« Last Edit: 2015-07-18 23:06:33 by kaspar01 »

Halfer

  • *
  • Posts: 142
    • View Profile
Re: [FF8] world map and objects (world.fs)
« Reply #96 on: 2015-07-20 11:38:12 »
While looking in the memory I found out the addresses used for the sections. I'm sure that Maki found them already but something interesting I noticed in that is that the engine adds or subtracts 2 from the value indicated in the wmx.obj for example if U1 = 40 and V1 = 60 the engine does some calculations on them and in the memory they are U1 = 42, V1 = 62 .

Sometimes it subtracts instead of adding. That is the reason why the UV's are misplaced in the PC version and the road does not show the white lines.

kaspar01

  • *
  • Posts: 118
  • FFVIII Fan & Collector , 3D Artist , FF8RP-WIP
    • View Profile
Re: [FF8] world map and objects (world.fs)
« Reply #97 on: 2015-07-20 11:49:58 »
That is the reason why the UV's are misplaced in the PC version and the road does not show the white lines.

Ok... that's why they're always flat grey colored..

Hey guys!  I'm not sure if I'm following the issues with the road/rail textures properly, but this is the image that Tonberry dumps in the debug folder:


The image is always 256x256 in tonberry, because it takes a snapshot of the vram, but it looks like the TIM is actually 128x64 in a set of anywhere from 13-18 palettes - I haven't found the .TIM in tim viewer.

Yesterday I edited my texture (added portals piece) and it perfectly match but I Wonder about the bottom of your dumped one.. what's all that noise in the lower half of the texture?

It's supposed to be empty or not? why does it looks that way?
I'm not sure but..any chance that is an animated material texture?
I should check but if I remember right that kind of noise on the image happen on "face texture" where it's supposed to change (eyes close - open) and maybe that part of the texture is for waves on the sea or something like that..
« Last Edit: 2015-07-20 11:59:20 by kaspar01 »

Maki

  • 0xBAADF00D
  • *
  • Posts: 602
  • 0xCCCCCCCC
    • View Profile
Re: [FF8] world map and objects (world.fs)
« Reply #98 on: 2015-07-20 14:24:35 »
The noise usually is the data, that cannot be image, but is forced to display as image. In that case, that looks, that this noise place probably may operate animation, like how to handle it and etc.
« Last Edit: 2015-07-21 13:14:22 by MaKiPL »

Mcindus

  • *
  • Posts: 917
  • Artist, Modder, Musician.
    • View Profile
    • Lunatic Pandora
Re: [FF8] world map and objects (world.fs)
« Reply #99 on: 2015-07-20 16:43:58 »
Yesterday I edited my texture (added portals piece) and it perfectly match but I Wonder about the bottom of your dumped one.. what's all that noise in the lower half of the texture?

It's supposed to be empty or not? why does it looks that way?
I'm not sure but..any chance that is an animated material texture?
I should check but if I remember right that kind of noise on the image happen on "face texture" where it's supposed to change (eyes close - open) and maybe that part of the texture is for waves on the sea or something like that..

What MaKiPL said.  The image is only 128x64 but when tonberry dumps the vram, it has to dump a 256x256 texture, so the rest of the image is remnants of other textures that are loaded.  I have different things that populate in the top right/bottom of the image if  i'm on different saves, etc.  The only information that matters in that image is the top left block.