Qhimm.com Forums

Final Fantasy 8 => FF8 Tools => Topic started by: Blue on 2015-07-06 11:52:17

Title: [PSX/PC] World map conversion tool - wmx2obj (2015-11-17)
Post by: Blue on 2015-07-06 11:52:17
Deprecated. Use this instead (http://forums.qhimm.com/index.php?topic=17068.0).

wmx2obj

wmx2obj is a reverse engineering conversion tool for Final Fantasy 8. It converts the world map file of Final Fantasy 8 into the known Wavefront .obj format — thus making it possible to import the world map geometry into Blender, for instance.

(http://i59.tinypic.com/106hj44.png)

To extract the world map model file wmx.obj from world.fs, use Deling (http://forums.qhimm.com/index.php?topic=13050).

Although the original world map file's format is listed as .obj, it's not the commonly known Wavefront .obj format. Hence, this program was created.

Download

wmx2obj on GitHub (https://github.com/cospi/wmx2obj)
Java Archive (JAR) (https://www.dropbox.com/s/c68udo6fla38hlt/wmx2obj.jar?dl=0)

Features


Possible future upgrades


I, myself, won't unfortunately be working on this program anymore for now. Those interested in upgrading the program, feel free to do so by gathering the source code from GitHub.
Title: Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
Post by: Halfer on 2015-07-06 12:03:41
Awesome job!

(http://i.gyazo.com/6ad7e9f9da1062aaf7ed0dad2df2e004.png)


I'll start improving the program by adding a support for texture exporting with the geometry. After that I'll make an importer. In theory the importer could work without texture support but it would be pretty pointless since you would only be able to modify current vertex and face positions.
Title: Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
Post by: kaspar01 on 2015-07-06 23:11:47
Amazing!

I love you guys :)

Can't wait to see it textured!
I already prepared all the textures whith the correct palette since 2 years ago for this moment XD


Title: Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
Post by: Blue on 2015-07-07 11:55:54
Just a quick note to those looking to upgrade the software: the source code is commented to clear confusion over its functionality. However, if you are having trouble trying to understand some parts of the code, just drop a message.
Title: Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
Post by: DanTsukasa on 2015-07-09 10:41:04
Oh man this is AWESOME! I've been waiting for exports for FF8 and FF9, I'm trying to create my own world map and looking at how these were done is really helpful.

Great work.
Title: Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
Post by: Blue on 2015-07-09 10:48:43
Thanks, everyone!

Halfer is currently in the progress of adding texture support to wmx2obj. Check out this thread (http://forums.qhimm.com/index.php?topic=15979.msg229910#msg229910) for smaller updates before a full release.
Title: Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
Post by: DanTsukasa on 2015-07-10 09:40:55
Is tere any way to fix one edge of the map where it seems to just be a bunch of pieces stuck together that otherwise make an island or 2?
Title: Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
Post by: kaspar01 on 2015-07-10 09:54:14
Is tere any way to fix one edge of the map where it seems to just be a bunch of pieces stuck together that otherwise make an island or 2?

As I was sayng in the other topic those "pieces stuck togheter" are extra interchangeable parts of the map (for example the balamb garden area before and after the school become a vehicle)

You can just delete them if you just want to see the full map only (or export only the segments you want instead of all the 835 ones).
Title: Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
Post by: Blue on 2015-07-10 11:10:40
As I was sayng in the other topic those "pieces stuck togheter" are extra interchangeable parts of the map (for example the balamb garden area before and after the school become a vehicle)

You can just delete them if you just want to see the full map only (or export only the segments you want instead of all the 835 ones).

This is correct. Only convert segments 1 to 768 to exclude the interchangeable parts.
Title: Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
Post by: Halfer on 2015-07-10 15:07:59
wmx2obj version 2!

This program exports the world map as a wavefront .obj and material tempalte library .mtl file.

https://www.dropbox.com/s/dg6su56s5x3dlv5/wmx2obj.jar?dl=0 (https://www.dropbox.com/s/dg6su56s5x3dlv5/wmx2obj.jar?dl=0)

What's new compared to prior version:

Texture support added with correct material and vertex groups. UV's have been corrected to minimize the UV errors present in the PC version for some reason. Please report any UV calculation errors or improvements if you find the current settings to be a bit off.

Instructions:

Good to know before use:


Thanks to kaspar01 for his corrected textures that he did. I was able to test my calculations and if the textures worked with these and this is how it looks:
(http://i.gyazo.com/f5120256f216c21958519f3a90e027bf.jpg)

Rails, roads and water have their own texture pages. Sadly the rails and roads are not in one texture page so I didn't spend time to correct this manually. However, you can create your own textures by looking up the UV maps from blender for example.

I purposely didn't put the source code for now, but if you want it I'll publish it!
 
Please report any bugs and wishes for improving the program. I'll start to make the importer once I have set my mind on how to wisely develop it. Also the fact that textures are hard to extract from the files is pretty stupid so maybe I'll look on to it someday.
Title: Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
Post by: kaspar01 on 2015-07-10 16:10:26
Awesome!

I'll test it (as soon as I get home) and try to manually fix rails and road texture.. and probably I'll try to re-create animated material for water :)

About new feature you could add.. maybe a gui for what I think could be useful..

The idea should be a simple switcher for the interchangeable zones that automatically replace the interchangeable mesh during the export..

A bunch of choice that let you chose which version you want to export (for example : Balamb garden "on ground" or "crater" )

We should list all the matches between interchangeable pieces (I can help whith that I guess)

I think this function could be useful for future modding :)

[Update]
For water texture it's been easy but... rails and road is not..

I'm looking at their UV and it makes me think they should be 256x256 texture like the others..

(http://s27.postimg.org/5tovphpkj/UV_Rail.png)(http://s18.postimg.org/i347c0juh/UV_Road.png)

I found some little pieces that seems to match whith the UV groups but I have to guess exactly where to place each..

[Update 2]

This is time consuming but my theory looks to be correct..

pieces of rails appear to be scattered in wmsetus file.. I'm manually rebuilding the texture for it :

(http://s13.postimg.org/bo8tw9itz/rails.jpg)
Title: Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
Post by: Halfer on 2015-07-11 08:40:20
Yeah, I should have probably mentioned that the game engine treats the rail and road textures as an one texture the same way as others. Timviewer shows them sepereated in wmsetXX.obj file but they are loaded to frame buffer in complete texture (128x128 I think). I found a program which can tell the VRAM locations here: http://www.romhacking.net/utilities/799/ (http://www.romhacking.net/utilities/799/)
Title: Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
Post by: kaspar01 on 2015-07-11 09:10:41
Yeah, I should have probably mentioned that the game engine treats the rail and road textures as an one texture the same way as others. Timviewer shows them sepereated in wmsetXX.obj file but they are loaded to frame buffer in complete texture (128x128 I think). I found a program which can tell the VRAM locations here: http://www.romhacking.net/utilities/799/ (http://www.romhacking.net/utilities/799/)

Thanks! Never seen that tool before but I'm gonna try it for sure :)

Anyway the size for rail and road is most likely 256x256 even if only a little portion of it is actually used (according to my UV tests..I placed those piece from wmsetXX and placed in a 256x256 texture and they fit correctly..just need time to understand which piece goes where.. so I guess I'm right about this..)

Another thing that should be said is that rails material does not striclty applies on wolrd map rails only but seems that is used for some "road" pieces (usually for city entrance like the green one you can see in my pic)..and for FH bridge.
Title: Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
Post by: Halfer on 2015-07-11 11:25:04
Well the UV's don't depend on the resolution so you can put higher or lower resolution images in blender or 3dsmax. However I'm not sure what happens to UV's when you put for example 1024x1024 texture, it may be messed up a bit, but then it's just a matter of new calculation and that's it. Speaking of it I can make an improvement so that when you export the world map you can choose which resolution textures you want to use so it doesn't mess up the UV's.

The problem with the UV placement is that they are presented in hex values 0-255, so I have to manually do the calculations because for example 0x7f is 127 in decimals and 127/255 = 0.498... while it's supposed to be 0.5 on UV grid.
Title: Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
Post by: kaspar01 on 2015-07-11 13:32:58
Well the UV's don't depend on the resolution so you can put higher or lower resolution images in blender or 3dsmax. However I'm not sure what happens to UV's when you put for example 1024x1024 texture, it may be messed up a bit, but then it's just a matter of new calculation and that's it. Speaking of it I can make an improvement so that when you export the world map you can choose which resolution textures you want to use so it doesn't mess up the UV's.

The problem with the UV placement is that they are presented in hex values 0-255, so I have to manually do the calculations because for example 0x7f is 127 in decimals and 127/255 = 0.498... while it's supposed to be 0.5 on UV grid.

The reasong why I think it's 256 is that if I export the UV map in that dimension to see where UV group are and then I pick the pieces from wmsetXX files and place in a map which is exactly that big they fit whih no need of scaling.
Moreover every other texture for the map is that big so..


I'm tryng to configure an emulator to rip out those 2 texture but whith bad result till now..


[Update]

Fun fact: most likely road and rail use same textures in same position..or apparently the things which are in the same position in the uv actually have the same texture so..basically they could probably use the same "merged" texture.. i guess..



[Update 2]

Done!

(http://s7.postimg.org/9vn9jgdhn/Wroldmap.jpg)

I'm not 100% sure about 2 or 3 texture pieces that could be switched but it looks almost the same so.. I guess it doesn't matter :)


I confirm that road and rail can use the same texture.

fun fact 2: there is a little piece of texture which is apparently not used (not from road or rail at least..)


Title: Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
Post by: Halfer on 2015-07-11 16:00:15
wmsetXX textures are 128x128 each. texl.obj contains only 256x256 big textures. I think that with the current calculations both works with the geometry, don't know about larger textures.

Quote
Fun fact: most likely road and rail use same textures in same position..or apparently the things which are in the same position in the uv actually have the same texture so..basically they could probably use the same "merged" texture.. i guess..

They may be on the same page with each other, but I don't know how the engine treats it since the texture call from wmx.obj indicates that the page is different. There may be some sort of memory handler in the engine that works with the PSX VRAM to ease the job for GPU. One reason why I believe that is when memory hacking the camera in the game to value of 300 (1024 is normal camera) which zooms out the camera from the player makes the game crash when flying over esthar and FH bridge. It makes the bridge textures disappear before crash. It could also mean that the memory buffer can't keep up with the textures, but it may be too soon to make this kind of conclusions.

Quote
I confirm that road and rail can use the same texture.

fun fact 2: there is a little piece of texture which is apparently not used (not from road or rail at least..)

Nice job! Looking great :P
Title: Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
Post by: halkun on 2015-07-11 16:53:44
Can I get a close-up of the train tracks? I'm interested in making a metro map of those (Once an for all)
Title: Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
Post by: kaspar01 on 2015-07-11 18:06:36
Can I get a close-up of the train tracks? I'm interested in making a metro map of those (Once an for all)
it's really low poly and low res but here it is : http://s2.postimg.org/jqe3yenjc/close_up.jpg

Title: Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
Post by: Halfer on 2015-07-11 18:28:05
it's really low poly and low res but here it is : http://s2.postimg.org/jqe3yenjc/close_up.jpg

Seems like the road now has the white lines like in the PSX version. Show me if you see any UV misplacement's so I can try to fix them :).
Title: Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
Post by: kaspar01 on 2015-07-11 19:38:23
Seems like the road now has the white lines like in the PSX version. Show me if you see any UV misplacement's so I can try to fix them :).

Of course :)

I found this not far from deling city (north from missile base).

(http://s10.postimg.org/6toeuj715/uv_bug.jpg)

This is the only "reallly bad" thing i found.

Some other are just "not perfect seamless" but I don't know if it's a problem from the original game.

Another weird thing is that many face have probably wrong normals.
Title: Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
Post by: Halfer on 2015-07-11 19:57:32
Can you post a larger scale image of the above, can't identify the beach. Never mind, found it, I've problems with reading  :-X

The textures not being seamless is probably my calculation error since it's not perfect yet because of the hex values, I'll try to search an answer for it.

About normals. I documented in reverse engineer forums that wmx.obj contains some sort of shadowing values. Well I now know that they are normals and probably used for Gourard shading which PSX GPU can utilize (look it up if interested). For the next version I'll include normals too so the shading is not a problem after it. Though before I put normals in use I want to improve to current algorithm for texture vertexes because right now it's taking too much memory as you can see from the file size. After optimizing the algorithm I would say the file size will drop about 1/4 or 1/3

I had to check the seam thing and it appears in FF8 also:
(http://i.gyazo.com/f8427cc171b70817cdf655c0c41fb64e.png) (http://i.gyazo.com/945347a822d3feca5d8f530aff39edc3.png)

Sorry for the first picture but I hope you can see it, my FF8 uses 128x128 textures since it's the older release. However these kind of problems occurs all over the map because of the developers. Also the odd stretches like in the kaspar01's picture are misplaced UV's by developers. There's probably nothing that can be done automatically to these unless all the UV's are re-generated.

However it looks better in game because of the renderer which makes it look more seamless, although the problem exists there even if it's harder to notice.
Title: Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
Post by: halkun on 2015-07-11 22:45:59
it's really low poly and low res but here it is : http://s2.postimg.org/jqe3yenjc/close_up.jpg


No, I mean the routes, not the actual track. I want to make a metro map like the London tube underground map

(http://i.imgur.com/HbWO2AY.png)

to something like below

(https://ic.arc.losrios.edu/~veiszep/11spr2004/Pangan/Pangan_2_Zone1map.gif)
Title: Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
Post by: kaspar01 on 2015-07-12 00:54:59
About normals. I documented in reverse engineer forums that wmx.obj contains some sort of shadowing values. Well I now know that they are normals and probably used for Gourard shading which PSX GPU can utilize (look it up if interested). For the next version I'll include normals too so the shading is not a problem after it. Though before I put normals in use I want to improve to current algorithm for texture vertexes because right now it's taking too much memory as you can see from the file size. After optimizing the algorithm I would say the file size will drop about 1/4 or 1/3

I had to check the seam thing and it appears in FF8 also

Sorry for the first picture but I hope you can see it, my FF8 uses 128x128 textures since it's the older release. However these kind of problems occurs all over the map because of the developers. Also the odd stretches like in the kaspar01's picture are misplaced UV's by developers. There's probably nothing that can be done automatically to these unless all the UV's are re-generated.

However it looks better in game because of the renderer which makes it look more seamless, although the problem exists there even if it's harder to notice.

Yep it looks better since it render pixels as they are I guess..

Anyway for the idea I gave you about the switcher I wrote a list that you could use for the switch funcion:

Code: [Select]
Interchangeables zones:

Eshtar

374 to 381  <==>  769 to 776
406 to 413  <==>  777 to 784
438 to 445  <==>  785 to 792
470 to 477  <==>  793 to 800
502 to 509  <==>  801 to 808
534 to 541  <==>  809 to 816
566 to 573  <==>  817 to 824

Trabia Garden

150 to 151  <==>  825 to 826


Galbadia Garden

268  <==>  827


Balamb Garden

275 to 276  <==>  828 to 829


Missile Base

328  <==>  830

Trabia Crater

215 to 216  <==>  831 to 832
247 to 248  <==>  833 to 834


Desert Prison

362  <==>  835


The list is not yet confirmed (I'll test it tomorrow..it's 3 A.M. here XD) Edit2 : Checked the list and it looks to be all correct


Edit:

@halkun this should be what you asked for: http://s16.postimg.org/m6kmcyl1h/Full_Map_HD.jpg
Title: Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
Post by: Zervox on 2015-07-13 01:40:09
Code: [Select]
public void exportFile(File file) throws IOException {
    PrintWriter printWriter = new PrintWriter(file);
    printWriter.println("# Converted to Wavefront .obj with wmx2obj");
    printWriter.println("# wmx2obj \u00A9 2015 Aleksanteri Hirvonen");
    printWriter.println();
for (Segment segment : segments) {
printWriter.println("g " + segment.getID());
writeVertices(segment,printWriter);
writeFaces(segment,printWriter);
}
    printWriter.close();
}

private void writeVertices(Segment segment,PrintWriter printWriter) {
    // take the coordinate values of each vertex instance
    // and write to file
    printWriter.println("# List of geometric vertices");
       
    for (Block block : segment.getBlocks()) {
        for (Vertex vertex : block.getVertices()) {
            printWriter.println(
                    "v "
                    + vertex.getCoordinate(Vertex.X) + " "
                    + -vertex.getCoordinate(Vertex.Y) + " "
                    + -vertex.getCoordinate(Vertex.Z)
            );
        }
    }
    printWriter.println();
}

private void writeFaces(Segment segment,PrintWriter printWriter) {
    // take the vertex index values of each face instance
    // and write to file
    printWriter.println("# Polygonal face elements");
    for (Block block : segment.getBlocks()) {
        for (Face face : block.getFaces()) {
            printWriter.print("f ");
            for (Integer vertexIndex : face.getVertexIndices()) {
                printWriter.print(vertexIndex + " ");
            }
            printWriter.println();
        }
    }
}

I added this code for exporting  each segment as a separate wavefront part hope it would please someone.
not sure about javascript and how I am passing the segments as I converted my code from C++
Title: Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
Post by: kaspar01 on 2015-07-13 07:08:06
I added this code for exporting  each segment as a separate wavefront part hope it would please someone.
not sure about javascript and how I am passing the segments as I converted my code from C++

This is really good indeed.
Since they're separate chunk it makes perfectly sense  (and will be handy for editing/modding) :)

Not sure it can be done but.. naming the chunk whith the segment number could be useful too (it's been quite frustrating calculate the number for interchangeable parts)
Title: Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
Post by: Halfer on 2015-07-13 10:13:39
I'll make that happen in new version. However would you really prefer if each segment was separate .obj file? I mean isn't it frustrating to import each file? If not then I can make a setting to change these properties so you can choose if you want it stitched or separated.
Title: Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
Post by: Zervox on 2015-07-13 11:15:26
My code already names the segments in the obj file with corresponding integer id,so that 3ds max will import them as seperate mesh parts.
regarding saving an area to separate .obj files, it probably isn't a good idea, already with segments alone they make about 1 part per area, meaning a total number of 835 .obj files.

Also I think we should do

Code: [Select]
iterate segments {
Segment seg = segmentiterator;
write("g "+seg.getID())
iterate  blocks(j, seg.getBlocks()) {
               Blocks b = blockiterator;
                writeVertices(b, printWriter);
                writeTextures(b, printWriter);
                writeFaces(b, printWriter);
to save some processing time removing uneeded for loops, I noticed atleast for me that the textured version takes FOREVER to finish, reducing the amount of loops would probably help alot.
Title: Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
Post by: kaspar01 on 2015-07-13 11:26:01
I'll make that happen in new version. However would you really prefer if each segment was separate .obj file? I mean isn't it frustrating to import each file? If not then I can make a setting to change these properties so you can choose if you want it stitched or separated.

As Zervox said that's not making 1 file for each part but make 1 file containing multiple mesh.

Actually having separated file could be a third option so that we have a choice:

-1 file 1 mesh (current)
-1 file multiple mesh (Zervox's one)
-multiple files

My code already names the segments in the obj file with corresponding integer id,so that 3ds max will import them as seperate mesh parts.

Very nice!
I'd like to test it as soon as possible.
Title: Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
Post by: Halfer on 2015-07-13 11:59:20
As Zervox said that's not making 1 file for each part but make 1 file containing multiple mesh.

Actually having separated file could be a third option so that we have a choice:

-1 file 1 mesh (current)
-1 file multiple mesh (Zervox's one)
-multiple files

Very nice!
I'd like to test it as soon as possible.


Yeah that's wiser, I'll make that an option.

to save some processing time removing uneeded for loops, I noticed atleast for me that the textured version takes FOREVER to finish, reducing the amount of loops would probably help alot.

Yeah it is taking time to finish right now because it's not really optimized yet, right now it just writes every texture indices to the file which isn't really optimal. However I'll optimize it to only write specific indices only once and make the faces to refer those which will speed up the exporting process and also reduces the lag seen in 3d editing programs.
Title: Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
Post by: Zervox on 2015-07-13 13:28:14
Ok so, tested doing blocks outside then it occured to me the way it is built that obviously you can't preloop blocks because a segment has multiple blocks which needs to be written in correct order else the retriangulate of 3ds max will destroy 90% of the vertices. but doing the segment then passing the looped segment into the write functions still works which would cut down the loops from 2 505 to 835 for the outer segment loop alone also it would reduce the complexity of the code allowing compiled code to execute better.

the texture indices would also help alot in this regard, another thing I found atleast when did the wavefront in C++ as the java version used to much time for impatient me to see, is that materials are written multiple times even within the same segment, so some sort of lookup regarding material type to make each block sort printed faces by material so that instead of

Code: [Select]
usemtl 9
f 390519//1550113 390518//1550114 390524//1550115
usemtl 1
f 390517//1550116 390530//1550117 390499//1550118
f 390491//1550119 390517//1550120 390492//1550121
usemtl 9
f 390523//1550122 390512//1550123 390519//1550124
usemtl 8
f 390516//1550125 390512//1550126 390523//1550127
usemtl 9
f 390521//1550128 390522//1550129 390517//1550130
f 390520//1550131 390521//1550132 390491//1550133
f 390497//1550134 390495//1550135 390521//1550136
f 390506//1550137 390497//1550138 390520//1550139
f 390518//1550140 390520//1550141 390490//1550142
f 390512//1550143 390506//1550144 390518//1550145
usemtl 0
f 390517//1550146 390529//1550147 390532//1550148
usemtl 8
f 390515//1550149 390511//1550150 390516//1550151
f 390514//1550152 390510//1550153 390515//1550154
f 390514//1550155 390508//1550156 390509//1550157
f 390505//1550158 390511//1550159 390510//1550160
f 390503//1550161 390510//1550162 390509//1550163
f 390508//1550164 390503//1550165 390509//1550166
f 390496//1550167 390505//1550168 390503//1550169
usemtl water
f 390502//1550170 390498//1550171 390501//1550172
usemtl 9
f 390490//1550173 390491//1550174 390493//1550175

it would be

Code: [Select]
usemtl 0
f 390517//1550146 390529//1550147 390532//1550148
usemtl 1
f 390517//1550116 390530//1550117 390499//1550118
f 390491//1550119 390517//1550120 390492//1550121
usemtl 8
f 390516//1550125 390512//1550126 390523//1550127
f 390515//1550149 390511//1550150 390516//1550151
f 390514//1550152 390510//1550153 390515//1550154
f 390514//1550155 390508//1550156 390509//1550157
f 390505//1550158 390511//1550159 390510//1550160
f 390503//1550161 390510//1550162 390509//1550163
f 390508//1550164 390503//1550165 390509//1550166
f 390496//1550167 390505//1550168 390503//1550169
usemtl 9
f 390519//1550113 390518//1550114 390524//1550115
f 390523//1550122 390512//1550123 390519//1550124
f 390521//1550128 390522//1550129 390517//1550130
f 390520//1550131 390521//1550132 390491//1550133
f 390497//1550134 390495//1550135 390521//1550136
f 390506//1550137 390497//1550138 390520//1550139
f 390518//1550140 390520//1550141 390490//1550142
f 390512//1550143 390506//1550144 390518//1550145
f 390490//1550173 390491//1550174 390493//1550175
usemtl water
f 390502//1550170 390498//1550171 390501//1550172

Hope I am not butting in on too much for you Halfer. ;)
Title: Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
Post by: Halfer on 2015-07-13 14:12:10
No no, that's really good that you point these things out :D! Hadn't really thought about that material thing yet but yes, that sounds like the most optimal way to do it, thanks for the tip! :)

Ok so, tested doing blocks outside then it occured to me the way it is built that obviously you can't preloop blocks because a segment has multiple blocks which needs to be written in correct order else the retriangulate of 3ds max will destroy 90% of the vertices. but doing the segment then passing the looped segment into the write functions still works which would cut down the loops from 2 505 to 835 for the outer segment loop alone also it would reduce the complexity of the code allowing compiled code to execute better.

Can you open this up a bit? I have troubles understanding what you are meaning with this.
Title: Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
Post by: Zervox on 2015-07-13 14:20:17
Ah, yeah, I expected it would probably be hard to follow due to crappy writing on my part.
Rewrite:

I first suggested doing
Code: [Select]
for loop (segments){
    WriteVertices(segments[i],PrintWriter);
    WriteTextures(segments[i],PrintWriter);
    WriteFaces(segments[i],PrintWriter);
}
doing a loop over segments only once is 835 iterations max, as the range of world chunks is a total of 835 due to the interchangable parts.
the  way it done now is
Code: [Select]
WriteVertices(PrintWriter);
WriteTextures(PrintWriter);
WriteFaces(PrintWriter);
Which loops segments 3 times over(one for each function call), so a total of 2505 loops, since these are seperate, each loop inside a loop adds complexity,
so being able to reduce amount of loops from 3 to 1 on first stage would reduce conversion rate noticably due to code complexity reduction(compiler wise).

I later suggested doing
Code: [Select]
for loop (segments){
   for loop(blocks)
    WriteVertices(blocks[j],PrintWriter);
    WriteTextures(blocks[j],PrintWriter);
    WriteFaces(blocks[j],PrintWriter);
}
Which didn't work as I wanted it to as it would make 3ds max tear the model to pieces(mangled mesh) on import, because the indices etc didn't match the segment.
Title: Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
Post by: Halfer on 2015-07-13 14:40:30
Thanks for this tip! That sure will release a lot of load during exporting. I will rearrange the code that way since I have to do some parts again after all.
Title: Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
Post by: Zervox on 2015-07-14 05:58:22
Also found in the block structure
Code: [Select]
public void generateFaces(byte[] data, int vertexIndexOffset) {
        // initialize face instances
        for (int face = 0; face < faceCount; face++) {
            addFace(new Face(face));
        }
       
        // find vertex indices of the face instances
        for (Face face : faces) {
            face.findVertexIndices(
                    data,
                    offset + face.getId() * FACE_BYTES,
                    vertexIndexOffset
            );
        }
    }
   
    public void generateVertices(byte[] data, int segmentX, int segmentZ) {
        // initialize vertex instances
        for (int vertex = 0; vertex < vertexCount; vertex++) {
            addVertex(new Vertex(vertex));
        }
       
        // find vertex coordinates of the vertex instances
        for (Vertex vertex : vertices) {
            vertex.findCoordinates(
                    data,
                    offset + faceCount * FACE_BYTES
                            + vertex.getId() * VERTEX_BYTES,
                    offsetX,
                    offsetZ,
                    segmentX,
                    segmentZ
            );
        }
    }
in C++ containers similar to ArrayList in java allows
Code: [Select]
public void generateFaces(byte[] data, int vertexIndexOffset) {
        // find vertex indices of the face instances
        for (int face = 0; face < faceCount; face++) {
            Face ref=faces.add(new Face(face));
            ref.findVertexIndices(
                    data,
                    offset + ref.getId() * FACE_BYTES,
                    vertexIndexOffset
            );
        }
    }
   
    public void generateVertices(byte[] data, int segmentX, int segmentZ) {
        // initialize vertex instances
        // find vertex coordinates of the vertex instances
        for (Vertex vertex : vertices) {
             Vertex ref=vertices.add(new Vertex(vertex));
            ref.findCoordinates(
                    data,
                    offset + faceCount * FACE_BYTES
                            + ref.getId() * VERTEX_BYTES,
                    offsetX,
                    offsetZ,
                    segmentX,
                    segmentZ
            );
        }
    }

Is something like this not possible in Java?
if it is, it would save alot of additional loop complexity.
it would still be possible to just access it straight after inserting it(as add(object) is put into the end) by doing
Code: [Select]
Face ref=faces.get(face);I'd still say the addVertex and addFace amongst other functions
does not need their
Code: [Select]
public void addFace(Face face) {
        if (faces.size() < faceCount) {
            faces.add(face);
        }
    }
   
    public void addVertex(Vertex vertex) {
        if (vertices.size() < vertexCount) {
            vertices.add(vertex);
        }
    }
ArrayList size checks, as the faceCount and vertexCount loop is always the same, meaning these branching if statements are just wasting CPU cycles.
Title: Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
Post by: Zervox on 2015-07-15 03:17:00
Had some fun exporting the mesh from 3ds max to a heightmap ;D

(http://i61.tinypic.com/k2k3yh.jpg)
Title: Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
Post by: Zervox on 2015-07-15 09:33:36
For Faces I noticed it really only ever uses one texturePage in the current code(if this is correct)
we should make
Code: [Select]
ArrayList<String> texturepages;
Code: [Select]
String texturepage;and do
Code: [Select]
String getPage(){return texturepage;}replace all
Code: [Select]
texturepages.add("***");with
Code: [Select]
texturepage="***";
Then do a custom compare( I don't know java enough for this)
Code: [Select]
CustomComparator {
    public boolean compare(Face a, Face b) {
        return a.texturepage.before(b.texturepage);
    }
}
Then we can after the loop in GenerateFaces do
Code: [Select]
faces.sort(CustomComparator);
Title: Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
Post by: Halfer on 2015-07-15 10:23:02
Well it does use 18 texture pages right now but the way the code adds them to ArrayList is that even if the next texture page is the same as last one it adds it to the ArrayList. To minimize the memory usage it should be done that it only adds the next texture page when the value changes. I think this is what you mean right?

Oh and the heightmap looks cool :P
Title: Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
Post by: Zervox on 2015-07-15 11:01:03
From what I saw in Blocks generate faces
Code: [Select]
void findTexturePage(byte data[], int offset, int faceIndexOffset, int textureIndexOffset)is only called once
and all the if checks in findTexturePage are checked at once, meaning if one succeds the others won't get added so it is technically only one texture page in Face class active as texturepages always returns 1 in size to me.
Also due to
Code: [Select]
public String getPage()
    {
        return (String)texturepages.get(texturepages.size() - 1);
    }
it still just sends the last created one, which means all other data stored in faces are of no use. or is this just for debugging code and that code will be changed?

at write out I just did in C++ that all the Texture indices that are equal got written out together once per block, this reduced the total size of the file from 80MB to 38.6MB which is quite some savings, it did however add a bit of processing time.

also a change I'd like to suggest is moving findTextureIndices to Block.
considering TextureCoords is only x and y , this way you can get away with Dbl x and Dbl y on textures. although it would write out 3xtextures per fetch, it is however alot easier for writing a code sort for texture coordinates. eg

in Texture
Code: [Select]
public class Texture
{

    public Texture(int id,double X,double Y)
    {
        this.id = id;
        this.X=X;
        this.Y=Y;
    }

    public int getId()
    {
        return id;
    }

    public double getX()
    {
        return X;
    }
   public double getY()
    {
        return Y;
    }

    public int getPage()
    {
        return ((Integer)texturepages.get(texturepages.size() - 1)).intValue();
    }

    public void print()
    {
        System.out.print(texturepages);
    }

    public static final int COORDINATE_BYTES = 1;
    public static final int p = 0;
    private final int id;
    private double X;
    private double Y;
    private final ArrayList texturepages = new ArrayList();
}
in Block
Code: [Select]
public void findTextureIndices(byte data[], int offset)
    {
        textures.add(new Texture(textures.size(),(data[offset + 6] & 0xff) + 0.5D) / 256D,(data[offset + 7] & 0xff) + 0.5D) / 256D)
        textures.add(new Texture(textures.size(),(data[offset + 8] & 0xff) + 0.5D) / 256D,(data[offset + 9] & 0xff) + 0.5D) / 256D)
        textures.add(new Texture(textures.size(),(data[offset + 10] & 0xff) + 0.5D) / 256D,(data[offset + 11] & 0xff) + 0.5D) / 256D)
    }
 public void generateTextures(byte data[], int faceIndexOffset, int textureIndexOffset)
    {
        for(int texture = 0; texture < faceCount; texture++) findTextureIndices(data, offset + texture * 16))
    }


by the way if you are wondering how I got your code changes is because I was too lazy to ask for source and decompiled your code, sorry about that. ;)


also, don't know where I read it, but someone somewhere said there was a limitation to how many faces or vertices can be on each terrain chunk, which I can't really see, a pure water tile uses 512 faces, 400 vertices, but the chunk with balamb town for example uses 790 vertices and 1156 faces, which suggests to me that it should be possible to put more detail onto the FF8 world.
Title: Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
Post by: kaspar01 on 2015-07-15 11:25:41
..by the way if you are wondering how I got your code changes is because I was too lazy to ask for source and decompiled your code, sorry about that. ;)

Yeah.. like many others here do what they do for the same reason.. because they are too lazy to ask Square for source code  :-D :-D :-D
Title: Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
Post by: Zervox on 2015-07-15 11:27:08
Yeah.. like many others here do what they do for the same reason.. because they are too lazy to ask Square for source code  :-D :-D :-D

Hah! Made me chuckle.  ;D
Title: Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
Post by: Halfer on 2015-07-15 15:02:07
Quote
by the way if you are wondering how I got your code changes is because I was too lazy to ask for source and decompiled your code, sorry about that. ;)

That's totally okay, after all I did say that I will publish the source if anyone wants it. Didn't just do it because I thought I would pull out next version out pretty quickly :P. It's in github though so I can just link it if needed https://github.com/Halferi/wmx2obj  Well there you have it :D, that's the same version as the compiled one though.

Quote
it still just sends the last created one, which means all other data stored in faces are of no use. or is this just for debugging code and that code will be changed?

Well you can say that from the whole code. I never intended it to be final yet because I knew that I still have to change things for optimization. That also includes the rearranging of functions to easier upgrades in future once the program is "ready". But don't get me wrong, I do really appreciate that you do these optimization checks, also saves time from me :P.

Quote
also, don't know where I read it, but someone somewhere said there was a limitation to how many faces or vertices can be on each terrain chunk, which I can't really see, a pure water tile uses 512 faces, 400 vertices, but the chunk with balamb town for example uses 790 vertices and 1156 faces, which suggests to me that it should be possible to put more detail onto the FF8 world.

I don't think that there is any limitation to it technically. Problems that may occur though is the performance, but testing that comes later. Each section can hold technically 0x9000 bytes which is 36864 bytes in decimals. The world map currently hardly utilizes this space fully at all, in fact I don't seem to remember any section to take over half of the maximum size.
Title: Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
Post by: Zervox on 2015-07-15 16:20:50
That's totally okay, after all I did say that I will publish the source if anyone wants it. Didn't just do it because I thought I would pull out next version out pretty quickly :P. It's in github though so I can just link it if needed https://github.com/Halferi/wmx2obj  Well there you have it :D, that's the same version as the compiled one though.

Well you can say that from the whole code. I never intended it to be final yet because I knew that I still have to change things for optimization. That also includes the rearranging of functions to easier upgrades in future once the program is "ready". But don't get me wrong, I do really appreciate that you do these optimization checks, also saves time from me :P.

I don't think that there is any limitation to it technically. Problems that may occur though is the performance, but testing that comes later. Each section can hold technically 0x9000 bytes which is 36864 bytes in decimals. The world map currently hardly utilizes this space fully at all, in fact I don't seem to remember any section to take over half of the maximum size.
I also haven't seen any section near that number.
I am a bit curious to the 36864 limit, I am not familiar with any data type which is capped to that range, is this due to a custom size check hardcoded into the segment read code?
Title: Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
Post by: Halfer on 2015-07-15 19:49:21
I do think that it is related to the padding which was used in psx. There's probably no reason why it should be exactly 0x9000 amount of bytes, but I think that padding was the only reason it exists. Otherwise they would probably be one after another.
Title: Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
Post by: Zervox on 2015-07-16 06:13:59
I am guessing that there is no lookup data for each block to know beforehand how much data has been encoded because of
Code: [Select]
this.faceIndexOffset = faceIndexOffset;inside block?
which is used later in Controller
Code: [Select]
int vertexIndexOffset = 0;
        int textureIndexOffset = 6;
        for(Iterator iterator = segments.iterator(); iterator.hasNext();)
        {
            Segment segment = (Segment)iterator.next();
            segment.generateBlocks(data, vertexIndexOffset, textureIndexOffset);
            vertexIndexOffset = segment.getFaceIndexOffset();
        }
Title: Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
Post by: Zervox on 2015-07-21 10:23:09
I was curious about a potential bug I found in extracting the faceIndices, is that alot of the faces would be reversed as to what they should be, you find this by using 3ds max render function, the fix I found was that in findFaceIndices, the first (eg offset+0) is actually the resulting z index and x being (+2)
Title: Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
Post by: Blue on 2015-07-30 12:23:27
Quickly updated Controller.java to avoid unnecessary loops for faster performance. I strongly recommend upgraded versions of the software apply the same changes to their source code.

wmx2obj on GitHub (https://github.com/AleksanteriHirvonen/wmx2obj)
Java Archive (JAR) (https://www.dropbox.com/s/c68udo6fla38hlt/wmx2obj.jar?dl=0)
Title: Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
Post by: Blue on 2015-08-27 14:07:53
Kudos to Zervox for pointing out run-time inefficiencies in the source code. Most, if not all, remnants of inefficiency caused by code formerly used for debugging should now be removed.

Both the repository and the direct download link are updated for now.
Title: Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
Post by: Blue on 2015-11-17 11:50:00
Before moving on to new projects, I decided to put the finishing touches on wmx2obj. I fully refactored the source code to remove known bugs and enhance the source code's readability.

The old source code was a giant chunk of iterative work and fairly complex to read. Now, the refactored software should be both efficient and easily readable.

wmx2obj on GitHub (https://github.com/AleksanteriHirvonen/wmx2obj)
Java Archive (JAR) (https://www.dropbox.com/s/c68udo6fla38hlt/wmx2obj.jar?dl=0)

Title: Re: [FF8] World map conversion tool - wmx2obj (2015-11-17)
Post by: Mcindus on 2015-11-17 18:21:40
Before moving on to new projects, I decided to put the finishing touches on wmx2obj. I fully refactored the source code to remove known bugs and enhance the source code's readability.

The old source code was a giant chunk of iterative work and fairly complex to read. Now, the refactored software should be both efficient and easily readable.

wmx2obj on GitHub (https://github.com/AleksanteriHirvonen/wmx2obj)
Java Archive (JAR) (https://www.dropbox.com/s/c68udo6fla38hlt/wmx2obj.jar?dl=0)



Alright... time to give this a whirl.   8-)

Quick question - is there still supposed to be a .mtl file?  I can't seem to find it.
Title: Re: [FF8] World map conversion tool - wmx2obj (2015-11-17)
Post by: Blue on 2015-11-17 19:45:14
Quick question - is there still supposed to be a .mtl file?  I can't seem to find it.

Not in the program I wrote. Halfer is working on an upgraded version of wmx2obj with material and texture support.