Author Topic: [PSX/PC] World map conversion tool - wmx2obj (2015-11-17)  (Read 31854 times)

Halfer

  • *
  • Posts: 142
    • View Profile
Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
« Reply #25 on: 2015-07-13 10:13:39 »
I'll make that happen in new version. However would you really prefer if each segment was separate .obj file? I mean isn't it frustrating to import each file? If not then I can make a setting to change these properties so you can choose if you want it stitched or separated.

Zervox

  • *
  • Posts: 55
    • View Profile
Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
« Reply #26 on: 2015-07-13 11:15:26 »
My code already names the segments in the obj file with corresponding integer id,so that 3ds max will import them as seperate mesh parts.
regarding saving an area to separate .obj files, it probably isn't a good idea, already with segments alone they make about 1 part per area, meaning a total number of 835 .obj files.

Also I think we should do

Code: [Select]
iterate segments {
Segment seg = segmentiterator;
write("g "+seg.getID())
iterate  blocks(j, seg.getBlocks()) {
               Blocks b = blockiterator;
                writeVertices(b, printWriter);
                writeTextures(b, printWriter);
                writeFaces(b, printWriter);
to save some processing time removing uneeded for loops, I noticed atleast for me that the textured version takes FOREVER to finish, reducing the amount of loops would probably help alot.
« Last Edit: 2015-07-13 11:30:38 by Zervox »

kaspar01

  • *
  • Posts: 118
  • FFVIII Fan & Collector , 3D Artist , FF8RP-WIP
    • View Profile
Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
« Reply #27 on: 2015-07-13 11:26:01 »
I'll make that happen in new version. However would you really prefer if each segment was separate .obj file? I mean isn't it frustrating to import each file? If not then I can make a setting to change these properties so you can choose if you want it stitched or separated.

As Zervox said that's not making 1 file for each part but make 1 file containing multiple mesh.

Actually having separated file could be a third option so that we have a choice:

-1 file 1 mesh (current)
-1 file multiple mesh (Zervox's one)
-multiple files

My code already names the segments in the obj file with corresponding integer id,so that 3ds max will import them as seperate mesh parts.

Very nice!
I'd like to test it as soon as possible.

Halfer

  • *
  • Posts: 142
    • View Profile
Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
« Reply #28 on: 2015-07-13 11:59:20 »
As Zervox said that's not making 1 file for each part but make 1 file containing multiple mesh.

Actually having separated file could be a third option so that we have a choice:

-1 file 1 mesh (current)
-1 file multiple mesh (Zervox's one)
-multiple files

Very nice!
I'd like to test it as soon as possible.


Yeah that's wiser, I'll make that an option.

to save some processing time removing uneeded for loops, I noticed atleast for me that the textured version takes FOREVER to finish, reducing the amount of loops would probably help alot.

Yeah it is taking time to finish right now because it's not really optimized yet, right now it just writes every texture indices to the file which isn't really optimal. However I'll optimize it to only write specific indices only once and make the faces to refer those which will speed up the exporting process and also reduces the lag seen in 3d editing programs.

Zervox

  • *
  • Posts: 55
    • View Profile
Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
« Reply #29 on: 2015-07-13 13:28:14 »
Ok so, tested doing blocks outside then it occured to me the way it is built that obviously you can't preloop blocks because a segment has multiple blocks which needs to be written in correct order else the retriangulate of 3ds max will destroy 90% of the vertices. but doing the segment then passing the looped segment into the write functions still works which would cut down the loops from 2 505 to 835 for the outer segment loop alone also it would reduce the complexity of the code allowing compiled code to execute better.

the texture indices would also help alot in this regard, another thing I found atleast when did the wavefront in C++ as the java version used to much time for impatient me to see, is that materials are written multiple times even within the same segment, so some sort of lookup regarding material type to make each block sort printed faces by material so that instead of

Code: [Select]
usemtl 9
f 390519//1550113 390518//1550114 390524//1550115
usemtl 1
f 390517//1550116 390530//1550117 390499//1550118
f 390491//1550119 390517//1550120 390492//1550121
usemtl 9
f 390523//1550122 390512//1550123 390519//1550124
usemtl 8
f 390516//1550125 390512//1550126 390523//1550127
usemtl 9
f 390521//1550128 390522//1550129 390517//1550130
f 390520//1550131 390521//1550132 390491//1550133
f 390497//1550134 390495//1550135 390521//1550136
f 390506//1550137 390497//1550138 390520//1550139
f 390518//1550140 390520//1550141 390490//1550142
f 390512//1550143 390506//1550144 390518//1550145
usemtl 0
f 390517//1550146 390529//1550147 390532//1550148
usemtl 8
f 390515//1550149 390511//1550150 390516//1550151
f 390514//1550152 390510//1550153 390515//1550154
f 390514//1550155 390508//1550156 390509//1550157
f 390505//1550158 390511//1550159 390510//1550160
f 390503//1550161 390510//1550162 390509//1550163
f 390508//1550164 390503//1550165 390509//1550166
f 390496//1550167 390505//1550168 390503//1550169
usemtl water
f 390502//1550170 390498//1550171 390501//1550172
usemtl 9
f 390490//1550173 390491//1550174 390493//1550175

it would be

Code: [Select]
usemtl 0
f 390517//1550146 390529//1550147 390532//1550148
usemtl 1
f 390517//1550116 390530//1550117 390499//1550118
f 390491//1550119 390517//1550120 390492//1550121
usemtl 8
f 390516//1550125 390512//1550126 390523//1550127
f 390515//1550149 390511//1550150 390516//1550151
f 390514//1550152 390510//1550153 390515//1550154
f 390514//1550155 390508//1550156 390509//1550157
f 390505//1550158 390511//1550159 390510//1550160
f 390503//1550161 390510//1550162 390509//1550163
f 390508//1550164 390503//1550165 390509//1550166
f 390496//1550167 390505//1550168 390503//1550169
usemtl 9
f 390519//1550113 390518//1550114 390524//1550115
f 390523//1550122 390512//1550123 390519//1550124
f 390521//1550128 390522//1550129 390517//1550130
f 390520//1550131 390521//1550132 390491//1550133
f 390497//1550134 390495//1550135 390521//1550136
f 390506//1550137 390497//1550138 390520//1550139
f 390518//1550140 390520//1550141 390490//1550142
f 390512//1550143 390506//1550144 390518//1550145
f 390490//1550173 390491//1550174 390493//1550175
usemtl water
f 390502//1550170 390498//1550171 390501//1550172

Hope I am not butting in on too much for you Halfer. ;)
« Last Edit: 2015-07-13 13:34:52 by Zervox »

Halfer

  • *
  • Posts: 142
    • View Profile
Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
« Reply #30 on: 2015-07-13 14:12:10 »
No no, that's really good that you point these things out :D! Hadn't really thought about that material thing yet but yes, that sounds like the most optimal way to do it, thanks for the tip! :)

Ok so, tested doing blocks outside then it occured to me the way it is built that obviously you can't preloop blocks because a segment has multiple blocks which needs to be written in correct order else the retriangulate of 3ds max will destroy 90% of the vertices. but doing the segment then passing the looped segment into the write functions still works which would cut down the loops from 2 505 to 835 for the outer segment loop alone also it would reduce the complexity of the code allowing compiled code to execute better.

Can you open this up a bit? I have troubles understanding what you are meaning with this.

Zervox

  • *
  • Posts: 55
    • View Profile
Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
« Reply #31 on: 2015-07-13 14:20:17 »
Ah, yeah, I expected it would probably be hard to follow due to crappy writing on my part.
Rewrite:

I first suggested doing
Code: [Select]
for loop (segments){
    WriteVertices(segments[i],PrintWriter);
    WriteTextures(segments[i],PrintWriter);
    WriteFaces(segments[i],PrintWriter);
}
doing a loop over segments only once is 835 iterations max, as the range of world chunks is a total of 835 due to the interchangable parts.
the  way it done now is
Code: [Select]
WriteVertices(PrintWriter);
WriteTextures(PrintWriter);
WriteFaces(PrintWriter);
Which loops segments 3 times over(one for each function call), so a total of 2505 loops, since these are seperate, each loop inside a loop adds complexity,
so being able to reduce amount of loops from 3 to 1 on first stage would reduce conversion rate noticably due to code complexity reduction(compiler wise).

I later suggested doing
Code: [Select]
for loop (segments){
   for loop(blocks)
    WriteVertices(blocks[j],PrintWriter);
    WriteTextures(blocks[j],PrintWriter);
    WriteFaces(blocks[j],PrintWriter);
}
Which didn't work as I wanted it to as it would make 3ds max tear the model to pieces(mangled mesh) on import, because the indices etc didn't match the segment.
« Last Edit: 2015-07-13 14:41:38 by Zervox »

Halfer

  • *
  • Posts: 142
    • View Profile
Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
« Reply #32 on: 2015-07-13 14:40:30 »
Thanks for this tip! That sure will release a lot of load during exporting. I will rearrange the code that way since I have to do some parts again after all.

Zervox

  • *
  • Posts: 55
    • View Profile
Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
« Reply #33 on: 2015-07-14 05:58:22 »
Also found in the block structure
Code: [Select]
public void generateFaces(byte[] data, int vertexIndexOffset) {
        // initialize face instances
        for (int face = 0; face < faceCount; face++) {
            addFace(new Face(face));
        }
       
        // find vertex indices of the face instances
        for (Face face : faces) {
            face.findVertexIndices(
                    data,
                    offset + face.getId() * FACE_BYTES,
                    vertexIndexOffset
            );
        }
    }
   
    public void generateVertices(byte[] data, int segmentX, int segmentZ) {
        // initialize vertex instances
        for (int vertex = 0; vertex < vertexCount; vertex++) {
            addVertex(new Vertex(vertex));
        }
       
        // find vertex coordinates of the vertex instances
        for (Vertex vertex : vertices) {
            vertex.findCoordinates(
                    data,
                    offset + faceCount * FACE_BYTES
                            + vertex.getId() * VERTEX_BYTES,
                    offsetX,
                    offsetZ,
                    segmentX,
                    segmentZ
            );
        }
    }
in C++ containers similar to ArrayList in java allows
Code: [Select]
public void generateFaces(byte[] data, int vertexIndexOffset) {
        // find vertex indices of the face instances
        for (int face = 0; face < faceCount; face++) {
            Face ref=faces.add(new Face(face));
            ref.findVertexIndices(
                    data,
                    offset + ref.getId() * FACE_BYTES,
                    vertexIndexOffset
            );
        }
    }
   
    public void generateVertices(byte[] data, int segmentX, int segmentZ) {
        // initialize vertex instances
        // find vertex coordinates of the vertex instances
        for (Vertex vertex : vertices) {
             Vertex ref=vertices.add(new Vertex(vertex));
            ref.findCoordinates(
                    data,
                    offset + faceCount * FACE_BYTES
                            + ref.getId() * VERTEX_BYTES,
                    offsetX,
                    offsetZ,
                    segmentX,
                    segmentZ
            );
        }
    }

Is something like this not possible in Java?
if it is, it would save alot of additional loop complexity.
it would still be possible to just access it straight after inserting it(as add(object) is put into the end) by doing
Code: [Select]
Face ref=faces.get(face);I'd still say the addVertex and addFace amongst other functions
does not need their
Code: [Select]
public void addFace(Face face) {
        if (faces.size() < faceCount) {
            faces.add(face);
        }
    }
   
    public void addVertex(Vertex vertex) {
        if (vertices.size() < vertexCount) {
            vertices.add(vertex);
        }
    }
ArrayList size checks, as the faceCount and vertexCount loop is always the same, meaning these branching if statements are just wasting CPU cycles.

Zervox

  • *
  • Posts: 55
    • View Profile
Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
« Reply #34 on: 2015-07-15 03:17:00 »
Had some fun exporting the mesh from 3ds max to a heightmap ;D


Zervox

  • *
  • Posts: 55
    • View Profile
Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
« Reply #35 on: 2015-07-15 09:33:36 »
For Faces I noticed it really only ever uses one texturePage in the current code(if this is correct)
we should make
Code: [Select]
ArrayList<String> texturepages;
Code: [Select]
String texturepage;and do
Code: [Select]
String getPage(){return texturepage;}replace all
Code: [Select]
texturepages.add("***");with
Code: [Select]
texturepage="***";
Then do a custom compare( I don't know java enough for this)
Code: [Select]
CustomComparator {
    public boolean compare(Face a, Face b) {
        return a.texturepage.before(b.texturepage);
    }
}
Then we can after the loop in GenerateFaces do
Code: [Select]
faces.sort(CustomComparator);

Halfer

  • *
  • Posts: 142
    • View Profile
Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
« Reply #36 on: 2015-07-15 10:23:02 »
Well it does use 18 texture pages right now but the way the code adds them to ArrayList is that even if the next texture page is the same as last one it adds it to the ArrayList. To minimize the memory usage it should be done that it only adds the next texture page when the value changes. I think this is what you mean right?

Oh and the heightmap looks cool :P

Zervox

  • *
  • Posts: 55
    • View Profile
Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
« Reply #37 on: 2015-07-15 11:01:03 »
From what I saw in Blocks generate faces
Code: [Select]
void findTexturePage(byte data[], int offset, int faceIndexOffset, int textureIndexOffset)is only called once
and all the if checks in findTexturePage are checked at once, meaning if one succeds the others won't get added so it is technically only one texture page in Face class active as texturepages always returns 1 in size to me.
Also due to
Code: [Select]
public String getPage()
    {
        return (String)texturepages.get(texturepages.size() - 1);
    }
it still just sends the last created one, which means all other data stored in faces are of no use. or is this just for debugging code and that code will be changed?

at write out I just did in C++ that all the Texture indices that are equal got written out together once per block, this reduced the total size of the file from 80MB to 38.6MB which is quite some savings, it did however add a bit of processing time.

also a change I'd like to suggest is moving findTextureIndices to Block.
considering TextureCoords is only x and y , this way you can get away with Dbl x and Dbl y on textures. although it would write out 3xtextures per fetch, it is however alot easier for writing a code sort for texture coordinates. eg

in Texture
Code: [Select]
public class Texture
{

    public Texture(int id,double X,double Y)
    {
        this.id = id;
        this.X=X;
        this.Y=Y;
    }

    public int getId()
    {
        return id;
    }

    public double getX()
    {
        return X;
    }
   public double getY()
    {
        return Y;
    }

    public int getPage()
    {
        return ((Integer)texturepages.get(texturepages.size() - 1)).intValue();
    }

    public void print()
    {
        System.out.print(texturepages);
    }

    public static final int COORDINATE_BYTES = 1;
    public static final int p = 0;
    private final int id;
    private double X;
    private double Y;
    private final ArrayList texturepages = new ArrayList();
}
in Block
Code: [Select]
public void findTextureIndices(byte data[], int offset)
    {
        textures.add(new Texture(textures.size(),(data[offset + 6] & 0xff) + 0.5D) / 256D,(data[offset + 7] & 0xff) + 0.5D) / 256D)
        textures.add(new Texture(textures.size(),(data[offset + 8] & 0xff) + 0.5D) / 256D,(data[offset + 9] & 0xff) + 0.5D) / 256D)
        textures.add(new Texture(textures.size(),(data[offset + 10] & 0xff) + 0.5D) / 256D,(data[offset + 11] & 0xff) + 0.5D) / 256D)
    }
 public void generateTextures(byte data[], int faceIndexOffset, int textureIndexOffset)
    {
        for(int texture = 0; texture < faceCount; texture++) findTextureIndices(data, offset + texture * 16))
    }


by the way if you are wondering how I got your code changes is because I was too lazy to ask for source and decompiled your code, sorry about that. ;)


also, don't know where I read it, but someone somewhere said there was a limitation to how many faces or vertices can be on each terrain chunk, which I can't really see, a pure water tile uses 512 faces, 400 vertices, but the chunk with balamb town for example uses 790 vertices and 1156 faces, which suggests to me that it should be possible to put more detail onto the FF8 world.
« Last Edit: 2015-07-15 11:20:51 by Zervox »

kaspar01

  • *
  • Posts: 118
  • FFVIII Fan & Collector , 3D Artist , FF8RP-WIP
    • View Profile
Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
« Reply #38 on: 2015-07-15 11:25:41 »
..by the way if you are wondering how I got your code changes is because I was too lazy to ask for source and decompiled your code, sorry about that. ;)

Yeah.. like many others here do what they do for the same reason.. because they are too lazy to ask Square for source code  :-D :-D :-D

Zervox

  • *
  • Posts: 55
    • View Profile
Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
« Reply #39 on: 2015-07-15 11:27:08 »
Yeah.. like many others here do what they do for the same reason.. because they are too lazy to ask Square for source code  :-D :-D :-D

Hah! Made me chuckle.  ;D

Halfer

  • *
  • Posts: 142
    • View Profile
Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
« Reply #40 on: 2015-07-15 15:02:07 »
Quote
by the way if you are wondering how I got your code changes is because I was too lazy to ask for source and decompiled your code, sorry about that. ;)

That's totally okay, after all I did say that I will publish the source if anyone wants it. Didn't just do it because I thought I would pull out next version out pretty quickly :P. It's in github though so I can just link it if needed https://github.com/Halferi/wmx2obj  Well there you have it :D, that's the same version as the compiled one though.

Quote
it still just sends the last created one, which means all other data stored in faces are of no use. or is this just for debugging code and that code will be changed?

Well you can say that from the whole code. I never intended it to be final yet because I knew that I still have to change things for optimization. That also includes the rearranging of functions to easier upgrades in future once the program is "ready". But don't get me wrong, I do really appreciate that you do these optimization checks, also saves time from me :P.

Quote
also, don't know where I read it, but someone somewhere said there was a limitation to how many faces or vertices can be on each terrain chunk, which I can't really see, a pure water tile uses 512 faces, 400 vertices, but the chunk with balamb town for example uses 790 vertices and 1156 faces, which suggests to me that it should be possible to put more detail onto the FF8 world.

I don't think that there is any limitation to it technically. Problems that may occur though is the performance, but testing that comes later. Each section can hold technically 0x9000 bytes which is 36864 bytes in decimals. The world map currently hardly utilizes this space fully at all, in fact I don't seem to remember any section to take over half of the maximum size.

Zervox

  • *
  • Posts: 55
    • View Profile
Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
« Reply #41 on: 2015-07-15 16:20:50 »
That's totally okay, after all I did say that I will publish the source if anyone wants it. Didn't just do it because I thought I would pull out next version out pretty quickly :P. It's in github though so I can just link it if needed https://github.com/Halferi/wmx2obj  Well there you have it :D, that's the same version as the compiled one though.

Well you can say that from the whole code. I never intended it to be final yet because I knew that I still have to change things for optimization. That also includes the rearranging of functions to easier upgrades in future once the program is "ready". But don't get me wrong, I do really appreciate that you do these optimization checks, also saves time from me :P.

I don't think that there is any limitation to it technically. Problems that may occur though is the performance, but testing that comes later. Each section can hold technically 0x9000 bytes which is 36864 bytes in decimals. The world map currently hardly utilizes this space fully at all, in fact I don't seem to remember any section to take over half of the maximum size.
I also haven't seen any section near that number.
I am a bit curious to the 36864 limit, I am not familiar with any data type which is capped to that range, is this due to a custom size check hardcoded into the segment read code?

Halfer

  • *
  • Posts: 142
    • View Profile
Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
« Reply #42 on: 2015-07-15 19:49:21 »
I do think that it is related to the padding which was used in psx. There's probably no reason why it should be exactly 0x9000 amount of bytes, but I think that padding was the only reason it exists. Otherwise they would probably be one after another.

Zervox

  • *
  • Posts: 55
    • View Profile
Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
« Reply #43 on: 2015-07-16 06:13:59 »
I am guessing that there is no lookup data for each block to know beforehand how much data has been encoded because of
Code: [Select]
this.faceIndexOffset = faceIndexOffset;inside block?
which is used later in Controller
Code: [Select]
int vertexIndexOffset = 0;
        int textureIndexOffset = 6;
        for(Iterator iterator = segments.iterator(); iterator.hasNext();)
        {
            Segment segment = (Segment)iterator.next();
            segment.generateBlocks(data, vertexIndexOffset, textureIndexOffset);
            vertexIndexOffset = segment.getFaceIndexOffset();
        }

Zervox

  • *
  • Posts: 55
    • View Profile
Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
« Reply #44 on: 2015-07-21 10:23:09 »
I was curious about a potential bug I found in extracting the faceIndices, is that alot of the faces would be reversed as to what they should be, you find this by using 3ds max render function, the fix I found was that in findFaceIndices, the first (eg offset+0) is actually the resulting z index and x being (+2)

Blue

  • *
  • Posts: 12
    • View Profile
Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
« Reply #45 on: 2015-07-30 12:23:27 »
Quickly updated Controller.java to avoid unnecessary loops for faster performance. I strongly recommend upgraded versions of the software apply the same changes to their source code.

wmx2obj on GitHub
Java Archive (JAR)

Blue

  • *
  • Posts: 12
    • View Profile
Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
« Reply #46 on: 2015-08-27 14:07:53 »
Kudos to Zervox for pointing out run-time inefficiencies in the source code. Most, if not all, remnants of inefficiency caused by code formerly used for debugging should now be removed.

Both the repository and the direct download link are updated for now.

Blue

  • *
  • Posts: 12
    • View Profile
Re: [FF8] World map conversion tool - wmx2obj (2015-07-06)
« Reply #47 on: 2015-11-17 11:50:00 »
Before moving on to new projects, I decided to put the finishing touches on wmx2obj. I fully refactored the source code to remove known bugs and enhance the source code's readability.

The old source code was a giant chunk of iterative work and fairly complex to read. Now, the refactored software should be both efficient and easily readable.

wmx2obj on GitHub
Java Archive (JAR)


Mcindus

  • *
  • Posts: 929
  • Artist, Modder, Musician.
    • View Profile
    • Lunatic Pandora
Re: [FF8] World map conversion tool - wmx2obj (2015-11-17)
« Reply #48 on: 2015-11-17 18:21:40 »
Before moving on to new projects, I decided to put the finishing touches on wmx2obj. I fully refactored the source code to remove known bugs and enhance the source code's readability.

The old source code was a giant chunk of iterative work and fairly complex to read. Now, the refactored software should be both efficient and easily readable.

wmx2obj on GitHub
Java Archive (JAR)



Alright... time to give this a whirl.   8-)

Quick question - is there still supposed to be a .mtl file?  I can't seem to find it.
« Last Edit: 2015-11-17 18:43:20 by Mcindus »

Blue

  • *
  • Posts: 12
    • View Profile
Re: [FF8] World map conversion tool - wmx2obj (2015-11-17)
« Reply #49 on: 2015-11-17 19:45:14 »
Quick question - is there still supposed to be a .mtl file?  I can't seem to find it.

Not in the program I wrote. Halfer is working on an upgraded version of wmx2obj with material and texture support.