Author Topic: [FF8 steam]Character importer  (Read 19879 times)

Shunsq

  • *
  • Posts: 142
  • 20 years to mod a game, that's insanely long
    • View Profile
[FF8 steam]Character importer
« on: 2016-01-28 14:01:25 »
Hi everyone,
I'm trying to get the bone positions from the chara.one file , present in each field archive. The wiki states that the model data is lzs compressed but i can't figure out how to decompress the file.
I've read about LZS compression and wrote a program to decompress chara.one.
It is based on this:
http://forums.qhimm.com/index.php?topic=2023.msg28900#msg28900

Anyway all i obtain is garbage. Or maybe it's not looking like what i'm expecting: isn't it supposed to look like the .MCH file,with number of bones, vertices , etc... in the header?

This is how my code works:
Code: [Select]
typedef struct ALONE ALONE;
struct ALONE
{
    long long int one_offset;
    long int one_datasize;//appears twice
    long long int has_tim;//if >0xd0 then no tim_offset and directly model data offset( which is 0)
    long long int tim_offset;//0 if has_tim>0xd0
    long long int alone_offset;//0 if has_tim>0xd0
    char name[9];
};

//Reads the header then creates a list of ALONE structures ( one for each character of the chara.one)
//I've not included the code here

//---LZS Decompression---//I create a .alone file for each character "i"
for(i=0; i<alone_count; i++)
    {
        sprintf(outputpath,"%s%s.alone",outdir_name,alone_list[i].name);
        FILE*alone=fopen(outputpath,"ab+");

        fseek(inputfile,alone_list[i].one_offset,SEEK_SET);

        memset(outputpath,'\0',255);
        int ctrl_bit[8];//control byte before the compressed data
        char comp[2];//the two bytes of the compressed data
        int comp_len=0;
        int comp_offset=0;
        long int new_pos=0;
        long int last_pos=0;

        memset(ctrl_bit,0,8*sizeof(int));
        memset(comp,'\0',2);

        while(ftell(inputfile)<alone_list[i].one_offset+alone_list[i].one_datasize)
        {
            fscanf(inputfile,"%c",&cbyte);
            convert_to_bits(cbyte,ctrl_bit);//Reference byte

            for(j=0; j<8; j++)
            {
                if(ctrl_bit[7-j]==0)//compressed data
                {
                    for(k=0; k<2; k++)
                    {
                        fscanf(inputfile,"%c",&cbyte);
                        comp[k]=cbyte;//read from left to right
                    }

                    comp_len=comp[1]%0x10+3;//minimum is 3bytes
                    comp_offset=comp[0]+(comp[1]-comp[1]%0x10)*0x10;//if the two byte are read from left to right 0xa4 0xb1 then comp_offset=0xba4;
                    new_pos=ftell(alone)-(ftell(alone)-18-comp_offset)%0x1000;//reference byte pos

                    k=0;
                    if(new_pos<0)
                    {
                        for(k=0;k<comp_len;k++)
                        {
                            fprintf(alone,"%c",NULL);
                        }
                    }
                    else if(new_pos>0)
                    {
                        fseek(alone,new_pos,SEEK_SET);
                        for(k=0;k<comp_len;k++)
                        {
                            fscanf(alone,"%c",&cbyte);//reads at comp_offset
                            fprintf(alone,"%c",cbyte);
                        }
                    }

                }
                else if(ctrl_bit[7-j]==1)//literal data
                {
                    cbyte=0;
                    fscanf(inputfile,"%c",&cbyte);
                    fprintf(alone,"%c",cbyte);
                }
            }
        }
        fclose(alone);
    }
    fclose(inputfile);
« Last Edit: 2016-02-13 17:48:44 by Shunsq »

Vehek

  • *
  • Posts: 215
    • View Profile
Re: [FF8 steam]Chara.one and lzs compression
« Reply #1 on: 2016-01-28 16:55:49 »
The wiki also mentions Playstation version when mentioning the compression. As in, only the Playstation version. You don't need to decompress further in the PC version.

Shunsq

  • *
  • Posts: 142
  • 20 years to mod a game, that's insanely long
    • View Profile
Re: [FF8 steam]Chara.one and lzs compression
« Reply #2 on: 2016-01-28 17:08:28 »
When i read the chara.one file, i don't see any similarity with the .MCH.That's why i thought it was compressed.
How did you proceed Vehek?

EDIT: I think i've found something that was not mentionned anywhere.
In chara.one, for main characters, there is only animation data. The first 4 bytes are likely to be animation count, frame count and bone count. It's exactly what Vehek wrote about animations here:http://forums.qhimm.com/index.php?topic=13261.0

On a side note why Vehek information is not in the wiki?




« Last Edit: 2016-01-28 17:28:50 by Shunsq »

Halfer

  • *
  • Posts: 142
    • View Profile
Re: [FF8 steam]Chara.one and lzs compression
« Reply #3 on: 2016-01-29 09:45:35 »
http://wiki.qhimm.com/view/FF8/FileFormat_ONE

It is on wiki, but i'm not sure how exact the information is.

A little reminder, world map chara.one seems to be completely different from field map chara.one

sithlord48

  • *
  • Posts: 1632
  • Dark Lord of the Savegame
    • View Profile
    • Blackchocobo
Re: [FF8 steam]Chara.one and lzs compression
« Reply #4 on: 2016-01-29 13:29:07 »
you can check you programs output by comparing it to a known working LZS decompressor..  such as the one myst6re has provided for ff7tk (he also wrote the wiki entry linked above) that should help you figure out if the file is different or your just decompressing it incorrectly. i added a compression tool to the ff7tk demo program but it currently only is using the LGP class. You would need to either mod it to use LZS class or just use the source as a reference for your code.

https://github.com/sithlord48/ff7tk/blob/master/utils/LZS.h
https://github.com/sithlord48/ff7tk/blob/master/utils/LZS.cpp


Shunsq

  • *
  • Posts: 142
  • 20 years to mod a game, that's insanely long
    • View Profile
Re: [FF8 steam]Chara.one and lzs compression
« Reply #5 on: 2016-01-29 16:02:50 »
Thank you sithlord48 but as Vehek said chara.one is not compressed; so i was doing something useless. Anyway it may be useful later if i want to compress an archive.
I think that the issue with chara.one is solved

Shunsq

  • *
  • Posts: 142
  • 20 years to mod a game, that's insanely long
    • View Profile
Re: Character importer
« Reply #6 on: 2016-02-13 17:48:11 »
HI everyone,
I managed to import a new model of Selphie in the game.
There are still many issues: the normals revert for some faces, there are no new textures   otherwise i need to include the possibility to convert PNG to TIM textures  and consequently the UVs need to match the original UVs. And finally the engine downscale my models resolution: if two vertices are too close they are merged ingame. A solution would be to multiply the size of my models, but the surrounding would need to be resized too :o. And there is another issue with resizing the models: the model position from floor is coded in each chara.one( one for each field) so i would need to change the value in each chara.one!



Shunsq

  • *
  • Posts: 142
  • 20 years to mod a game, that's insanely long
    • View Profile
Re: [FF8 steam]Character importer
« Reply #7 on: 2016-02-15 03:20:16 »
OK, yesterday i found a way to import an higher resolution model in the game 8-):
-I export my model 16 times bigger.
-I wrote an auto-assemble script in cheat engine that check if the function that reads the vertices is reading my Selphie model
-If it is Selphie, then divide by 16 .

Consequently the real model is really detailled but it appears smaller on the screen.


Shunsq

  • *
  • Posts: 142
  • 20 years to mod a game, that's insanely long
    • View Profile
Re: [FF8 steam]Character importer
« Reply #8 on: 2016-02-29 21:59:14 »
I managed to import new textures, i coded a built-in tim importer-exporter. My main problem is that i can't import a texture that uses more quadrants( the texture is 256*256 divided in 4 quadrants) than the original:
-1024*1024 textures crash the game
-512*512 textures are spread between the field textures and the model texture.
-256*256 ->OK

Shunsq

  • *
  • Posts: 142
  • 20 years to mod a game, that's insanely long
    • View Profile
Re: [FF8 steam]Character importer
« Reply #9 on: 2016-07-10 09:27:34 »
Hi guys,
I'm back on this program. The original program is coded in C but it is not easy to import animation data in Blender. So i translated it in python and lauch it directly from Blender. I can't find a way to import correctly the skeleton data in .MCH. For those who have already tried, how do you read the animation data ?
Vehek said years ago that it was 6 bytes data per bone, but how to you convert this in degrees or radians ?
I supposed that it was read like this(in blender python):
Code: [Select]
#Seek the animation data in the file. From the anim offset we skip 12 bytes. 2 for the animation count, 2 for the frame count,2 for the bone count and 6 for the coordinate offset.

mchfile.seek(header.ModelAddress+header.AnimOffset+2+10,0)

# read 2 bytes from mchfile for each rotation
rotX=int.from_bytes(mchfile.read(2), byteorder='little')
rotY=int.from_bytes(mchfile.read(2), byteorder='little')
rotZ=int.from_bytes(mchfile.read(2), byteorder='little')

#convert to float angle in radians. I assumed the maximum angle was 0x1000.
rotX_rad=math.radians(rotX*360/0x1000)
rotY_rad=math.radians(rotY*360/0x1000)
rotZ_rad=math.radians(rotZ*360/0x1000)

#Create a Euler rotation matrix then rotate the Unit vector by this matrix.This gives me the bone direction vector.
eul=Euler((rotX_rad, rotY_rad, rotZ_rad),'XYZ')

Unit=Vector((0,0,1))
Unit.rotate(eul)


But this gave me :

Vehek

  • *
  • Posts: 215
    • View Profile
Re: [FF8 steam]Character importer
« Reply #10 on: 2016-07-10 22:54:29 »
I did briefly mention back then that I don't load them in XYZ order. As I said there, I read them in ZXY, and used YXZ as the rotation mode. I don't think I've tried things with direct vector rotations in a long time; I've been using poses. I'm not sure how much will be directly applicable since I did things in my script like swap around the axes for the vertex coordinates to YXZ so I wouldn't have to rotate the model and to get left-right correct instead of being mirrored.

Code: [Select]
mdlPoseBones['Bone.%02d' % (rNum + 1)].rotation_mode = 'YXZ'
mdlPoseBones['Bone.%02d' % (rNum + 1)].rotation_euler = (-rots[1] * math.pi / 2048, -rots[2] * math.pi / 2048, rots[0] * math.pi / 2048)
« Last Edit: 2016-07-10 22:57:33 by Vehek »

Shunsq

  • *
  • Posts: 142
  • 20 years to mod a game, that's insanely long
    • View Profile
Re: [FF8 steam]Character importer
« Reply #11 on: 2016-07-16 08:48:19 »
Hello Vehek, thank you for your help. 3 years ago you wrote the text below about the rotation reading. Is it working? I've tried it but i don't have a correct pose.
When you create your armature in blender in edit mode, are the bones aligned with Z axis?Or do you put them in a Rest pose ?
Because for myself i draw the bones in a rest position, extracted from the .MCH file.


Animation data
Here, I'm not only repeating what's written in previous threads, I'm also repeating what I said earlier in this thread.

It doesn't have to be written this way; it's just my preference to divide it up like this.

Code: [Select]
struct
{
u16 animationCount;
Animation[animationCount];
}animationData;

struct
{
u16 frameCount;
u16 boneCount
AnimationFrame[frameCount]

}Animation

struct
{
s16 coordinatesShift[3];
Rotation[boneCount];
}AnimationFrame;

The format of the rotation data differs between the files in main_chr.fs and standard-format chara.one files. In main_chr.fs, they are 3 s16s (6 bytes) per bone.
The other format is 4 bytes per bone. I'm not sure how to explain how it works in words. Shift each one of the first 3 bytes left 2 bits and use 2 bits from the 4th byte as bits 10 and 11?

Here's a generalized example in code.
Code: [Select]
unsigned char rots[4];
s16 rotations[3];
for (int i = 0; i < 3; i++)
rotations[i] = rots[i] << 2 | (rots[3] >> 2 * i & 3) << 10;


Vehek

  • *
  • Posts: 215
    • View Profile
Re: [FF8 steam]Character importer
« Reply #12 on: 2016-07-16 16:55:25 »
I put them all along one axis (Z in my case). The "rest pose" is just an animation itself, and has no impact on other animations.
« Last Edit: 2016-07-16 17:03:23 by Vehek »

Shunsq

  • *
  • Posts: 142
  • 20 years to mod a game, that's insanely long
    • View Profile
Re: [FF8 steam]Character importer
« Reply #13 on: 2016-07-17 16:22:44 »
Hmmm...I really don't get the correct poses,even with your directives. For the rotation are you certain it is:
Code: [Select]
rotations[i]=( rots[i]<<2)|(    (rots[3]>>(2*i)) &3   )<<10)

Shunsq

  • *
  • Posts: 142
  • 20 years to mod a game, that's insanely long
    • View Profile
Re: [FF8 steam]Character importer
« Reply #14 on: 2016-07-21 19:25:09 »
So Vehek, do you agree with my previous assumption?

Shunsq

  • *
  • Posts: 142
  • 20 years to mod a game, that's insanely long
    • View Profile
Re: [FF8 steam]Character importer
« Reply #15 on: 2016-07-31 12:40:34 »
Hi everyone,
I managed to import the animations correctly, by scrubing extremely old post of Vehek on qhimm  back in 2009. See this page at reply #47:http://forums.qhimm.com/index.php?topic=8130.25.
Thank you for your researching Vehek!
My problem now is that all skeletons are not structured the same. Actually it works with Zell and Rinoa. When i try Squall his feet, upperbody and head are rotated 90degrees on Z axis. When i try Selphie all the skeleton is laid on the floor. And Seifer bone hierarchy is completly messed up ( his head is fixed on his forearm :-D)

« Last Edit: 2016-07-31 12:45:14 by Shunsq »

Shunsq

  • *
  • Posts: 142
  • 20 years to mod a game, that's insanely long
    • View Profile
Re: [FF8 steam]Character importer
« Reply #16 on: 2016-11-10 19:11:48 »
Hello fellow hackers,
I'm able to import new models but i'm limited in terms of polycount and resolution. What i mean is that for a certain surface on the screen, the FF8 engine is limiting the number of vertices shown. It merges vertices when the distance between them is under a certain threshold. My first objective is to increase this threshold, or to remove the function completly.

Then i have a second objective. As you may know the characters textures in FF8 are using only 1 or 2 quadrants of a 256x256 UV-grid. I want to use all 4 quadrants, an have any size of UV-grid. I've already looked into the .mch format and it is possible bytewise speaking. Anyway it is causing graphic issues, like textures used by wrong objects or crashing the game.

Could someone help me?

Mcindus

  • *
  • Posts: 929
  • Artist, Modder, Musician.
    • View Profile
    • Lunatic Pandora
Re: [FF8 steam]Character importer
« Reply #17 on: 2016-11-12 05:12:03 »
Hello fellow hackers,
I'm able to import new models but i'm limited in terms of polycount and resolution. What i mean is that for a certain surface on the screen, the FF8 engine is limiting the number of vertices shown. It merges vertices when the distance between them is under a certain threshold. My first objective is to increase this threshold, or to remove the function completly.

Then i have a second objective. As you may know the characters textures in FF8 are using only 1 or 2 quadrants of a 256x256 UV-grid. I want to use all 4 quadrants, an have any size of UV-grid. I've already looked into the .mch format and it is possible bytewise speaking. Anyway it is causing graphic issues, like textures used by wrong objects or crashing the game.

Could someone help me?

Try treating the textures as 128x256?? - I believe they are constructed the same way that that enemy textures are constructed.  This might help with some of the UV locations.  The files are either 128x128 or 128x256, using a top/bottom quadrant if the second, I believe.  Everything I see in the .fs files seems to be set up like this for characters, enemies, and npcs.

Shunsq

  • *
  • Posts: 142
  • 20 years to mod a game, that's insanely long
    • View Profile
Re: [FF8 steam]Character importer
« Reply #18 on: 2016-12-31 21:47:36 »
Hello everyone,
Please find below the upscaled model of Selphie (d027.mch).
You have to use deling.exe to replace the model of Selphie.
1)open deling.exe and go to import/export.
2)Open field.fs
3)Extract mainchr.fs/fl/fi on your desktop.
4)Open mainchr with Deling, you will see the list of models. Delete d027.mch and replace it with d027-new.mch. A pop-up will ask you if you want to compress the file.Say NO.
5)rename d027-new to d027.mch
6)Open field.fs again
7)delete mainchar.fs/fi/fl. Then replace them with your modified version of mainchr.fs/fi/fl saved on your desktop. A pop-up will ask you if you want to compress the files. Say yes.

https://www.dropbox.com/s/zugiq6k2xdqetxt/Selphie_upscale.zip?dl=0

I also included the obj with a texture if you want to visualize it in 3dsmax or whatever.

Mcindus

  • *
  • Posts: 929
  • Artist, Modder, Musician.
    • View Profile
    • Lunatic Pandora
Re: [FF8 steam]Character importer
« Reply #19 on: 2017-01-01 03:29:45 »
Hello everyone,
Please find below the upscaled model of Selphie (d027.mch).
You have to use deling.exe to replace the model of Selphie.
1)open deling.exe and go to import/export.
2)Open field.fs
3)Extract mainchr.fs/fl/fi on your desktop.
4)Open mainchr with Deling, you will see the list of models. Delete d027.mch and replace it with d027-new.mch. A pop-up will ask you if you want to compress the file.Say NO.
5)rename d027-new to d027.mch
6)Open field.fs again
7)delete mainchar.fs/fi/fl. Then replace them with your modified version of mainchr.fs/fi/fl saved on your desktop. A pop-up will ask you if you want to compress the files. Say yes.

https://www.dropbox.com/s/zugiq6k2xdqetxt/Selphie_upscale.zip?dl=0

I also included the obj with a texture if you want to visualize it in 3dsmax or whatever.

Screenshots????

Tsuna

  • Global moderator
  • *
  • Posts: 823
  • Working together to create awesome things!
    • View Profile
    • The home of Tsunamods
Re: [FF8 steam]Character importer
« Reply #20 on: 2017-01-01 03:53:33 »
For ff8 can you just import a whole model? I've oblyever worked ff7 stuff. Im able to get quite a few really good ff8 models if so

Shunsq

  • *
  • Posts: 142
  • 20 years to mod a game, that's insanely long
    • View Profile
Re: [FF8 steam]Character importer
« Reply #21 on: 2017-01-01 07:53:36 »
Tsunamix, today it is not possible to import whatever you want to ff8,there is no tool.That is why i'm creating this tool.What i gave you is a new model of selphie,not the original.

Tsuna

  • Global moderator
  • *
  • Posts: 823
  • Working together to create awesome things!
    • View Profile
    • The home of Tsunamods
Re: [FF8 steam]Character importer
« Reply #22 on: 2017-01-01 16:25:57 »
Yeah okay. I've never even looked at modding ff8 before but I poayes mainly because on my travels to find ff7 models I have come across many ff8 models too. If you manage to get it going then I'll point you to the ones I found to help out a little

red_evilspirit

  • *
  • Posts: 41
    • View Profile
Re: [FF8 steam]Character importer
« Reply #23 on: 2018-03-21 05:26:29 »
If you don't mind, can you share tool/source read data in mch fie?
I stuck at putting bone to right position and exporting animation.

Shunsq

  • *
  • Posts: 142
  • 20 years to mod a game, that's insanely long
    • View Profile
Re: [FF8 steam]Character importer
« Reply #24 on: 2018-03-26 16:54:10 »
Hello red_evilspirit,
I used python in blender to import mesh, create a skeleton and import animation from mch an chara.one file. I don't intend to share my code for the moment, i'm still working on it.
But if you want to do your own import-export script :
-Read description of the format in the qhimm wiki:http://wiki.ffrtt.ru/index.php/FF8/FileFormat_MCH
-Gather data on skeleton format from Vehek:http://forums.qhimm.com/index.php?topic=13261.0. Search for his posts on qhimm you will learn a lot.
-Gather data from Xentax ( xna lara forum):http://forum.xentax.com/viewtopic.php?f=16&t=10810

The mch format only holds mesh information and rest-pose data for one character. In each field you have a chara.one file which holds the information about animation for main characters and mesh+animation for NPC.

MCH format for skeleton is not the same as in the chara.one format. You need both files to rebuild the animation. The MCH format holds the length of the bones. The charaone holds the rotations of the bones.