Qhimm.com Forums
Miscellaneous Forums => Scripting and Reverse Engineering => Topic started by: Shunsq on 2016-01-28 14:01:25
-
Hi everyone,
I'm trying to get the bone positions from the chara.one file , present in each field archive. The wiki states that the model data is lzs compressed but i can't figure out how to decompress the file.
I've read about LZS compression and wrote a program to decompress chara.one.
It is based on this:
http://forums.qhimm.com/index.php?topic=2023.msg28900#msg28900 (http://forums.qhimm.com/index.php?topic=2023.msg28900#msg28900)
Anyway all i obtain is garbage. Or maybe it's not looking like what i'm expecting: isn't it supposed to look like the .MCH file,with number of bones, vertices , etc... in the header?
This is how my code works:
typedef struct ALONE ALONE;
struct ALONE
{
long long int one_offset;
long int one_datasize;//appears twice
long long int has_tim;//if >0xd0 then no tim_offset and directly model data offset( which is 0)
long long int tim_offset;//0 if has_tim>0xd0
long long int alone_offset;//0 if has_tim>0xd0
char name[9];
};
//Reads the header then creates a list of ALONE structures ( one for each character of the chara.one)
//I've not included the code here
//---LZS Decompression---//I create a .alone file for each character "i"
for(i=0; i<alone_count; i++)
{
sprintf(outputpath,"%s%s.alone",outdir_name,alone_list[i].name);
FILE*alone=fopen(outputpath,"ab+");
fseek(inputfile,alone_list[i].one_offset,SEEK_SET);
memset(outputpath,'\0',255);
int ctrl_bit[8];//control byte before the compressed data
char comp[2];//the two bytes of the compressed data
int comp_len=0;
int comp_offset=0;
long int new_pos=0;
long int last_pos=0;
memset(ctrl_bit,0,8*sizeof(int));
memset(comp,'\0',2);
while(ftell(inputfile)<alone_list[i].one_offset+alone_list[i].one_datasize)
{
fscanf(inputfile,"%c",&cbyte);
convert_to_bits(cbyte,ctrl_bit);//Reference byte
for(j=0; j<8; j++)
{
if(ctrl_bit[7-j]==0)//compressed data
{
for(k=0; k<2; k++)
{
fscanf(inputfile,"%c",&cbyte);
comp[k]=cbyte;//read from left to right
}
comp_len=comp[1]%0x10+3;//minimum is 3bytes
comp_offset=comp[0]+(comp[1]-comp[1]%0x10)*0x10;//if the two byte are read from left to right 0xa4 0xb1 then comp_offset=0xba4;
new_pos=ftell(alone)-(ftell(alone)-18-comp_offset)%0x1000;//reference byte pos
k=0;
if(new_pos<0)
{
for(k=0;k<comp_len;k++)
{
fprintf(alone,"%c",NULL);
}
}
else if(new_pos>0)
{
fseek(alone,new_pos,SEEK_SET);
for(k=0;k<comp_len;k++)
{
fscanf(alone,"%c",&cbyte);//reads at comp_offset
fprintf(alone,"%c",cbyte);
}
}
}
else if(ctrl_bit[7-j]==1)//literal data
{
cbyte=0;
fscanf(inputfile,"%c",&cbyte);
fprintf(alone,"%c",cbyte);
}
}
}
fclose(alone);
}
fclose(inputfile);
-
The wiki also mentions Playstation version when mentioning the compression. As in, only the Playstation version. You don't need to decompress further in the PC version.
-
When i read the chara.one file, i don't see any similarity with the .MCH.That's why i thought it was compressed.
How did you proceed Vehek?
EDIT: I think i've found something that was not mentionned anywhere.
In chara.one, for main characters, there is only animation data. The first 4 bytes are likely to be animation count, frame count and bone count. It's exactly what Vehek wrote about animations here:http://forums.qhimm.com/index.php?topic=13261.0 (http://forums.qhimm.com/index.php?topic=13261.0)
On a side note why Vehek information is not in the wiki?
-
http://wiki.qhimm.com/view/FF8/FileFormat_ONE
It is on wiki, but i'm not sure how exact the information is.
A little reminder, world map chara.one seems to be completely different from field map chara.one
-
you can check you programs output by comparing it to a known working LZS decompressor.. such as the one myst6re has provided for ff7tk (he also wrote the wiki entry linked above) that should help you figure out if the file is different or your just decompressing it incorrectly. i added a compression tool to the ff7tk demo program but it currently only is using the LGP class. You would need to either mod it to use LZS class or just use the source as a reference for your code.
https://github.com/sithlord48/ff7tk/blob/master/utils/LZS.h
https://github.com/sithlord48/ff7tk/blob/master/utils/LZS.cpp
-
Thank you sithlord48 but as Vehek said chara.one is not compressed; so i was doing something useless. Anyway it may be useful later if i want to compress an archive.
I think that the issue with chara.one is solved
-
HI everyone,
I managed to import a new model of Selphie in the game.
There are still many issues: the normals revert for some faces, there are no new textures otherwise i need to include the possibility to convert PNG to TIM textures and consequently the UVs need to match the original UVs. And finally the engine downscale my models resolution: if two vertices are too close they are merged ingame. A solution would be to multiply the size of my models, but the surrounding would need to be resized too :o. And there is another issue with resizing the models: the model position from floor is coded in each chara.one( one for each field) so i would need to change the value in each chara.one!
(http://orig15.deviantart.net/b534/f/2016/044/2/9/test_import_by_shunsquall-d9rl8yk.png)
-
OK, yesterday i found a way to import an higher resolution model in the game 8-):
-I export my model 16 times bigger.
-I wrote an auto-assemble script in cheat engine that check if the function that reads the vertices is reading my Selphie model
-If it is Selphie, then divide by 16 .
Consequently the real model is really detailled but it appears smaller on the screen.
-
I managed to import new textures, i coded a built-in tim importer-exporter. My main problem is that i can't import a texture that uses more quadrants( the texture is 256*256 divided in 4 quadrants) than the original:
-1024*1024 textures crash the game
-512*512 textures are spread between the field textures and the model texture.
-256*256 ->OK
-
Hi guys,
I'm back on this program. The original program is coded in C but it is not easy to import animation data in Blender. So i translated it in python and lauch it directly from Blender. I can't find a way to import correctly the skeleton data in .MCH. For those who have already tried, how do you read the animation data ?
Vehek said years ago that it was 6 bytes data per bone, but how to you convert this in degrees or radians ?
I supposed that it was read like this(in blender python):
#Seek the animation data in the file. From the anim offset we skip 12 bytes. 2 for the animation count, 2 for the frame count,2 for the bone count and 6 for the coordinate offset.
mchfile.seek(header.ModelAddress+header.AnimOffset+2+10,0)
# read 2 bytes from mchfile for each rotation
rotX=int.from_bytes(mchfile.read(2), byteorder='little')
rotY=int.from_bytes(mchfile.read(2), byteorder='little')
rotZ=int.from_bytes(mchfile.read(2), byteorder='little')
#convert to float angle in radians. I assumed the maximum angle was 0x1000.
rotX_rad=math.radians(rotX*360/0x1000)
rotY_rad=math.radians(rotY*360/0x1000)
rotZ_rad=math.radians(rotZ*360/0x1000)
#Create a Euler rotation matrix then rotate the Unit vector by this matrix.This gives me the bone direction vector.
eul=Euler((rotX_rad, rotY_rad, rotZ_rad),'XYZ')
Unit=Vector((0,0,1))
Unit.rotate(eul)
But this gave me :
(https://drive.google.com/uc?export=view&id=0B9Tz03JlBAsHRXJrR29zbmhpcG8)
-
I did briefly mention back then that I don't load them in XYZ order. As I said there, I read them in ZXY, and used YXZ as the rotation mode. I don't think I've tried things with direct vector rotations in a long time; I've been using poses. I'm not sure how much will be directly applicable since I did things in my script like swap around the axes for the vertex coordinates to YXZ so I wouldn't have to rotate the model and to get left-right correct instead of being mirrored.
mdlPoseBones['Bone.%02d' % (rNum + 1)].rotation_mode = 'YXZ'
mdlPoseBones['Bone.%02d' % (rNum + 1)].rotation_euler = (-rots[1] * math.pi / 2048, -rots[2] * math.pi / 2048, rots[0] * math.pi / 2048)
-
Hello Vehek, thank you for your help. 3 years ago you wrote the text below about the rotation reading. Is it working? I've tried it but i don't have a correct pose.
When you create your armature in blender in edit mode, are the bones aligned with Z axis?Or do you put them in a Rest pose ?
Because for myself i draw the bones in a rest position, extracted from the .MCH file.
Animation data
Here, I'm not only repeating what's written in previous threads, I'm also repeating what I said earlier in this thread.
It doesn't have to be written this way; it's just my preference to divide it up like this.
struct
{
u16 animationCount;
Animation[animationCount];
}animationData;
struct
{
u16 frameCount;
u16 boneCount
AnimationFrame[frameCount]
}Animation
struct
{
s16 coordinatesShift[3];
Rotation[boneCount];
}AnimationFrame;
The format of the rotation data differs between the files in main_chr.fs and standard-format chara.one files. In main_chr.fs, they are 3 s16s (6 bytes) per bone.
The other format is 4 bytes per bone. I'm not sure how to explain how it works in words. Shift each one of the first 3 bytes left 2 bits and use 2 bits from the 4th byte as bits 10 and 11?
Here's a generalized example in code.
unsigned char rots[4];
s16 rotations[3];
for (int i = 0; i < 3; i++)
rotations[i] = rots[i] << 2 | (rots[3] >> 2 * i & 3) << 10;
-
I put them all along one axis (Z in my case). The "rest pose" is just an animation itself, and has no impact on other animations.
-
Hmmm...I really don't get the correct poses,even with your directives. For the rotation are you certain it is:
rotations[i]=( rots[i]<<2)|( (rots[3]>>(2*i)) &3 )<<10)
-
So Vehek, do you agree with my previous assumption?
-
Hi everyone,
I managed to import the animations correctly, by scrubing extremely old post of Vehek on qhimm back in 2009. See this page at reply #47:http://forums.qhimm.com/index.php?topic=8130.25 (http://forums.qhimm.com/index.php?topic=8130.25).
Thank you for your researching Vehek!
My problem now is that all skeletons are not structured the same. Actually it works with Zell and Rinoa. When i try Squall his feet, upperbody and head are rotated 90degrees on Z axis. When i try Selphie all the skeleton is laid on the floor. And Seifer bone hierarchy is completly messed up ( his head is fixed on his forearm :-D)
https://vimeo.com/176920453 (https://vimeo.com/176920453)
-
Hello fellow hackers,
I'm able to import new models but i'm limited in terms of polycount and resolution. What i mean is that for a certain surface on the screen, the FF8 engine is limiting the number of vertices shown. It merges vertices when the distance between them is under a certain threshold. My first objective is to increase this threshold, or to remove the function completly.
Then i have a second objective. As you may know the characters textures in FF8 are using only 1 or 2 quadrants of a 256x256 UV-grid. I want to use all 4 quadrants, an have any size of UV-grid. I've already looked into the .mch format and it is possible bytewise speaking. Anyway it is causing graphic issues, like textures used by wrong objects or crashing the game.
Could someone help me?
-
Hello fellow hackers,
I'm able to import new models but i'm limited in terms of polycount and resolution. What i mean is that for a certain surface on the screen, the FF8 engine is limiting the number of vertices shown. It merges vertices when the distance between them is under a certain threshold. My first objective is to increase this threshold, or to remove the function completly.
Then i have a second objective. As you may know the characters textures in FF8 are using only 1 or 2 quadrants of a 256x256 UV-grid. I want to use all 4 quadrants, an have any size of UV-grid. I've already looked into the .mch format and it is possible bytewise speaking. Anyway it is causing graphic issues, like textures used by wrong objects or crashing the game.
Could someone help me?
Try treating the textures as 128x256?? - I believe they are constructed the same way that that enemy textures are constructed. This might help with some of the UV locations. The files are either 128x128 or 128x256, using a top/bottom quadrant if the second, I believe. Everything I see in the .fs files seems to be set up like this for characters, enemies, and npcs.
-
Hello everyone,
Please find below the upscaled model of Selphie (d027.mch).
You have to use deling.exe to replace the model of Selphie.
1)open deling.exe and go to import/export.
2)Open field.fs
3)Extract mainchr.fs/fl/fi on your desktop.
4)Open mainchr with Deling, you will see the list of models. Delete d027.mch and replace it with d027-new.mch. A pop-up will ask you if you want to compress the file.Say NO.
5)rename d027-new to d027.mch
6)Open field.fs again
7)delete mainchar.fs/fi/fl. Then replace them with your modified version of mainchr.fs/fi/fl saved on your desktop. A pop-up will ask you if you want to compress the files. Say yes.
https://www.dropbox.com/s/zugiq6k2xdqetxt/Selphie_upscale.zip?dl=0 (https://www.dropbox.com/s/zugiq6k2xdqetxt/Selphie_upscale.zip?dl=0)
I also included the obj with a texture if you want to visualize it in 3dsmax or whatever.
-
Hello everyone,
Please find below the upscaled model of Selphie (d027.mch).
You have to use deling.exe to replace the model of Selphie.
1)open deling.exe and go to import/export.
2)Open field.fs
3)Extract mainchr.fs/fl/fi on your desktop.
4)Open mainchr with Deling, you will see the list of models. Delete d027.mch and replace it with d027-new.mch. A pop-up will ask you if you want to compress the file.Say NO.
5)rename d027-new to d027.mch
6)Open field.fs again
7)delete mainchar.fs/fi/fl. Then replace them with your modified version of mainchr.fs/fi/fl saved on your desktop. A pop-up will ask you if you want to compress the files. Say yes.
https://www.dropbox.com/s/zugiq6k2xdqetxt/Selphie_upscale.zip?dl=0 (https://www.dropbox.com/s/zugiq6k2xdqetxt/Selphie_upscale.zip?dl=0)
I also included the obj with a texture if you want to visualize it in 3dsmax or whatever.
Screenshots????
-
For ff8 can you just import a whole model? I've oblyever worked ff7 stuff. Im able to get quite a few really good ff8 models if so
-
Tsunamix, today it is not possible to import whatever you want to ff8,there is no tool.That is why i'm creating this tool.What i gave you is a new model of selphie,not the original.
-
Yeah okay. I've never even looked at modding ff8 before but I poayes mainly because on my travels to find ff7 models I have come across many ff8 models too. If you manage to get it going then I'll point you to the ones I found to help out a little
-
If you don't mind, can you share tool/source read data in mch fie?
I stuck at putting bone to right position and exporting animation.
(https://i.imgur.com/m81oNCe.png)
-
Hello red_evilspirit,
I used python in blender to import mesh, create a skeleton and import animation from mch an chara.one file. I don't intend to share my code for the moment, i'm still working on it.
But if you want to do your own import-export script :
-Read description of the format in the qhimm wiki:http://wiki.ffrtt.ru/index.php/FF8/FileFormat_MCH (http://wiki.ffrtt.ru/index.php/FF8/FileFormat_MCH)
-Gather data on skeleton format from Vehek:http://forums.qhimm.com/index.php?topic=13261.0 (http://forums.qhimm.com/index.php?topic=13261.0). Search for his posts on qhimm you will learn a lot.
-Gather data from Xentax ( xna lara forum):http://forum.xentax.com/viewtopic.php?f=16&t=10810 (http://forum.xentax.com/viewtopic.php?f=16&t=10810)
The mch format only holds mesh information and rest-pose data for one character. In each field you have a chara.one file which holds the information about animation for main characters and mesh+animation for NPC.
MCH format for skeleton is not the same as in the chara.one format. You need both files to rebuild the animation. The MCH format holds the length of the bones. The charaone holds the rotations of the bones.
-
Hey Shun! it's been a while and I see you're still on FF8 stuff (and you're doing geat as well! )
Can't wait to see how this is going to evolve..
Hopefully my newly acquired 3D skills will be of any use for creating a HD mods or something sooner or later :)
-
kaspar - you and I both!!
It would be really nice to be able to import models with more polys.
Keep it up Shunsq! I can't wait to be able to give the characters a REAL overhaul :)
-
Good to see you back kaspar!
I'm not showing anything because right now the code is a bit useless. I can import new model into the game but on screen all the vertices are messed up because of "fixed point precision" of the engine. Google the term on the net. In modern game, or even ff7 if i'm not wrong,the vertices are "floating point precision". That means that a fixed number of coordinates are allowed on screen . If my vertex is at coordinate (20.233 pix; 123.3 pix) then it will be snapped to (20 pix; 123 pix). That is the "shaky" effect of old psx games. Some emulators fix that issue (see PGXP).I would like something similar i could inject with dlpb tools.
My other issue is the size of the ram of ff8. I can increase the texture size to 256×256 (twice the actual size) but the new texture overlaps with other textures. So i tried to build a code that stores the textures somewhere else in memory. But the problem is still the same.
-
Good to see you back kaspar!
I'm not showing anything because right now the code is a bit useless. I can import new model into the game but on screen all the vertices are messed up because of "fixed point precision" of the engine. Google the term on the net. In modern game, or even ff7 if i'm not wrong,the vertices are "floating point precision". That means that a fixed number of coordinates are allowed on screen . If my vertex is at coordinate (20.233 pix; 123.3 pix) then it will be snapped to (20 pix; 123 pix). That is the "shaky" effect of old psx games. Some emulators fix that issue (see PGXP).I would like something similar i could inject with dlpb tools.
My other issue is the size of the ram of ff8. I can increase the texture size to 256×256 (twice the actual size) but the new texture overlaps with other textures. So i tried to build a code that stores the textures somewhere else in memory. But the problem is still the same.
Aali released his source - and they've been working on it and chatting about it in the FF7 dev area of discord. Aali was able to crate a modpath for the (lower resolution is icon.tex) - iconfl00.tex, iconfl01.tex, iconfl02.tex, iconfl03.tex for the english version and make an external folder path interceptor for the textures. If you could do something like this with your models, then you might be able to get the functionality you're looking for. Would Kalderasha have anything to add to this topic?
-
Not really, if I understand Shunsq correctly then the game use int var to calculate the 3D. So the function needs to be rewritten with float or double precision. I probably said this but it might be a good idea to ask Kaldaien on steam if he has an idea how to deal with this. He seems to be very competent with working on graphic interfaces and with the source of Aali's driver he may have more possibilities to hack the game.
-
I write script like this, is it right ?
bs.seek(animofs, NOESEEK_ABS)
animCount = bs.readShort()
framCount = bs.readShort()
boneCount = bs.readShort()
coordinatesOffset = bs.readBytes(6)
gido = []
for i in range(0,boneCount):
byte1 = bs.readByte()
byte2 = bs.readByte()
byte3 = bs.readByte()
byte4 = bs.readByte()
rotation1 = (byte1 << 0x02) | ((byte4 & 0x03) << 0x0A)
rotation2 = (byte2 << 0x02) | ((byte4 & 0x0C) << 0x08)
rotation3 = (byte3 << 0x02) | ((byte4 & 0x30) << 0x06)
xyz = {
"x": rotation1,
"y": rotation2,
"z": rotation3
}
gido.append(xyz)
when i dump 'xyz', all number is really big :(
-
Hello red,
Sorry for the late reaction,
Your reading program is ok for chara.one file. The rotations are written on 12 bits, which give you a max angle of 4095 for each rotation.
For some reason blender understands that 4095 equals 180 degrees. That means 2048 is 90 deg and so on.
In my code i don't do any conversion to radians or degrees. I read the raw value and apply it to the bone rotations in pose mode.
On the contrary the data is different in the mch file.
The mch only contains the bone hierarchy, their length and a T pose.
I still have big issues to find a code that creates a good T pose for all characters.Today i extract 2 bytes for each rotations then i convert to degree, then to euler angles.
Just so you know, you don't need the T pose to anim the character. That means that even if the T pose is a total mess, the bones will magically anim perfectly in posemode.
-
I thought i can get T-pose with that code.
By the way, can you show me the structure of chara.one file?
Does it like http://wiki.ffrtt.ru/index.php/FF8/FileFormat_ONE