Qhimm.com Forums
Miscellaneous Forums => Scripting and Reverse Engineering => Topic started by: ZeaLitY on 2007-10-20 00:41:29
-
With FaustWolf leading, we've been attempting to map Cross's file structure at the Chrono Compendium and rip its assets. Aside from what PSXMultiripper, PSMPlay, and VRAM viewer give us, it's been a tough process. Of course, it's been a problem that Cross seems to behave like Xenogears:
http://wiki.qhimm.com/Xeno/IndexFileFormat
There's no real file structure, so we think that Cross is using the same process (the games were released in close proximity, too). FW has been trying to find Cross's index file, and he's got it down to two candidates. Here are his notes:
It seems the disc sectors in Xenogears are 2048 bytes long, not 2352 (which I thought was the iso standard). If 2048 is the magic number, then everything seems to be shaping up with what's in the qhimm wiki reference on Xeno. That makes the index file's starting offset in Xenogears C000, BTW. Not sure what it'll be in Cross, though the PlayStation logo model is stored at the same offset in both Cross and Xeno.
Anyone familiar enough with disc sectors to tell me if 2048 sounds like a reasonable number of bytes per sector on a PlayStation CD, and if it's likely that Cross has the same number of bytes per sector?
In other news, I'm attaching a hex string that might just be the Chrono Cross table of contents. It is significantly different from its Xenogears counterpart; for one thing, it's only 13 sectors long as opposed to 16, and secondly, the byte repeat pattern* happens every four bytes as opposed to every seven bytes. But I believe it may be what we're looking for.
Still no idea on how to open it.
Relevant attachment is at this linked post (http://www.chronocompendium.com/Forums/index.php/topic,4729.msg82936.html#msg82936).
I guess this is just a request for any advice. The Chrono community isn't the most active one, so we're in short supply when it comes to experienced talent who have a better idea of what to do. We did discover that someone name Yartrebo who successfully ripped Serge's character model by hacking PeOPs, but his files are long since gone and we can't contact him. We've also sent an e-mail out to Terminus Traduction, who successfully extracted and translated the game's script (using these tools and documentations (http://chronofan.com/Black/Other/Chrono_Cross_-_Translation_tools.zip)). FW hasn't gotten around to studying their notes in depth yet.
Anyway, thanks for reading!
Edit: Oh cool, seems I've been here before...I recognize halkun's name. We added your comments on the Chrono Trigger PSX file structure to the Compendium's encyclopedia. Until your post, no one knew why that ROM was on there.
-
Edit: Oh cool, seems I've been here before...I recognize halkun's name. We added your comments on the Chrono Trigger PSX file structure to the Compendium's encyclopedia. Until your post, no one knew why that ROM was on there.
Or, it might be because I quoted Halkun's post here over at the Chrono Compendium.
Edit- (...which is likely why you visited...)
-
Usefull disk sector size is 2048. This it is readed by usual CDrom. 2352 is complete sector size on disk. It includes disc header. Size 2352 used for some videos. All usual data stored in normal 2048 part of sector and calculated according to it.
-
I feel Chrono Cross is possibly Squares best attempt at a PSX game to date ( I think it is more advanced than FF9).
But Chrono Cross was almost their last game on the PSX where as XenoGears came out almost in line after FF7.
Akari is the master of Xenogears, I think he might know alot more.
-
Okay, finally got an account here. :lol:
First of all, thank you Akari for that confirmation on the 2048 bytes per sector.
A while back, in this thread: http://forums.qhimm.com/index.php?topic=4919.0 , you posted some code that is supposed to parse the Xenogears index file. Specifically...
extract(char *filename)
{
unsigned long start_sector, file_size;
unsigned char start[3], size[4];
char numero[10];
// first directory number 0
int directory_number = 0;
// create file with cd file index
create_index_file();
index_file = fopen("index_cd.bin", "rb");
for (i = 0; ; i++)
{
file_position = fread(start, 1, 3, index_file);
file_position = fread(size, 1, 4, index_file);
// reverse bytes order (low endian)
start_sector = start[0] | (start[1] << 8) | (start[2] << 16);
file_size = size[0] | (size[1] << 8) | (size[2] << 16) | (size[3] << 24);
// end of file
if (file_size == 0x00ffffff)
{
break;
}
if (start_sector == 0 && file_size > 0)
{
filename[9] = 0;
}
// directory
if (file_size > 0xff000000)
{
// create directory name (name - number of directory)
filename[9] = 0;
sprintf(numero, "%d", directory_number++);
strcat(filename, numero);
strcat(filename, "\\");
mkdir(filename);
}
// file
if (start_sector != 0 && file_size < 0xff000000 && file_size > 0)
{
printf("file number %d, size: 0x%x byte - ", i, file_size);
parse_file(start_sector, file_size, i, filename);
}
}
}
But I can barely tell the difference between C++ and Java anymore; could some kind soul tell me how to use this code? Does it need to be compiled into an executable, perhaps? And does it simply extract the Table of Contents data from the CD, or will it convert the data into something that says where the archives and other files are located on the CD?
Also, if this works for Xenogears, could it work for Chrono Cross' index file/Table of Contents as well, or would it need to be modified?
Thanks! :-D
-
From what I can tell, this code takes the image file for Xenogears (or only a file from the CD) - index_cd.bin, and parses it to get directory structure and extract files. No, You can't compile it right now, because it's incomplete - there are at least two missing functions:
- create_index_file(); // Creates table which will be populated with data later, or reads the data from CD first (?)
- parse_file(start_sector, file_size, i, filename); - extracts the file from image
Akari is still around here though, so You could ask for complete source.
-
This source won't help you. it works for Xeno but will not work for Chrono. I looked at Chrono CD today and wasn't found anything like table of contents. You will need find it first. The easiest yay will probably finfd few files start sector and try to find this numbers on disk. This is pretty easy with WinHex. Remember you need start sector and size of file in byte to extract it, so this data surely on disk in some form.
By the way it was not me, who found found TOC for xeno. It was guys from http://www.sadnescity.it/. You could ask them. they most likely extract all files since they translate it.
-
Akari, I've got what I *think* may be a Chrono Cross Table of Contents / Index file:
http://rapidshare.com/files/63794276/CC_Index_Possibility.zip.html
If you'd like to take a look at it, I can host it on another filesharing site if necessary; I don't think we can post attachments here, can we?
This is a giant hex string pulled from offset C000 in the Chrono Cross iso (sector 24), and it's 13 sectors long I think. It is somewhat similar to the Xenogears index file - it's got a byte repeat pattern, only the pattern happens every 4 bytes as opposed to every 7 bytes as in Xenogears.
-
My guess would be that that's it - it contains ~6000 integer entries that look like offsets. But then, You should have filenames list somewhere, too...
-
By filenames list, M4v3R, do you mean there should actually be filenames sitting right in the ASCII code?
-
Well, maybe not in pure ASCII, mayber they are encoded in some way. That's only my random guess, because I never dived into PSX game hacking that much. Did you tried to search for some file names in the image? You could use Scan function in Translhextion editor (I don't know if others have this function) to scan the image for text not only encoded in ASCII, but also shifted by some amount of bytes.
I guess "kernel" would be a good string to search for (if you consider that square used this as name for directories/files many times).
-
Well, maybe not in pure ASCII, mayber they are encoded in some way. That's only my random guess, because I never dived into PSX game hacking that much. Did you tried to search for some file names in the image? You could use Scan function in Translhextion editor (I don't know if others have this function) to scan the image for text not only encoded in ASCII, but also shifted by some amount of bytes.
I guess "kernel" would be a good string to search for (if you consider that square used this as name for directories/files many times).
There is no such thing as file names as I believe. Game internally don't use filenames for sure. Even FFVII don't use file names although file system has been left on disk..
-
I've run a search on both kernel and KERNEL and found nothing. However, there does seem to be a .EXE starting approximately at offset 13000 of the first Chrono Cross disc. Additionally, there's a short area of the disc that reads in ASCII:
(http://img143.imageshack.us/img143/9891/image1vw0.gif) (http://imageshack.us)
The individual files this slice of hex seems to refer to - files with .WAA, .WEP, or .KMD extensions - aren't readily detectable from the ASCII I believe. Though I'll check again later. Could this info be useful potentially? Both the .EXE header and the chunk of ASCII shown above occur very far away from where I believe the Table of Contents is.
-
And if it's relevant, I've got the Square's Preview Demo Disc, with the file structure intact. The director for "KID", or the Chrono Cross demo, has these files in it:
KID.EXE
CRONO.HED
CROCRO.IMG
TITLE.XAM
BLOOD.XAM
TIDAL.XAM
END.DAT
I can't rip the XAM ones (I get some "invalid MS-DOS function" error). CROCRO.IMG is the actual game of course, and the others I've hosted here in case they are of use:
http://chronofan.com/Black/Cross/CRONO.HED
http://chronofan.com/Black/Cross/END.DAT
http://chronofan.com/Black/Cross/KID.EXE
-
There is no such thing as file names as I believe. Game internally don't use filenames for sure. Even FFVII don't use file names although file system has been left on disk..
you might remember MPQ files from blizzard games (you know, warcraft, starcraft etc.). they are said to use one-way filename hashes instead of filenames. just thought that would be a good clue.
-
About index file you posted:
1) The differences between offsets are relatively small. Smaller than 256 in general. That could mean that if these are considered to be offsets to files in bytes, then they can't be files, they're to small (I didn't found any bigger differenes).
2) Last number (offset?) is 0x30D7372, which is 51 213 170 bytes. Too small for whole CD index.
Considering this, maybe it's a part if some sort of archive (like LGP in FF7)? That could make sense, but these numbers are not incrementing in normal manner. For eg.:
00 50 16 00 3B 60 16 00 A8 75 16 00 B9 A1 16 -->80 00 A1 16 80 00 A1 16 80 00
A1 16 80 00 A1 16<-- 00 D2 AE 16 00 BC B7 16 00 BB C3 16 00 BC CC 16 00
The marked pattern, especially the 80 in (imo) most significant byte of integer repeats itself in few places. It's like there's +80h to most significant byte in these places.
Any ideas?
-
You need to look at the programmer's perspective.
Nor that I'm a software manager, you have to see what Square wanted and then how the programmer implemented it. Let's analaze a little about the history of what Square is doing...
The technology being employed here is called "obfuscation". It's the art of hiding data. The problem is that you can only hide the data so far before it starts to impact performance. The PSX CPU was only $5.99 from LSI. When you are spending so little on a CPU, you don't have a lot of power left over to dig data out of some obscure format.
During the development of FF7, it was discovered that direct sector access was the only way to get the performance they needed. Even though the PSX supported loading files in the BIOS, it was very slow and locked the entire system until the fire was loaded. A lot of the 1st generation games simply put a "Now Loading" screen up and let the BIOS do all the work.
Because the FF7 kernel was multitasking, it was decided that creating a threaded load routine in the game was better. The reads were still done though the BIOS via a syscall, but it was only a sector at a time and the syscall would promptly give control back to the kernel when it was done with. The other major thread was the music thread, this made sure that music played during disk accesses, so it didn't look like it was doing "nothing". Neat trick.
As evidenced, with the lack of filenames, you needed to make some kind of index so the computer wouldn't get lost. Looking at the data from FF7 you can see the process that the disk was mastered.
1) When it was time to build a disk, all the uncompressed data was preloaded in a particular data structure. I'm going to guess much like how it was on the original disk.
2) Compile-time - The executables is built, it's length is recorded. Because these can be so dynamic, my theory is there is no sector data in this file. After the compile is completed, it's length is recorded.
3) PSX data created : The ANI, TIM, RSD, HRC, and all the other PSX data is created from the source assets.
4) Compression - Pass 1 - GZIP : All the files that are not sector critical are gzipped up. The compressed and uncompressed lengths are recorded.
5) Compression - Pass 2 - LZS : Here all the files that are going to be sector critical that need LZS compression are done. After they are compressed the lengths are recoded.
6) Concatenation - Headers are made for all the files that need to be stuck together. They are then done so. Sector locations are kept blank.
7) File finalazation - All files are manifested and are in the final form that will be going on the disk, minus sector information.
8) Preburn - A disk image is created on a hard drive completely blank. Blank indexes are written first. Then, when a file is copied to this image, sector information is written in the files that need it, and recorded in the master index. Each file gets this treatment.
9 ) Burn - The disk is mastered.
Now that does this mean, First this meas that the files written higher up on the CD-ROM were written first, and therefore most location-critical. This means that the master index is going to be first, or last. If someone has the time, could you see the order that the filesystem was written in? (Not by time, but by sector) I bet you are going to find some interesting surprises.
By that, we can then figure out how the other Square games were authored.
-
Thanks for the explanation on that Halkun - I have to digest it all still, but I'm glad to know that sort of thing since I have a personal interest in PSX ripping in general. And thanks everyone for your input on this particular issue.
It appears that we've probably found a way to bypass the need for an index file altogether for Chrono Cross -- we've rediscovered a disc dumping tool already written by Yazoo!
Ramsus at the Compendium has modified the code to make it run more effectively, and we've apparently got all files on the first disc dumped! If anyone's interested in what's going on at the Compendium, the relevant posts start here:
http://www.chronocompendium.com/Forums/index.php/topic,4729.msg83045.html#msg83045
Now that we've got the files, we'll need to worry about decompression of models and other data - that's the hard part, and we'll probably be back here with other questions. Yazoo created decompression tools as well, but we're unsure yet if they work for all the various file types that are in Cross.
-
Edit: FW beat me to it, but I"ll leave my old post here:
~
We had a breakthrough with Yazoo's CC hacking tools. Ramsus was able to compile them with a few changes, and they apparently dump 650 MB worth of assets.
Now, the task ahead is to slowly figure out what the other executables do, and determine what's compressed. CCTools is missing the source file "lzss-dec.cpp", so we'll probably have to try and track down Yazoo. Ramsus also plans to clean up some of the other executables; it took a couple small hacks to get the dump working.
Anyway, no telling whether documentation exists or whether the index can be inferred or found in those programs. It's great that we were able to rip everything, but I guess the real legwork starts now. FW and I are going to create a page on the Compendium's encyclopedia to record all our findings and documentation.
Thanks for the explanation post. This is all pretty fascinating.
-
Other than being messily hacked together and completely missing a file, there isn't anything wrong with any the code when compiled as C++ (if I were the author, I'd have left it as is too; it works, and it's short enough not to matter). I only made a few minor/trivial usage changes (since it has a lot of hardcoded filenames and what not that make a lot of assumptions about the directory layout the user is using) along with some minor changes to make it compile as plain C (also unnecessary, but useful for those without C++ compilers), and really, I'm only just now beginning to read the code. When I find the actual game, I'll be able to actually see what the code is looking at and get a complete picture.
From just skimming it though, the code just pulls out some kind of index of the files from the ISO and then walks through the index and copies each file's contents one-by-one. In other words, we haven't bypassed anything. Someone else just happened to find the index for us.
-
Ah, the Compendium is quickly migrating over here.
Yeah, it'd be interesting to get a hold of Yazoo and find out how he processed the original TOC/index file. Is this sort of dumper possible for every game CD, provided you know how to tailor the code to each game's individual file structure, I wonder?
-
The trick now is going to be getting these utilities to work:
http://chronofan.com/Black/Other/Chrono_Cross_-_Translation_tools.zip
They're Terminus Traduction's final set of utilities; most of the basic foundation was what Yazoo did in his CCTools.
-
The trick now is going to be getting these utilities to work:
http://chronofan.com/Black/Other/Chrono_Cross_-_Translation_tools.zip
They're Terminus Traduction's final set of utilities; most of the basic foundation was what Yazoo did in his CCTools.
Honestly, if I had known about those, I would've saved myself the trouble of even looking at the source code of the previous program. It seems like all the work there is pretty much done, and all that's left is reading the instructions.
Now you guys can start looking for the model files and figuring out what format they're in.
-
The models are most likely going to be in a binary form of RSD/HRC. The same as FF7, because that was the native format used by Psy-Q
-
Halkun, I'm now aware of the following Qhimm wiki entries on RSD and HRC format:
http://wiki.qhimm.com/PSX/RSD
http://wiki.qhimm.com/PSX/HRC
Do you happen to know offhand if there's any more info on these formats, and do you have any tips on what these formats would look like in hexadecimal? Would they have a consistent file header, you think?
@Ramsus & Zeality: Whoa! I could have sworn I came across Yazoo's Translation Tools on the Compendium earlier, but I think they were uncompiled. These look great! Anyone know whether they should be run on the entire disc image or on the .OUT files instead? I'll try both.
-
The models are most likely going to be in a binary form of RSD/HRC. The same as FF7, because that was the native format used by Psy-Q
FFVII and XENO format are not in RSD or HRC but in compiled format that almost ready to be sent to GPU. So it's better to read documentation about PSOne than RSD/RHC specification.
-
The models are most likely going to be in a binary form of RSD/HRC. The same as FF7, because that was the native format used by Psy-Q
FFVII and XENO format are not in RSD or HRC but in compiled format that almost ready to be sent to GPU. So it's better to read documentation about PSOne than RSD/RHC specification.
I believe that was what halkun meant by saying "binary".
-
Ironically I know a bit about the CC save state format.
However my time is still a bit stretched these days.
If you are going to keep the information you find public this is one of the games I wanted to migrate to Q-gears.
As I thought a fair bit of this stuff was available, I could be wrong.
I thought Chrono Cross was released about the same time as FF9?
Cyb
-
Do you happen to know offhand if there's any more info on these formats, and do you have any tips on what these formats would look like in hexadecimal? Would they have a consistent file header, you think?
The compiled versions of the HRC/RSD will be a lot like the battle models/field models.
Here's the format on those
http://wiki.qhimm.com/FF7/Battle_model_format_%28PSX%29
-
Thanx Halkun!
Yeah Cyb, FF9 and CC were both released in 2000 in the US. Maybe that means model formats and various other aspects of the games will be similar?
Not to get off topic, but how is Qhimm's FF9Viewer coming along? It's not available for download yet, is it? It looks to me that CC and FF9 game resources are pretty compatible from an aesthetic standpoint. I'm starting to get wild ideas. :evil:
Everything we're discovering over at the Compendium is publicly viewable, I think. It's just a matter of gathering all the utilities and notes into a single place, which is no small task now.
EDIT: We've freshly acquired battle model textures and files that *may* be main character battle models. I'm linking to a sample of three character data folders. The "5" files are definitely battle model textures, and there's eyeball TIMs in these folders as well. But what I'm really interested in are the "3" files. Can anyone give me an opinion as to whether there's model data in the "3"s?
http://rapidshare.com/files/64618291/MainCharFiles.rar.rar.html
-
Well the backgrounds used 16bit palletized TIMS that were loaded into the right section of the display memory (IE 320x240 == 16bit display leaving 704x 240 pixels to the right of the display to hold the background data in). The background format however is kind of unknown as I didn't get too far with it. However it looks like instead of the complicated system they used for FF7 they just stored the palettes and the background information (8bpp likely) all together and used quads to dump the background information to the display. I think they used the transparency bit for some of the transparent/walk behind parts of the palletized background. However the backgrounds were pretty much one BIG TIM.
Cyb
-
When you say "background," Cyb, what do you mean exactly? I'm thinking pre-rendered backgrounds, that's not what you're referring to, is it?
-
When you say "background," Cyb, what do you mean exactly? I'm thinking pre-rendered backgrounds, that's not what you're referring to, is it?
2D backgrounds is picture that you can see when looking at field. It tiled with 16x16 or 32x32 tiles. It depends on layer in FFVII. Data format of backgrounds in ffvii are like SPRITE packet to GPU without header. And there are 4 layers of background. 1 - always rendered at the bottlom, 2 - each tile has it's own depth (distance from screen) for correct sorting, 3 and 4 are dynamic layers that managing through script.
Look at http://q-gears.svn.sourceforge.net/viewvc/q-gears/trunk/src/ffvii/field/DatFile.cpp?revision=177&view=markup for GetBackground().
-
Thanks Akari. Is QGears intended to dump files from games eventually, or is it an engine that plays the games from the disc?
And we've opened the files in the "rooms" directory finally at the Compendium. There's data in these that I'm told is 3D, which might be your walkmeshes - though I'm still not sure how to determine which files belong to which fields/dungeons/etc. in-game yet.
-
Pre rendered backgrounds indeed was what I was referring too. :D
Maybe I should add TCL or python scripting to my PS1 ISO viewer to make it easy to make data viewers for various games quickly.
The interesting thing about said idea is that even though each game has unique data access attributes and unique structures etc. They all use the same hardware.
I was perusing my Medievil game and noticed that this game had similar uses of data as all the other games.
If one is going to do TCL or python or any script engine to render content from games, the end method of rendering the data is mostly similar so such a methodology is not a waste of time.
3d data should be in several sets IE vertex pools (sets of 3d data ) etc. They are likely ALL 32bit aligned (3 vertices's and one dummy 16bit signed for example). Keep this in mind and the format will be easier to decipher. If you looked at the FF7 battle format information (I think I put most of that in, it's been a while), you'll notice the structures follow this explicitly. Since a walk mesh for example is not visible 3d data but more a collision surface (IE gravity collision) it will be arranged whatever is convenient for the GTE to process and not the GPU. My guess is whatever arrangement was most convenient.
Q-gears is to play the data from the games. Since the versions for the PC of FF7 and FF8 are buggy and are difficult to get to work on said platforms, and playing FF9 xenogears Vagrant Story etc. require an emulator. This is a good alternative, as content distribution is verboten (IE textures and stuff being ripped from the game) it is a good idea to keep this in mind. As I've noticed a lot of people getting C&D'd projects because of this. Lawyers will follow if you distribute anything that is from a copyrighted work.
As for enhancements that has been a difficult thing to contemplate. As to remain legally safe it can only be done in vitro. New background content could be a problem. Of course there is duke nukem 3d (eduke) with the 3d models instead of sprites. I think it's possible to make a method to facilitate the content but not distribute it with Q-gears (as this might bring up trademark and copyright issues again).
Cyb
-
Quick and silly question for you Cyb, or anyone else who can answer - is the PSX battle model format wiki describing what the data looks like when it's uncompressed? Meaning I'd have to take suspected model data from Chrono Cross and run it through an LZS decompressor (assuming that's the compression scheme CC uses, of course) to be able to compare to what's in the wiki?
http://wiki.qhimm.com/FF7/Battle_model_format_%28PSX%29
-
Quick and silly question for you Cyb, or anyone else who can answer - is the PSX battle model format wiki describing what the data looks like when it's uncompressed? Meaning I'd have to take suspected model data from Chrono Cross and run it through an LZS decompressor (assuming that's the compression scheme CC uses, of course) to be able to compare to what's in the wiki?
http://wiki.qhimm.com/FF7/Battle_model_format_%28PSX%29
The FF7 model data is compressed as a complete set. However I believe CC is uncompressed data. It is likely similar to FF9's model format. You should look for a SECTOR that contains a TIM unaligned to the sector boundry. That will be a clue that it's either background data or model data. If you inspect the TIM and it appears to be skin data then THAT is your model. In FF8 there was an 128 byte header and then a TIM or two. FF9 is similar. Although I don't know the specifics for the FF9 model data.
Cyb
-
Ah, THANK YOU once again Cyb. So the character model textures are typically stored right along with the models then? I'm currently mining for model data in a section of the CD image that contains the character battle model textures, weapon model textures, and unknown data sandwiched in between that I suspect might be character and weapon models.
When you speak of TIM textures unaligned with the sector boundary, you mean that the TIM begins at some point after a new sector, correct? Would 00 "buffer" bytes fill the space in between the sector start and the TIM then? And are we talking about 2048-byte sectors or 2352-byte sectors? I'm working with a CD image that has 2048 bytes per sector I think; not sure if that changes things any.
And I know now that sound data is certainly stored similarly in FF9 and Chrono Cross as well (both sound file formats start with AKAO in ASCII). I'm starting to wonder just how closely the file formats in Chrono Cross resemble those in FF9. Is it possible that Qhimm's FF9Viewer program would be able to detect Chrono Cross data as well?
EDIT: By the way, I don't think I've mentioned it above, but Ramsus and Zeality figured out Chrono Cross' pre-rendered background format (at least as it pertains to the visible 2D stuff) over at the Compendium. There's not much documentation on it yet, but I can relay what we know so far here if anyone is interested.
-
I can add it to the wiki here or someone else can depends on how much time I can find.
Model data is likely in Mode 1 sectors so that would be 2048 sized.
The space between the sector beginning and the TIM is likely to be offsets to the TIM and model data. It's very efficient to load everything that way so that the TIM files are loaded from memory straight into the GPU VRAM. Generally they've used a table of offsets to point to data later in the file. So each DWORD sized value is very likely the OFFSET into the actual data. The total file size is already known so ... :) What's likely to follow the TIM is the bone data however there is no certainty of that.
Cyb
-
If you find data that repeats that has the GPU quad opcode in it, follow the data and see if it traces out a UV map on the TIM. That's how I found the texture maps in FF8.
Remember, the PSX can do Quads "out of the box" and does not need to be broken down into triangles.
-
Well, I think we (the people from the Chrono Compendium) don't really know what a "quad" is.
-
Heh heh. Very true, Vehek. :-D Visiting the Qhimm forums sure has been a learning experience!
We may luck out and not have to do the tracing Halkun suggests if I'm correct in my suspicion that model data is sandwiched between the texture files I'm examining. But that's a BIG if. Later on I'll get some data isolated (say, all of Serge or Kid's battle and weapon textures, with the possible model data in between) and post a download link here so everyone can take a look at it with me. I have yet to confirm that the data fulfills the conditions described by Cyberman above, but it looks promising.
BTW Cyb, when we're talking DWORDs, is that 32 bytes in length? Is word size hardware specific, i.e., should it be consistent across all PSX games?
-
I checked out the battle field OUTs for the hell of it. As expected, they contain the battle field textures, and each is prefaced with a 3-byte descriptive header. The header seems to be quick romaji. For instance, "kumo" goes before the cloud texture. Anyway, after removing the TIMs from the source OUT file, around 53 kb is left. The OUT file starts with "DRP", so I'm assuming what's left are the field model box and texturing instructions. I've attached a chart of the TIMs and their corresponding headers, along with the remaining OUT file here (http://www.chronocompendium.com/Forums/index.php/topic,4770.msg83790.html#msg83790).
-
Now for a report on the potential battle model / weapon model data.
http://www.chronocompendium.com/Forums/index.php?action=dlattach;topic=4770.0;attach=1950
Firstly, the link is a zip containing a chunk of data from the CD image that I have named "ModelMine1." It's just raw hex data, and its beginning corresponds to a fresh sector (2048 bytes/sector as always) in the CD image. The data fall in this order: A header; Serge's battle model texture; Serge's eyeball texture; unknown data; Serge's first weapon model texture; unknown data; etc. Also inside the zip is a copy of my datamap for "ModelMine1," which I reproduce below. The chunk seems to be 11 sectors long exactly, if that means anything.
Also included in the zip are Serge's battle model texture, eyeball texture, and the seven weapon texture TIMs that occur within the ModelMine.
DATAMAP: All offsets given in hexadecimal notation.
00000000 ~ 0000000F: 16 byte header
00000010 ~ 0000823B: Serge Battle Texture
0000823C ~ 00009467: Serge Eyeball Blink Texture
00009468 ~ 000097FF: 00 Buffer Bytes
00009800 ~ 00014847: Unknown Data (Battle Model?)
00014848 ~ 00015DC7: Serge Weapon Texture 1
00015DC8 ~ 00016BE7: Unknown Data (Weapon Model?)
00016BE8 ~ 00018167: Serge Weapon Texture 2
00018168 ~ 00018C5F: Unknown Data (Weapon Model?)
00018C60 ~ 0001A1DF: Serge Weapon Texture 3
0001A1E0 ~ 0001AF0F: Unknown Data (Weapon Model?)
0001AF10 ~ 0001C48F: Serge Weapon Texture 4
0001C490 ~ 0001D2EF: Unkonown Data (Weapon Model?)
0001D2F0 ~ 0001E86F: Serge Weapon Texture 5
0001E870 ~ 0001F687: Unkknown Data (Weapon Model?)
0001F688 ~ 00020C07: Serge Weapon Texture 6
00020C08 ~ 00021897: Unknown Data (Weapon Model?)
00021898 ~ 00022E17: Serge Weapon Texture 7
00022E18 ~ 00023FFF: Unknown Data (Weapon Model?)
After this begins my ModelMine2 file, which contains Kid's suspected battle model and weapon model data. I'll post that later for comparison if anyone's interested.
SUSPECTED MODEL FILE INFO: The offsets and file lengths may actually be one byte too "short" depending on whether I'm interpreting the hex code correctly. For example, the First "Weapon Model" listed below is described as 3615 bytes long. If it makes more sense, feel free to interpret the file size as 3616 bytes.
First "Model" : 45127 bytes long. Header begins "06 00 00 00 20 00 00 00"
First "Weapon Model" : 3615 bytes long. Header begins "01 00 00 80 48 00 5E 00"
Second "Weapon Model" : 2807 bytes long. Header begins "01 00 00 80 39 00 46 00"
Third "Weapon Model" : 3375 bytes long. Header begins "01 00 00 80 43 00 58 00"
The weapon "models" are consistent in their first 8 bytes, which may constitute a model file header; the first 8 bytes for the weapon "models" read: 01 00 00 80 xx 00 xx 00 (where "xx" represents a byte position that varies in value among the "models")
My notes on the data were hastily scribed, so if anyone wants clarification on any point, please ask. Once again, the "ModelMine" for Serge's (suspected) battle and weapon models and all the associated textures are in the attached zip if anyone would like to investigate.
ADDENDUM: Also, a question for Halkun or anyone else who understands PlayStation design/architecture: Are PsyQ opcodes for the Playstation GPU actually planted amongst the game data? I'm interpreting Halkun's previous post about quads as saying that I should be able to see the applicable opcode just before the model geometry, TIM textures, etc. Am I understanding this correctly?
If GPU opcode *is* in with the game data, is it also in the SLUS file? If both the SLUS file and the data following the SLUS file on the game CD contain opcode, and that opcode features the same call functions, then perhaps I could view the call functions in the SLUS file in hex, determine their ASCII signature, and then find the call functions further on in the game CD as well, pointing to model data.
Does that make any sense whatsoever? I've yet to read the PlayStation design documents, but will begin doing so in the near future. Oh, and another question -- what "language" are the PsyQ opcodes written in? MIPS Assembly language?
-
I'm double-posting with full knowledge that such practices are discouraged. However, I feel it is necessary given the extraordinary length of my previous post.
I've found some very interesting ASCII blocks in Chrono Cross' SLUS file. First up...
(http://img165.imageshack.us/img165/8956/filepathsim9.gif) (http://imageshack.us)
Seems to be file paths. Is it possible that this is a signature left from the disc burning process, meaning there's MDE, FID, FND, and PRG files on the game CD? Anyone know what these file extensions even refer to?
Next up...
(http://img292.imageshack.us/img292/3513/possiblemodelopcodeskp1.gif) (http://imageshack.us)
If I had to take a wild guess, I'd say these might be assembly language opcodes that tell the Playstation GPU how to handle model data, but of course "I R a NooB" at this sort of thing. Now, this wouldn't be functional opcode data, right? Just ASCII evidence that the opcodes are in the SLUS. I expect I'd find these elsewhere in the SLUS when I investigate with a disassembler such as PS2Dis. But I'm not sure PS2Dis knows to "tag" the various assembly language opcode functions with these labels. Anyone know of a full-featured PlayStation-centric disassembler that already does the trick?
I just wanted to throw this out there in case others have seen this sort of thing and can comment.
It's about time I delved into Halkun's PlayStation architecture/design doc -- grabbed it off Zohar.net just a few minutes ago. I shall return enlightened. :mrgreen:
-
ADDENDUM: Also, a question for Halkun or anyone else who understands PlayStation design/architecture: Are PsyQ opcodes for the Playstation GPU actually planted amongst the game data? I'm interpreting Halkun's previous post about quads as saying that I should be able to see the applicable opcode just before the model geometry, TIM textures, etc. Am I understanding this correctly?
Not everything will have a GPU code, only the quadric's and triangles used in the model would use that.
I'm not positive but TIM et all use bios calls that set up a DMA into the GPU to load the data into the VRAM.
If GPU opcode *is* in with the game data, is it also in the SLUS file? If both the SLUS file and the data following the SLUS file on the game CD contain opcode, and that opcode features the same call functions, then perhaps I could view the call functions in the SLUS file in hex, determine their ASCII signature, and then find the call functions further on in the game CD as well, pointing to model data. [/qoute]Well actually it is more likely there is a buffer of where to load the model data. I know the background files contain fixed load points in each file in FF7. It's less likely GPU opcodes will be interspersed in the executable. The code in the executable is more likely to know what to do with it and use it to set up the GTE to transform the data to be sent to the GPU instead.
Does that make any sense whatsoever? I've yet to read the PlayStation design documents, but will begin doing so in the near future. Oh, and another question -- what "language" are the PsyQ opcodes written in? MIPS Assembly language?
Halkun has a nice document he wrote "everything you wanted to know about the playstation but were afraid to ask" or something like that. I recommend reading the GPU commands. That is one way you can distinguish between quads and triangles in the data you are looking at.
Remember ALL CODES ARE 32 bits this includes the GPU so be sure to think that when you are looking for GPU codes.
What you have is part of the library for loading stuff into the GPU likely :)
Cyb
-
Thanks Cyb! :-D Yes, Halkun's "Everything You Ever Wanted to Know But Were Afraid to Ask..." is first on my reading list over the next few weeks. This is fun!
-
Short History on Psy-Q
Once Upon a time, Sony made an arcade board for Namco called "System 11". Namco used this arcade system and created games such as "Ridge Racer" and "Tekken". During the fall out between Sony and Nintendo over the cd-rom addon, Sony took the System 11 board, tacked on the PSX cd-rom technology, put it on a box, and called it "The Playstation"
Now Sony had a problem.
Sony is a hardware manufacturer. They are not coders, and for this new system, had no real game development kits to go along with it. They invited Psygnosis, a very popular video game company at the time to create a 1st party game and Sony wanted dibs on the development kit they created for it. Psygnosis created "Psy-Q" and went on to create the first "wipeout" game with it. Psy-Q was a good game library, and had the ability to abstract all the hardware away from programming. You called the Psy-Q commands, which in turn called the BIOS, which in turn commanded the hardware to do cool things.
A snag was hit early on when Sony made the decree that software companies could only go though the PSX BIOS or Psy-Q, when making the game. No direct hardware calls were allowed. (This was because they wanted to preserve a kind of "reference" that could put into a later machine for backwards compatibility).
Psygnosis thought because they made the dev kit, they were exempt from the "no direct hardware programming" rule. The first release of Wipeout talked directly with the PSX. Sadly when the first revision of the hardware came out, it crashed all the first-generation Wipeout games and Psygnosis had to make another version more compatible with Sony reference.
Psy-Q is a game library and toolset written in C, and was used on DOS. It used a modified gcc compiler. Looking on the internets, you can find copies of the Psy-Q documentation, but having is illegal. You can also find the library too if you look hard enough. This is why you will find programs that seem to use the same dataset over and over again. (TIM, for example, or SEQ) these are all Psy-Q data formats that are "native" to the PSX.
TIM = bitmap picture/texture format
HRC = hierarchy format
TMD = 3d object format
RSD = model+hierarchy+texture format (resource data, it's the whole "3D entity")
ANI = 3D Animation format
VAG/VAB = digital Audio format (group/single)
SEQ = Sequence (midi) data
STR = Streaming video format
You can see how this becomes a basis for a game. If you can find this data hidden on the disk, youhave an idea of how to "expand" it into something to use. Keep in mind: The PSX only had 4 meg, that's not a lot, especially when you realize that you have no dynamic libraries and most of the time Psy-Q was statically linked to the executable somehow.
That should start you off.
-
That starts me off alright. :-o Thanks Halkun!
Now that I've figured out how to take advantage of PS2Dis' built-in hex viewer (offset disparities between that and the SLUS file as shown in my hex editor were giving me a headache for awhile), I should be good to go once I find out what the quad GPU opcodes look like in assembly language. Sharing that info on this board is strictly "Verboten," right? Meaning I'll have to figure it out from some documentation I downloa- well, nevermind. :-)
Ooh, and that "ANI" format - do all PSX models use that for animation, or only TMDs? Do developers typically come up with their own animation formats as they do with their proprietary model formats?
-
There are two major animation formats that have been found, and both are used in FF7. One is a bitwise delta format that is used for the battle models, and is closely related to the Psy-Q animation format. The other is a "frame" format that simply holds all the angles for the bones in that frame.(used for the field characters) I don't know if the both come from Psy-Q. I know MGS used a special animation format that was a kind of compressed delta, and Square uses the modified AKAO format for sound. The Psy-Q animation format has support for not only delta positions, but also machine states, so it can get really complex.
Oops, let me define:
Delta - Change over time. As opposed to listing joint angles every animation frame, it only lists how much the angle has changed via a +/- value. It's smaller, but more complex to decode.
State machine: Basically a "mode" that a particular program runs in. When it comes to animation it can have a "stand" state, a "walking' state", a "running" state, and a "jumping" state. These are all linked togeather to from a branch-loop.
stand->walk->stand->jump->stand->walk->run->stand.
I know we decoded the delta format, I think it's in the wiki. I'm not sure about the state machine though.
Talking about the PSX GPU here is fine. It's open and not a trade secret. Pretty much anything about the PSX is fair game except for the BIOS function names. Psy-Q, on the other hand, is a little more tricky. If it looks like you are copying out of the Psy-Q manual, that is a sure-fire way to get yourself banned. However, If you can get a clue about a format by analyzing the data, you should be in the clear.
Also, take the strings you find in executables with a grain of salt. Some of that data looks like garbage left over after the compile process. Sometimes compilers compile a program in memory and bulk-copy the finished heap to disk. When that happens it can "scoop up" chunks of the compile process as filler to round out the memory boundary. It's interesting that it might lead a clue to the compile process, but not necessarily anything useful about the executable.
P.S.--> Oh and by the way... Why can't we talk about the Psy-Q/BIOS functions?
Because Sony told me not to, that's why.
P.P.S. --> V hfrq gb jbex sbe Anzpb
-
Halkun, you're a walking PSX encyclopedia! Truly amazing!
Ah, so nothing wrong with GPU opcode discussion then. *Breathes sigh of relief* Is there any documentation on what GPU opcode / functions look like in assembly language, Halkun? (At least related to quads, if that's the only opcode that appears near model data) I didn't see anything about that on the wiki.
-
The gpu opcodes are not in any language. How they work is you make a list of opcode commands that you want to the GPU to accomplish, line them all up in a row, and then fire them to the GPU for execution.
You can talk to the GPU two ways, by giving the opcode directly to the GPU, or by fireing them off via DMA. The GPU hand off only works one at a time. The DMA method is much faster and supports Ordering Tables (OT) or linked lists of GPU opcodes.
What makes OTs so easy is you can pre-rig your list of opcodes, and then use the GTE to sort each "triangle" opcode by Z depth. Then you have a paint order that the GPU can render triangles to the screen. (That's why they are called "ordering tables") In FF8, the texture data consisted of pre-made GPU packets that was missing some data (to be filled by the GTE).
Of course it's been almost 7 years since I looked at them.
FF7 was more HRC-ish when they were built.
-
The other is a "frame" format that simply holds all the angles for the bones in that frame.(used for the field characters)
In PSX format not that easy. It is not set all rotation for all bones, but only rotation for defined bones and to defined axis (many bones have only one or two axis for rotation). And, in addition root bone movement. And all this stored not frame by frame, but all rotations for 1 axis of one bone, next next axis, next bone and so on.
-
Speaking of animation formats, a few days ago I was trying to look at the Vagrant story animation data and watching some animations in-game I noticed they definately aren't skeletal like the TMD seems to be, but more like a list of vertex coordinates for each frame (like the MD2 of quake 2 I think) It's very noticeable on some weird monsters like the slimes and the Dark Eyes with theyr glowing and theyr texture "animations" but also on some humanoid models there are some visible distortions.
I guess Psy-Q doens't support anything like this, eh?
And Faustwolf, since VS and CC are both from Square and from the same age, I looked at those files you posted but they seem very different beasts.
-
Thanks for looking at the files, Lupus! Once we get past the hurdles associated with viewing Chrono Cross' models, Vagrant Story is actually very high on my list of games to explore next. Have you succeeded in viewing VS model data?
-
No, but I haven't really investigated, and probably I won't for a good while....
OTOH, some time ago I started uploading some VS data on http://www.datacrystal.org/wiki/Main_Page (http://www.datacrystal.org/wiki/Main_Page)
-
Quick question:
Headers example
00 00 00 00 64 65 67 75 ....degu
04 00 22 01 00 00 00 00 64 65 67 31 04 00 22 01 00 00 00 00 ..".....deg1..".....
64 65 67 32 04 00 22 01 00 00 00 00 64 65 67 33 04 00 22 01 deg2..".....deg3..".
00 00 00 00 64 65 67 34 04 00 22 01 00 00 00 00 64 65 67 35 ....deg4..".....deg5
04 00 22 01 00 00 00 00 65 6e 6b 65 04 00 22 08 00 00 00 00 ..".....enke..".....
65 6e 6b 31 04 00 22 04 00 00 00 00 67 61 6b 65 04 00 22 01 enk1..".....gake..".
00 00 00 00 67 61 6b 31 04 00 22 01 00 00 00 00 67 61 6b 32 ....gak1..".....gak2
04 00 22 01 00 00 00 00 67 61 6b 33 04 00 22 01 00 00 00 00 ..".....gak3..".....
67 61 6b 34 04 00 22 01 00 00 00 00 67 61 6b 35 04 00 22 01 gak4..".....gak5..".
00 00 00 00 67 61 6b 36 04 00 22 01 00 00 00 00 67 61 6b 37 ....gak6..".....gak7
04 00 22 01 00 00 00 00 67 61 6b 38 04 00 22 01 00 00 00 00 ..".....gak8..".....
67 73 30 31 04 00 22 01 00 00 00 00 67 73 30 32 04 00 22 01 gs01..".....gs02..".
00 00 00 00 67 73 30 33 04 00 22 01 00 00 00 00 67 73 30 34 ....gs03..".....gs04
04 00 22 01 00 00 00 00 67 75 30 31 04 00 22 01 00 00 00 00 ..".....gu01..".....
67 75 30 32 04 00 22 01 00 00 00 00 67 75 30 33 04 00 22 01 gu02..".....gu03..".
00 00 00 00 67 75 30 34 04 00 22 01 00 00 00 00 6a 69 6d 65 ....gu04..".....jime
04 00 22 01 00 00 00 00 6a 69 6d 31 04 00 22 01 00 00 00 00 ..".....jim1..".....
6a 69 6d 32 04 00 22 01 00 00 00 00 6a 69 6d 33 04 00 22 01 jim2..".....jim3..".
00 00 00 00 6a 69 6d 34 04 00 22 01 00 00 00 00 6b 75 6d 6f ....jim4..".....kumo
04 00 22 02 00 00 00 00 6b 61 6e 6b 04 00 22 04 ..".....kank..".
Would those be the actual filenames for the TIMs? These three byte headers go before each TIM in the battle data. I think pretty soon we can begin asset extraction in the files produced by Yazoo's tools at the Compendium...sort of like, getting identifiable data out of OUT files, leaving the rest / deleting them if they are completely convertible.
-
Speaking of animation formats, a few days ago I was trying to look at the Vagrant story animation data and watching some animations in-game I noticed they definately aren't skeletal like the TMD seems to be, but more like a list of vertex coordinates for each frame (like the MD2 of quake 2 I think) It's very noticeable on some weird monsters like the slimes and the Dark Eyes with theyr glowing and theyr texture "animations" but also on some humanoid models there are some visible distortions.
This is common sence in bones animation. Each vertex has weight parameter bones (or how much bone movement influence it). You can set vertex linked to two bones at once, so you will see ettect of distortion (when dress move and stretch). This was used in FFIX.
-
This is posted at the Chrono Compendium also, but I'm reproducing here. It relates to a comparison I'm attempting between FF8 model data and Chrono Cross model data:
I've just done some investigation of the data following Zell's battle model texture(s) in my disassembler (because viewing data through a disassembler makes me feel cool 8-)), and it appears the data immediately following Zell's texture starts with the following assembly language instructions:
dsrav zero, zero, zero
sync
dsllv zero, zero, zero
Halkun, does this provide any clue as to whether or not I'm looking at quad opcodes? Or, an alternative question, how is one able to tell that he/she is even looking at quad opcodes? :oops:
And I've got a few quick questions about your PSX doc, which I am still reading through (rather haphazardly, as time allows):
1.) In your GPU packet command list, "0x2c" constitutes a packet command for a textured 4-point polygon. What would that look like in hex? Just "2C"? And would that appear in a quad opcode?
2.) You differentiate between "monochrome" and "textured" polygons. Would I be correct in stating that Final Fantasy 7 uses monochrome polygons (excluding faces, I guess?), whereas Final Fantasy 8 (and, by association, Chrono Cross) uses textured polygons? This is an obvious NooB question, but I just want to make sure.
EDIT: I received your answer on Chrono Compendium Halkun! Thanx!
-
Posting this here for maximum exposure; it's on the Compendium as well:
The "Uknown Data" in my ModelMine1 post can now be referred to as "Model Data." :D
Here's tonight's update. The vid quality is less than I would have liked, but hopefully we can find some insight within its grainy-ness. No sound either; I'll report later as to whether the "Chimera" uses Kid's or Serge's sounds in battle, because I've forgotten. :-D
The first video is just a demonstration of how the characters Serge and Kid look in battle. Folks already familiar with Chrono Cross battle models can probably skip to the second vid, but this provides a useful reference:
http://www.youtube.com/watch?v=uxts0pisnI0
Now the fun begins. As boring as it will be at times, please watch this in its entirety to get a full impression of what's going on with the model data. The final 30 seconds are especially important:
http://www.youtube.com/watch?v=Ysh*t-thL5Q
Please finish the web address of the second vid with the letter "I". Silly Youtube. :roll:
As you can see, the "Chimera" is Serge's model data with Kid's battle texture. Serge's standing, running, and spell-casting animations have carried over, but not his physical attack animations for some reason. But a complicating factor is the significant difference in file size between Serge's and Kid's model data. Specifically (and surprisingly), Kid's model takes up 8184 bytes more in the game CD than Serge's! It could be that Kid's animations are more complex than Serge's or something, and therefore need more space.
I'll illustrate graphically what I did exactly. Below is a representation of Serge's battle model data (sans skin & eye blink textures). In ModelMine1, the unskinned model lasts from hex offsets 00009800 ~ 00014848:
(http://img233.imageshack.us/img233/4286/sergemodeluo7.gif) (http://imageshack.us)
The following is a representation of Kid's unskinned battle model data. I will release her battle data set - ModelMine2 - tomorrow evening, once I jot down some datamapping notes on it. Then we can find potential model file headers.
(http://img264.imageshack.us/img264/948/kidmodelrr6.gif) (http://imageshack.us)
Now, with Serge's model data written over Kid's:
(http://img264.imageshack.us/img264/5765/chimeramodellq5.gif) (http://imageshack.us)
So some of Kid's model data is still left over. At some point I'll wipe the rest of the "red" (Kid) part out with 00 bytes and see how that changes things.
At least this shows that we've got battle models. Now, the question is how to go about deciphering the model format so it can be viewed in 3DS Max, Blender, etc.
-
I had a look at the "modelmine" file, I just noticed only now.
The file is nicely structured a header and relative offsets to subsections, and each subsection has a similar header. The only exception seems to be the (probable) model data starting at 9800, this may be hardcoded in the game code. Same for 14800 which is a header containig pointers to the other subsections until the end of the file.
Section 00009800 ~ 00014800:
the first $20 bytes are a header. The first value may be the number of sections, the others are relative the offsets (relative to the beginning of current section, that is). Some of the sections are similarly split in subsections. I have no clue what they may be :)
Section 14800 - end of file:
header as above; the "unknown" sections between weapon textures are certainly weapon models.
for example, the first is at 15dc8. From there:
0-7 unknown
8-11 offset to vertices section
12-15 probably offset to unknown section (at 15df0)
this seems to be made of groups of 4 bytes. I nocited some textures have weid coloring,
maybe models are textured AND color shaded. They could be RGB values. (but the 4th byte?)
16-19 probably offset to unknown section (at 15ddc)
The vertices section is a Cyber said above: groups of 4 2-bytes values, the last is 00 00. Vagrant Story is like this both for characters and weapons, but your model data here seems different.
Note how the first value also is often 00 00 or little more, that's because of the flatness of weapon models. However the last value sometimes is 01 00 or 02 00 :|
-
Thanks Lupus! I'm copying your post and re-producing it at the Chrono Compendium, if you don't mind. Here's the latest updates there:
http://www.chronocompendium.com/Forums/index.php/topic,4770.msg83974.html#msg83974
-
Just stopping in to report that the static model at rest has been (99%) successfully imported into Blender.
(http://www.chronocompendium.com/images/wiki/a/ab/Sergemodelpreview.png) (http://www.chronocompendium.com/images/wiki/1/1d/Sergemodel.png)
Thanks for the help, everyone.