Chrono Cross exploration & file structure

[ZeaLitY]

With FaustWolf leading, we've been attempting to map Cross's file structure at the Chrono Compendium and rip its assets. Aside from what PSXMultiripper, PSMPlay, and VRAM viewer give us, it's been a tough process. Of course, it's been a problem that Cross seems to behave like Xenogears:

http://wiki.qhimm.com/Xeno/IndexFileFormat

There's no real file structure, so we think that Cross is using the same process (the games were released in close proximity, too). FW has been trying to find Cross's index file, and he's got it down to two candidates. Here are his notes:

It seems the disc sectors in Xenogears are 2048 bytes long, not 2352 (which I thought was the iso standard). If 2048 is the magic number, then everything seems to be shaping up with what's in the qhimm wiki reference on Xeno. That makes the index file's starting offset in Xenogears C000, BTW. Not sure what it'll be in Cross, though the PlayStation logo model is stored at the same offset in both Cross and Xeno.

Anyone familiar enough with disc sectors to tell me if 2048 sounds like a reasonable number of bytes per sector on a PlayStation CD, and if it's likely that Cross has the same number of bytes per sector?

In other news, I'm attaching a hex string that might just be the Chrono Cross table of contents. It is significantly different from its Xenogears counterpart; for one thing, it's only 13 sectors long as opposed to 16, and secondly, the byte repeat pattern* happens every four bytes as opposed to every seven bytes. But I believe it may be what we're looking for.

Still no idea on how to open it.

Relevant attachment is at this linked post.

I guess this is just a request for any advice. The Chrono community isn't the most active one, so we're in short supply when it comes to experienced talent who have a better idea of what to do. We did discover that someone name Yartrebo who successfully ripped Serge's character model by hacking PeOPs, but his files are long since gone and we can't contact him. We've also sent an e-mail out to Terminus Traduction, who successfully extracted and translated the game's script (using these tools and documentations). FW hasn't gotten around to studying their notes in depth yet.

Anyway, thanks for reading!

Edit: Oh cool, seems I've been here before...I recognize halkun's name. We added your comments on the Chrono Trigger PSX file structure to the Compendium's encyclopedia. Until your post, no one knew why that ROM was on there.

[Vehek]

Edit: Oh cool, seems I've been here before...I recognize halkun's name. We added your comments on the Chrono Trigger PSX file structure to the Compendium's encyclopedia. Until your post, no one knew why that ROM was on there.

Or, it might be because I quoted Halkun's post here over at the Chrono Compendium.
Edit- (...which is likely why you visited...)

[Akari]

Usefull disk sector size is 2048. This it is readed by usual CDrom. 2352 is complete sector size on disk. It includes disc header. Size 2352 used for some videos. All usual data stored in normal 2048 part of sector and calculated according to it.

[The Skillster]

I feel Chrono Cross is possibly Squares best attempt at a PSX game to date ( I think it is more advanced than FF9).
But Chrono Cross was almost their last game on the PSX where as XenoGears came out almost in line after FF7.
Akari is the master of Xenogears, I think he might know alot more.

[FaustWolf]

Okay, finally got an account here.   

First of all, thank you Akari for that confirmation on the 2048 bytes per sector.

A while back, in this thread: http://forums.qhimm.com/index.php?topic=4919.0 , you posted some code that is supposed to parse the Xenogears index file. Specifically...

Code: [Select]

extract(char *filename)
{
    unsigned long start_sector, file_size;
    unsigned char start[3], size[4];
    char numero[10];
    // first directory number 0
    int directory_number = 0;

    // create file with cd file index
    create_index_file();

    index_file = fopen("index_cd.bin", "rb");

    for (i = 0; ; i++)
    {
        file_position = fread(start, 1, 3, index_file);
        file_position = fread(size, 1, 4, index_file);

        // reverse bytes order (low endian)
        start_sector = start[0] | (start[1] << 8) | (start[2] << 16);
        file_size = size[0] | (size[1] << 8) | (size[2] << 16) | (size[3] << 24);

        // end of file
        if (file_size == 0x00ffffff)
        {
            break;
        }
        if (start_sector == 0 && file_size > 0)
        {
            filename[9] = 0;
        }
        // directory
        if (file_size > 0xff000000)
        {
            // create directory name (name - number of directory)
            filename[9] = 0;
            sprintf(numero, "%d", directory_number++);
            strcat(filename, numero);
            strcat(filename, "\\");
            mkdir(filename);
        }
        // file
        if (start_sector != 0 && file_size < 0xff000000 && file_size > 0)
        {
            printf("file number %d, size: 0x%x byte - ", i, file_size);
            parse_file(start_sector, file_size, i, filename);
        }
    }
}
But I can barely tell the difference between C++ and Java anymore; could some kind soul tell me how to use this code? Does it need to be compiled into an executable, perhaps? And does it simply extract the Table of Contents data from the CD, or will it convert the data into something that says where the archives and other files are located on the CD?

Also, if this works for Xenogears, could it work for Chrono Cross' index file/Table of Contents as well, or would it need to be modified?

Thanks!

[mav]

From what I can tell, this code takes the image file for Xenogears (or only a file from the CD) - index_cd.bin, and parses it to get directory structure and extract files. No, You can't compile it right now, because it's incomplete - there are at least two missing functions:

- create_index_file(); // Creates table which will be populated with data later, or reads the data from CD first (?)
- parse_file(start_sector, file_size, i, filename); - extracts the file from image

Akari is still around here though, so You could ask for complete source.

[Akari]

This source won't help you. it works for Xeno but will not work for Chrono. I looked at Chrono CD today and wasn't found anything like table of contents. You will need find it first. The easiest yay will probably finfd few files start sector and try to find this numbers on disk. This is pretty easy with WinHex. Remember you need start sector and size of file in byte to extract it, so this data surely on disk in some form.

By the way it was not me, who found found TOC for xeno. It was guys from http://www.sadnescity.it/. You could ask them. they most likely extract all files since they translate it.

[FaustWolf]

Akari, I've got what I *think* may be a Chrono Cross Table of Contents / Index file:

http://rapidshare.com/files/63794276/CC_Index_Possibility.zip.html

If you'd like to take a look at it, I can host it on another filesharing site if necessary; I don't think we can post attachments here, can we?

This is a giant hex string pulled from offset C000 in the Chrono Cross iso (sector 24), and it's 13 sectors long I think. It is somewhat similar to the Xenogears index file - it's got a byte repeat pattern, only the pattern happens every 4 bytes as opposed to every 7 bytes as in Xenogears.

[mav]

My guess would be that that's it - it contains ~6000 integer entries that look like offsets. But then, You should have filenames list somewhere, too...

[FaustWolf]

By filenames list, M4v3R, do you mean there should actually be filenames sitting right in the ASCII code?

[mav]

Well, maybe not in pure ASCII, mayber they are encoded in some way. That's only my random guess, because I never dived into PSX game hacking that much. Did you tried to search for some file names in the image? You could use Scan function in Translhextion editor (I don't know if others have this function) to scan the image for text not only encoded in ASCII, but also shifted by some amount of bytes.

I guess "kernel" would be a good string to search for (if you consider that square used this as name for directories/files many times).

[Akari]

Well, maybe not in pure ASCII, mayber they are encoded in some way. That's only my random guess, because I never dived into PSX game hacking that much. Did you tried to search for some file names in the image? You could use Scan function in Translhextion editor (I don't know if others have this function) to scan the image for text not only encoded in ASCII, but also shifted by some amount of bytes.

I guess "kernel" would be a good string to search for (if you consider that square used this as name for directories/files many times).

There is no such thing as file names as I believe. Game internally don't use filenames for sure. Even FFVII don't use file names although file system has been left on disk..

[FaustWolf]

I've run a search on both kernel and KERNEL and found nothing. However, there does seem to be a .EXE starting approximately at offset 13000 of the first Chrono Cross disc. Additionally, there's a short area of the disc that reads in ASCII:



The individual files this slice of hex seems to refer to - files with .WAA, .WEP, or .KMD extensions - aren't readily detectable from the ASCII I believe. Though I'll check again later. Could this info be useful potentially? Both the .EXE header and the chunk of ASCII shown above occur very far away from where I believe the Table of Contents is.

[ZeaLitY]

And if it's relevant, I've got the Square's Preview Demo Disc, with the file structure intact. The director for "KID", or the Chrono Cross demo, has these files in it:

KID.EXE
CRONO.HED
CROCRO.IMG
TITLE.XAM
BLOOD.XAM
TIDAL.XAM
END.DAT

I can't rip the XAM ones (I get some "invalid MS-DOS function" error). CROCRO.IMG is the actual game of course, and the others I've hosted here in case they are of use:

http://chronofan.com/Black/Cross/CRONO.HED
http://chronofan.com/Black/Cross/END.DAT
http://chronofan.com/Black/Cross/KID.EXE

[yoshi314]

There is no such thing as file names as I believe. Game internally don't use filenames for sure. Even FFVII don't use file names although file system has been left on disk..

you might remember MPQ files from blizzard games (you know, warcraft, starcraft etc.). they are said to use one-way filename hashes instead of filenames. just thought that would be a good clue.

[mav]

About index file you posted:

1) The differences between offsets are relatively small. Smaller than 256 in general. That could mean that if these are considered to be offsets to files in bytes, then they can't be files, they're to small (I didn't found any bigger differenes).
2) Last number (offset?) is 0x30D7372, which is 51 213 170 bytes. Too small for whole CD index.

Considering this, maybe it's a part if some sort of archive (like LGP in FF7)? That could make sense, but these numbers are not incrementing in normal manner. For eg.:

Code: [Select]

00 50 16 00 3B 60 16 00 A8 75 16 00 B9 A1 16 -->80 00 A1 16 80 00 A1 16 80 00
A1 16 80 00 A1 16<-- 00 D2 AE 16 00 BC B7 16 00 BB C3 16 00 BC CC 16 00
The marked pattern, especially the 80 in (imo) most significant byte of integer repeats itself in few places. It's like there's +80h to most significant byte in these places.

Any ideas?

[halkun]

You need to look at the programmer's perspective.

Nor that I'm a software manager, you have to see what Square wanted and then how the programmer implemented it. Let's analaze a little about the history of what Square is doing...

The technology being employed here is called "obfuscation". It's the art of hiding data. The problem is that you can only hide the data so far before it starts to impact performance. The PSX CPU was only $5.99 from LSI. When you are spending so little on a CPU, you don't have a lot of power left over to dig data out of some obscure format.

During the development of FF7, it was discovered that direct sector access was the only way to get the performance they needed. Even though the PSX supported loading files in the BIOS, it was very slow and locked the entire system until the fire was loaded. A lot of the 1st generation games simply put a "Now Loading" screen up and let the BIOS do all the work.

Because the FF7 kernel was multitasking, it was decided that creating a threaded load routine in the game was better. The reads were still done though the BIOS via a syscall, but it was only a sector at a time and the syscall would promptly give control back to the kernel when it was done with. The other major thread was the music thread, this made sure that music played during disk accesses, so it didn't look like it was doing "nothing". Neat trick.

As evidenced, with the lack of filenames, you needed to make some kind of index so the computer wouldn't get lost. Looking at the data from FF7 you can see the process that the disk was mastered.

1) When it was time to build a disk, all the uncompressed data was preloaded in a particular data structure. I'm going to guess much like how it was on the original disk.

2) Compile-time - The executables is built, it's length is recorded. Because these can be so dynamic, my theory is there is no sector data in this file. After the compile is completed, it's length is recorded.

3) PSX data created : The ANI, TIM, RSD, HRC, and all the other PSX data is created from the source assets.

4) Compression - Pass 1 - GZIP : All the files that are not sector critical are gzipped up. The compressed and uncompressed lengths are recorded.

5) Compression - Pass 2 - LZS :  Here all the files that are going to be sector critical that need LZS compression are done. After they are compressed the lengths are recoded.

6) Concatenation - Headers are made for all the files that need to be stuck together. They are then done so. Sector locations are kept blank.

7) File finalazation - All files are manifested and are in the final form that will be going on the disk, minus sector information.

Preburn - A disk image is created on a hard drive completely blank. Blank indexes are written first. Then, when a file is copied to this image, sector information is written in the files that need it, and recorded in the master index. Each file gets this treatment.

9 ) Burn - The disk is mastered.

Now that does this mean, First this meas that the files written higher up on the CD-ROM were written first, and therefore most location-critical. This means that the master index is going to be first, or last. If someone has the time, could you see the order that the filesystem was written in? (Not by time, but by sector) I bet you are going to find some interesting surprises.

By that, we can then figure out how the other Square games were authored.

[FaustWolf]

Thanks for the explanation on that Halkun - I have to digest it all still, but I'm glad to know that sort of thing since I have a personal interest in PSX ripping in general.  And thanks everyone for your input on this particular issue.

It appears that we've probably found a way to bypass the need for an index file altogether for Chrono Cross -- we've rediscovered a disc dumping tool already written by Yazoo!

Ramsus at the Compendium has modified the code to make it run more effectively, and we've apparently got all files on the first disc dumped! If anyone's interested in what's going on at the Compendium, the relevant posts start here:

http://www.chronocompendium.com/Forums/index.php/topic,4729.msg83045.html#msg83045

Now that we've got the files, we'll need to worry about decompression of models and other data - that's the hard part, and we'll probably be back here with other questions. Yazoo created decompression tools as well, but we're unsure yet if they work for all the various file types that are in Cross.

[ZeaLitY]

Edit: FW beat me to it, but I"ll leave my old post here:

~

We had a breakthrough with Yazoo's CC hacking tools. Ramsus was able to compile them with a few changes, and they apparently dump 650 MB worth of assets.

Now, the task ahead is to slowly figure out what the other executables do, and determine what's compressed. CCTools is missing the source file "lzss-dec.cpp", so we'll probably have to try and track down Yazoo. Ramsus also plans to clean up some of the other executables; it took a couple small hacks to get the dump working.

Anyway, no telling whether documentation exists or whether the index can be inferred or found in those programs. It's great that we were able to rip everything, but I guess the real legwork starts now. FW and I are going to create a page on the Compendium's encyclopedia to record all our findings and documentation.

Thanks for the explanation post. This is all pretty fascinating.

[Ramsus]

Other than being messily hacked together and completely missing a file, there isn't anything wrong with any the code when compiled as C++ (if I were the author, I'd have left it as is too; it works, and it's short enough not to matter). I only made a few minor/trivial usage changes (since it has a lot of hardcoded filenames and what not that make a lot of assumptions about the directory layout the user is using) along with some minor changes to make it compile as plain C (also unnecessary, but useful for those without C++ compilers), and really, I'm only just now beginning to read the code. When I find the actual game, I'll be able to actually see what the code is looking at and get a complete picture.

From just skimming it though, the code just pulls out some kind of index of the files from the ISO and then walks through the index and copies each file's contents one-by-one. In other words, we haven't bypassed anything. Someone else just happened to find the index for us.

[FaustWolf]

Ah, the Compendium is quickly migrating over here.

Yeah, it'd be interesting to get a hold of Yazoo and find out how he processed the original TOC/index file. Is this sort of dumper possible for every game CD, provided you know how to tailor the code to each game's individual file structure, I wonder?

[ZeaLitY]

The trick now is going to be getting these utilities to work:

http://chronofan.com/Black/Other/Chrono_Cross_-_Translation_tools.zip

They're Terminus Traduction's final set of utilities; most of the basic foundation was what Yazoo did in his CCTools.

[Ramsus]

The trick now is going to be getting these utilities to work:

http://chronofan.com/Black/Other/Chrono_Cross_-_Translation_tools.zip

They're Terminus Traduction's final set of utilities; most of the basic foundation was what Yazoo did in his CCTools.

Honestly, if I had known about those, I would've saved myself the trouble of even looking at the source code of the previous program. It seems like all the work there is pretty much done, and all that's left is reading the instructions.

Now you guys can start looking for the model files and figuring out what format they're in.

[halkun]

The models are most likely going to be in a binary form of RSD/HRC. The same as FF7, because that was the native format used by Psy-Q

[FaustWolf]

Halkun, I'm now aware of the following Qhimm wiki entries on RSD and HRC format:

http://wiki.qhimm.com/PSX/RSD
http://wiki.qhimm.com/PSX/HRC

Do you happen to know offhand if there's any more info on these formats, and do you have any tips on what these formats would look like in hexadecimal? Would they have a consistent file header, you think?

@Ramsus & Zeality: Whoa! I could have sworn I came across Yazoo's Translation Tools on the Compendium earlier, but I think they were uncompiled. These look great! Anyone know whether they should be run on the entire disc image or on the .OUT files instead? I'll try both.