Chrono Cross exploration & file structure

[Akari]

The models are most likely going to be in a binary form of RSD/HRC. The same as FF7, because that was the native format used by Psy-Q

FFVII and XENO format are not in RSD or HRC but in compiled format that almost ready to be sent to GPU. So it's better to read documentation about PSOne than RSD/RHC specification.

[dziugo]

The models are most likely going to be in a binary form of RSD/HRC. The same as FF7, because that was the native format used by Psy-Q

FFVII and XENO format are not in RSD or HRC but in compiled format that almost ready to be sent to GPU. So it's better to read documentation about PSOne than RSD/RHC specification.

I believe that was what halkun meant by saying "binary".

[Cyberman]

Ironically I know a bit about the CC save state format.
However my time is still a bit stretched these days.
If you are going to keep the information you find public this is one of the games I wanted to migrate to Q-gears.
As I thought a fair bit of this stuff was available, I could be wrong.
I thought Chrono Cross was released about the same time as FF9?

Cyb

[halkun]

Do you happen to know offhand if there's any more info on these formats, and do you have any tips on what these formats would look like in hexadecimal? Would they have a consistent file header, you think?

The compiled versions of the HRC/RSD will be a lot like the battle models/field models.

Here's the format on those

http://wiki.qhimm.com/FF7/Battle_model_format_%28PSX%29

[FaustWolf]

Thanx Halkun!

Yeah Cyb, FF9 and CC were both released in 2000 in the US. Maybe that means model formats and various other aspects of the games will be similar?

Not to get off topic, but how is Qhimm's FF9Viewer coming along? It's not available for download yet, is it? It looks to me that CC and FF9 game resources are pretty compatible from an aesthetic standpoint. I'm starting to get wild ideas.

Everything we're discovering over at the Compendium is publicly viewable, I think. It's just a matter of gathering all the utilities and notes into a single place, which is no small task now.

EDIT: We've freshly acquired battle model textures and files that *may* be main character battle models. I'm linking to a sample of three character data folders. The "5" files are definitely battle model textures, and there's eyeball TIMs in these folders as well. But what I'm really interested in are the "3" files. Can anyone give me an opinion as to whether there's model data in the "3"s?

http://rapidshare.com/files/64618291/MainCharFiles.rar.rar.html

[Cyberman]

Well the backgrounds used 16bit palletized TIMS that were loaded into the right section of the display memory (IE 320x240 == 16bit display leaving 704x 240 pixels to the right of the display to hold the background data in). The background format however is kind of unknown as I didn't get too far with it. However it looks like instead of the complicated system they used for FF7 they just stored the palettes and the background information (8bpp likely) all together and used quads to dump the background information to the display. I think they used the transparency bit for some of the transparent/walk behind parts of the palletized background.  However the backgrounds were pretty much one BIG TIM.

Cyb

[FaustWolf]

When you say "background," Cyb, what do you mean exactly? I'm thinking pre-rendered backgrounds, that's not what you're referring to, is it?

[Akari]

When you say "background," Cyb, what do you mean exactly? I'm thinking pre-rendered backgrounds, that's not what you're referring to, is it?

2D backgrounds is picture that you can see when looking at field. It tiled with 16x16 or 32x32 tiles. It depends on layer in FFVII. Data format of backgrounds in ffvii are like SPRITE packet to GPU without header. And there are 4 layers of background. 1 - always rendered at the bottlom, 2 - each tile has it's own depth (distance from screen) for correct sorting, 3 and 4 are dynamic layers that managing through script.

Look at http://q-gears.svn.sourceforge.net/viewvc/q-gears/trunk/src/ffvii/field/DatFile.cpp?revision=177&view=markup for GetBackground().

[FaustWolf]

Thanks Akari. Is QGears intended to dump files from games eventually, or is it an engine that plays the games from the disc?

And we've opened the files in the "rooms" directory finally at the Compendium. There's data in these that I'm told is 3D, which might be your walkmeshes - though I'm still not sure how to determine which files belong to which fields/dungeons/etc. in-game yet.

[Cyberman]

Pre rendered backgrounds indeed was what I was referring too.
Maybe I should add TCL or python scripting to my PS1 ISO viewer to make it easy to make data viewers for various games quickly.
The interesting thing about said idea is that even though each game has unique data access attributes and unique structures etc. They all use the same hardware.
I was perusing my Medievil game and noticed that this game had similar uses of data as all the other games.
If one is going to do TCL or python or any script engine to render content from games, the end method of rendering the data is mostly similar so such a methodology is not a waste of time.

3d data should be in several sets IE vertex pools (sets of 3d data ) etc. They are likely ALL 32bit aligned (3 vertices's and one dummy 16bit signed for example).  Keep this in mind and the format will be easier to decipher.  If you looked at the FF7 battle format information (I think I put most of that in, it's been a while), you'll notice the structures follow this explicitly.  Since a walk mesh for example is not visible 3d data but more a collision surface (IE gravity collision) it will be arranged whatever is convenient for the GTE to process and not the GPU. My guess is whatever arrangement was most convenient.

Q-gears is to play the data from the games.  Since the versions for the PC of FF7 and FF8 are buggy and are difficult to get to work on said platforms, and playing FF9 xenogears Vagrant Story etc. require an emulator.  This is a good alternative, as content distribution is verboten (IE textures and stuff being ripped from the game) it is a good idea to keep this in mind. As I've noticed a lot of people getting C&D'd projects because of this.  Lawyers will follow if you distribute anything that is from a copyrighted work.

As for enhancements that has been a difficult thing to contemplate.  As to remain legally safe it can only be done in vitro. New background content could be a problem.  Of course there is duke nukem 3d (eduke) with the 3d models instead of sprites.  I think it's possible to make a method to facilitate the content but not distribute it with Q-gears (as this might bring up trademark and copyright issues again).

Cyb

[FaustWolf]

Quick and silly question for you Cyb, or anyone else who can answer - is the PSX battle model format wiki describing what the data looks like when it's uncompressed? Meaning I'd have to take suspected model data from Chrono Cross and run it through an LZS decompressor (assuming that's the compression scheme CC uses, of course) to be able to compare to what's in the wiki?

http://wiki.qhimm.com/FF7/Battle_model_format_%28PSX%29

[Cyberman]

Quick and silly question for you Cyb, or anyone else who can answer - is the PSX battle model format wiki describing what the data looks like when it's uncompressed? Meaning I'd have to take suspected model data from Chrono Cross and run it through an LZS decompressor (assuming that's the compression scheme CC uses, of course) to be able to compare to what's in the wiki?

http://wiki.qhimm.com/FF7/Battle_model_format_%28PSX%29

The FF7 model data is compressed as a complete set. However I believe CC is uncompressed data.  It is likely similar to FF9's model format. You should look for a SECTOR that contains a TIM unaligned to the sector boundry. That will be a clue that it's either background data or model data. If you inspect the TIM and it appears to be skin data then THAT is your model.  In FF8 there was an 128 byte header and then a TIM or two. FF9 is similar. Although I don't know the specifics for the FF9 model data.

Cyb

[FaustWolf]

Ah, THANK YOU once again Cyb. So the character model textures are typically stored right along with the models then? I'm currently mining for model data in a section of the CD image that contains the character battle model textures, weapon model textures, and unknown data sandwiched in between that I suspect might be character and weapon models.

When you speak of TIM textures unaligned with the sector boundary, you mean that the TIM begins at some point after a new sector, correct? Would 00 "buffer" bytes fill the space in between the sector start and the TIM then? And are we talking about 2048-byte sectors or 2352-byte sectors? I'm working with a CD image that has 2048 bytes per sector I think; not sure if that changes things any.

And I know now that sound data is certainly stored similarly in FF9 and Chrono Cross as well (both sound file formats start with AKAO in ASCII). I'm starting to wonder just how closely the file formats in Chrono Cross resemble those in FF9. Is it possible that Qhimm's FF9Viewer program would be able to detect Chrono Cross data as well?

EDIT: By the way, I don't think I've mentioned it above, but Ramsus and Zeality figured out Chrono Cross' pre-rendered background format (at least as it pertains to the visible 2D stuff) over at the Compendium. There's not much documentation on it yet, but I can relay what we know so far here if anyone is interested.

[Cyberman]

I can add it to the wiki here or someone else can depends on how much time I can find.
Model data is likely in Mode 1 sectors so that would be 2048 sized.
The space between the sector beginning and the TIM is likely to be offsets to the TIM and model data.  It's very efficient to load everything that way so that the TIM files are loaded from memory straight into the GPU VRAM. Generally they've used a table of offsets to point to data later in the file.  So each DWORD sized value is very likely the OFFSET into the actual data.  The total file size is already known so ...   What's likely to follow the TIM is the bone data however there is no certainty of that.

Cyb

[halkun]

If you find data that repeats that has the GPU quad opcode in it, follow the data and see if it traces out a UV map on the TIM. That's how I found the texture maps in FF8.

Remember, the PSX can do Quads "out of the box" and does not need to be broken down into triangles.

[Vehek]

Well, I think we (the people from the Chrono Compendium) don't really know what a "quad" is.

[FaustWolf]

Heh heh. Very true, Vehek.   Visiting the Qhimm forums sure has been a learning experience!

We may luck out and not have to do the tracing Halkun suggests if I'm correct in my suspicion that model data is sandwiched between the texture files I'm examining. But that's a BIG if. Later on I'll get some data isolated (say, all of Serge or Kid's battle and weapon textures, with the possible model data in between) and post a download link here so everyone can take a look at it with me. I have yet to confirm that the data fulfills the conditions described by Cyberman above, but it looks promising.

BTW Cyb, when we're talking DWORDs, is that 32 bytes in length? Is word size hardware specific, i.e., should it be consistent across all PSX games?

[ZeaLitY]

I checked out the battle field OUTs for the hell of it. As expected, they contain the battle field textures, and each is prefaced with a 3-byte descriptive header. The header seems to be quick romaji. For instance, "kumo" goes before the cloud texture. Anyway, after removing the TIMs from the source OUT file, around 53 kb is left. The OUT file starts with "DRP", so I'm assuming what's left are the field model box and texturing instructions. I've attached a chart of the TIMs and their corresponding headers, along with the remaining OUT file here.

[FaustWolf]

Now for a report on the potential battle model / weapon model data.

http://www.chronocompendium.com/Forums/index.php?action=dlattach;topic=4770.0;attach=1950

Firstly, the link is a zip containing a chunk of data from the CD image that I have named "ModelMine1." It's just raw hex data, and its beginning corresponds to a fresh sector (2048 bytes/sector as always) in the CD image. The data fall in this order: A header; Serge's battle model texture; Serge's eyeball texture; unknown data; Serge's first weapon model texture; unknown data; etc. Also inside the zip is a copy of my datamap for "ModelMine1," which I reproduce below. The chunk seems to be 11 sectors long exactly, if that means anything.

Also included in the zip are Serge's battle model texture, eyeball texture, and the seven weapon texture TIMs that occur within the ModelMine.

DATAMAP: All offsets given in hexadecimal notation.

00000000 ~ 0000000F: 16 byte header
00000010 ~ 0000823B: Serge Battle Texture
0000823C ~ 00009467: Serge Eyeball Blink Texture
00009468 ~ 000097FF: 00 Buffer Bytes
00009800 ~ 00014847: Unknown Data (Battle Model?)
00014848 ~ 00015DC7: Serge Weapon Texture 1
00015DC8 ~ 00016BE7: Unknown Data (Weapon Model?)
00016BE8 ~ 00018167: Serge Weapon Texture 2
00018168 ~ 00018C5F: Unknown Data (Weapon Model?)
00018C60 ~ 0001A1DF: Serge Weapon Texture 3
0001A1E0 ~ 0001AF0F: Unknown Data (Weapon Model?)
0001AF10 ~ 0001C48F: Serge Weapon Texture 4
0001C490 ~ 0001D2EF: Unkonown Data (Weapon Model?)
0001D2F0 ~ 0001E86F: Serge Weapon Texture 5
0001E870 ~ 0001F687: Unkknown Data (Weapon Model?)
0001F688 ~ 00020C07: Serge Weapon Texture 6
00020C08 ~ 00021897: Unknown Data (Weapon Model?)
00021898 ~ 00022E17: Serge Weapon Texture 7
00022E18 ~ 00023FFF: Unknown Data (Weapon Model?)

After this begins my ModelMine2 file, which contains Kid's suspected battle model and weapon model data. I'll post that later for comparison if anyone's interested.

SUSPECTED MODEL FILE INFO: The offsets and file lengths may actually be one byte too "short" depending on whether I'm interpreting the hex code correctly. For example, the First "Weapon Model" listed below is described as 3615 bytes long. If it makes more sense, feel free to interpret the file size as 3616 bytes.

First "Model" : 45127 bytes long. Header begins "06 00 00 00 20 00 00 00"
First "Weapon Model" : 3615 bytes long. Header begins "01 00 00 80 48 00 5E 00"
Second "Weapon Model" : 2807 bytes long. Header begins "01 00 00 80 39 00 46 00"
Third "Weapon Model" : 3375 bytes long. Header begins "01 00 00 80 43 00 58 00"

The weapon "models" are consistent in their first 8 bytes, which may constitute a model file header; the first 8 bytes for the weapon "models" read:  01 00 00 80 xx 00 xx 00 (where "xx" represents a byte position that varies in value among the "models")

My notes on the data were hastily scribed, so if anyone wants clarification on any point, please ask. Once again, the "ModelMine" for Serge's (suspected) battle and weapon models and all the associated textures are in the attached zip if anyone would like to investigate.


ADDENDUM: Also, a question for Halkun or anyone else who understands PlayStation design/architecture: Are PsyQ opcodes for the Playstation GPU actually planted amongst the game data? I'm interpreting Halkun's previous post about quads as saying that I should be able to see the applicable opcode just before the model geometry, TIM textures, etc. Am I understanding this correctly?

If GPU opcode *is* in with the game data, is it also in the SLUS file? If both the SLUS file and the data following the SLUS file on the game CD contain opcode, and that opcode features the same call functions, then perhaps I could view the call functions in the SLUS file in hex, determine their ASCII signature, and then find the call functions further on in the game CD as well, pointing to model data.

Does that make any sense whatsoever? I've yet to read the PlayStation design documents, but will begin doing so in the near future. Oh, and another question -- what "language" are the PsyQ opcodes written in? MIPS Assembly language?

[FaustWolf]

I'm double-posting with full knowledge that such practices are discouraged. However, I feel it is necessary given the extraordinary length of my previous post.

I've found some very interesting ASCII blocks in Chrono Cross' SLUS file. First up...

Seems to be file paths. Is it possible that this is a signature left from the disc burning process, meaning there's MDE, FID, FND, and PRG files on the game CD? Anyone know what these file extensions even refer to?

Next up...

If I had to take a wild guess, I'd say these might be assembly language opcodes that tell the Playstation GPU how to handle model data, but of course "I R a NooB" at this sort of thing. Now, this wouldn't be functional opcode data, right? Just ASCII evidence that the opcodes are in the SLUS. I expect I'd find these elsewhere in the SLUS when I investigate with a disassembler such as PS2Dis. But I'm not sure PS2Dis knows to "tag" the various assembly language opcode functions with these labels. Anyone know of a full-featured PlayStation-centric disassembler that already does the trick?

I just wanted to throw this out there in case others have seen this sort of thing and can comment.

It's about time I delved into Halkun's PlayStation architecture/design doc -- grabbed it off Zohar.net just a few minutes ago. I shall return enlightened.

[Cyberman]

ADDENDUM: Also, a question for Halkun or anyone else who understands PlayStation design/architecture: Are PsyQ opcodes for the Playstation GPU actually planted amongst the game data? I'm interpreting Halkun's previous post about quads as saying that I should be able to see the applicable opcode just before the model geometry, TIM textures, etc. Am I understanding this correctly?

Not everything will have a GPU code, only the quadric's and triangles used in the model would use that.
I'm not positive but TIM et all use bios calls that set up a DMA into the GPU to load the data into the VRAM.

If GPU opcode *is* in with the game data, is it also in the SLUS file? If both the SLUS file and the data following the SLUS file on the game CD contain opcode, and that opcode features the same call functions, then perhaps I could view the call functions in the SLUS file in hex, determine their ASCII signature, and then find the call functions further on in the game CD as well, pointing to model data. [/qoute]Well actually it is more likely there is a buffer of where to load the model data.  I know the background files contain fixed load points in each file in FF7.  It's less likely GPU opcodes will be interspersed in the executable.  The code in the executable is more likely to know what to do with it and use it to set up the GTE to transform the data to be sent to the GPU instead.

Does that make any sense whatsoever? I've yet to read the PlayStation design documents, but will begin doing so in the near future. Oh, and another question -- what "language" are the PsyQ opcodes written in? MIPS Assembly language?

Halkun has a nice document he wrote "everything you wanted to know about the playstation but were afraid to ask" or something like that. I recommend reading the GPU commands. That is one way you can distinguish between quads and triangles in the data you are looking at.
Remember ALL CODES ARE 32 bits this includes the GPU so be sure to think that when you are looking for GPU codes.

What you have is part of the library for loading stuff into the GPU likely

Cyb

[FaustWolf]

Thanks Cyb! Yes, Halkun's "Everything You Ever Wanted to Know But Were Afraid to Ask..." is first on my reading list over the next few weeks. This is fun!

[halkun]

Short History on Psy-Q

Once Upon a time, Sony made an arcade board for Namco called "System 11". Namco used this arcade system and created games such as "Ridge Racer" and "Tekken". During the fall out between Sony and Nintendo over the cd-rom addon, Sony took the System 11 board, tacked on the PSX cd-rom technology, put it on a box, and called it "The Playstation"

Now Sony had a problem.

Sony is a hardware manufacturer. They are not coders, and for this new system, had no real game development kits to go along with it. They invited Psygnosis, a very popular video game company at the time to create a 1st party game and Sony wanted dibs on the development kit they created for it. Psygnosis created "Psy-Q" and went on to create the first "wipeout" game with it. Psy-Q was a good game library, and had the ability to abstract all the hardware away from programming. You called the Psy-Q commands, which in turn called the BIOS, which in turn commanded the hardware to do cool things.

A snag was hit early on when Sony made the decree that software companies could only go though the PSX BIOS or Psy-Q, when making the game. No direct hardware calls were allowed. (This was because they wanted to preserve a kind of "reference" that could put into a later machine for backwards compatibility).

Psygnosis thought because they made the dev kit, they were exempt from the "no direct hardware programming" rule. The first release of Wipeout talked directly with the PSX. Sadly when the first revision of the hardware came out, it crashed all the first-generation Wipeout games and Psygnosis had to make another version more compatible with Sony reference.

Psy-Q is a game library and toolset written in C, and was used on DOS. It used a modified gcc compiler. Looking on the internets, you can find copies of the Psy-Q documentation, but having is illegal. You can also find the library too if you look hard enough. This is why you will find programs that seem to use the same dataset over and over again. (TIM, for example, or SEQ) these are all Psy-Q data formats that are "native" to the PSX.

TIM = bitmap picture/texture format
HRC = hierarchy format
TMD = 3d object format
RSD = model+hierarchy+texture format (resource data, it's the whole "3D entity")
ANI = 3D Animation format
VAG/VAB = digital Audio format (group/single)
SEQ = Sequence (midi) data
STR = Streaming video format

You can see how this becomes a basis for a game. If you can find this data hidden on the disk, youhave an idea of how to "expand" it into something to use. Keep in mind: The PSX only had 4 meg, that's not a lot, especially when you realize that you have no dynamic libraries and most of the time Psy-Q was statically linked to the executable somehow.

That should start you off.

[FaustWolf]

That starts me off alright.  Thanks Halkun!

Now that I've figured out how to take advantage of PS2Dis' built-in hex viewer (offset disparities between that and the SLUS file as shown in my hex editor were giving me a headache for awhile), I should be good to go once I find out what the quad GPU opcodes look like in assembly language. Sharing that info on this board is strictly "Verboten," right? Meaning I'll have to figure it out from some documentation I downloa- well, nevermind.

Ooh, and that "ANI" format - do all PSX models use that for animation, or only TMDs? Do developers typically come up with their own animation formats as they do with their proprietary model formats?

[halkun]

There are two major animation formats that have been found, and both are used in FF7. One is a bitwise delta format that is used for the battle models, and is closely related to the Psy-Q animation format. The other is a "frame" format that simply holds all the angles for the bones in that frame.(used for the field characters) I don't know if the both come from Psy-Q. I know MGS used a special animation format that was a kind of compressed delta, and Square uses the modified AKAO format for sound. The Psy-Q animation format has support for not only delta positions, but also machine states, so it can get really complex.

Oops, let me define:

Delta - Change over time. As opposed to listing joint angles every animation frame, it only lists how much the angle has changed via a +/- value. It's smaller, but more complex to decode.

State machine: Basically a "mode" that a particular program runs in. When it comes to animation it can have a "stand" state, a  "walking' state", a "running" state, and a "jumping" state. These are all linked togeather to from a branch-loop.

stand->walk->stand->jump->stand->walk->run->stand.

I know we decoded the delta format, I think it's in the wiki. I'm not sure about the state machine though.

Talking about the PSX GPU here is fine. It's open and not a trade secret. Pretty much anything about the PSX is fair game except for the BIOS function names. Psy-Q, on the other hand, is a little more tricky. If it looks like you are copying out of the Psy-Q manual, that is a sure-fire way to get yourself banned. However, If you can get a clue about a format by analyzing the data, you should be in the clear.

Also, take the strings you find in executables with a grain of salt. Some of that data looks like garbage left over after the compile process. Sometimes compilers compile a program in memory and bulk-copy the finished heap to disk. When that happens it can "scoop up" chunks of the compile process as filler to round out the memory boundary. It's interesting that it might lead a clue to the compile process, but not necessarily anything useful about the executable.

P.S.-->   Oh and by the way... Why can't we talk about the Psy-Q/BIOS functions?
             Because Sony told me not to, that's why.

P.P.S. --> V hfrq gb jbex sbe Anzpb