Let's talk about pointers in PSX Square games

[FaustWolf]

Much has been said about text pointers for translation hacking, but I've seen precious little about the pointers a game engine uses for other objects, such as character models and textures. I've been searching every nook and cranny of the Chrono Cross disc 1 game image for pointers to the battle and overworld character model data, but to no avail. I'm 99% confident that such pointers exist, because if I, say, overwrite a given character model with another model that has much more data and spills over the sector boundary at which the original model normally ends, the game freezes on me. A model swap iin which the new model is less than or equal to the file size of the original model works just fine, on the other hand.

The only place I can find pointers to the battle models is in the game image's Table of Contents, but it appears that the game engine does not "interact" with this giant pointer table in any meaningful way -- while it helps immensely in dumping all files from the CD, changing the Table of Contents to match file alterations does nothing to solve the problems associated with those alterations, and a major alteration to the Table of Contents absent any other changes to the game image causes no problems. If anyone's interested, there's some notes on Chrono Cross' Table of Contents here: http://www.chronocompendium.com/Forums/index.php/topic,5181.0.html

Given the difficulty I've had in locating pointers to overworld and battle models, it is my great suspicion that those pointers may be embedded within the battle script and fieldscripts, which may in turn be LZSS compressed (hence my difficulty in finding the pointers straight-out). My question for the qhimm community is, has there been any investigation of the pointers to model data and other graphical objects in the Playstation versions of Xenogears, FF7, FF8 and FF9? I suppose FF9's pointer system would be most applicable to Chrono Cross given the overlapping development periods.

Also, is the expectation of finding pointers to various objects within uncompressed fieldscripts and battle scripts realistic? I figure if there's a command that says, "load model ()" it would make sense that an "oh, by the way, the models start here, here, here, and here..." list would directly follow. However, this is an aspect of game hacking to which I am completely new.

[Destiny]

what a strange but interesting avatar you got 

[FaustWolf]

  It's from a little-known strategy/RPG called Brigandine.

I've actually found what *might* just be a huge, general pointer table toward the end of Chrono Cross' SLUS file. Thing is, the pointers appear to have a two-byte stride -- not very promising, even if they're sector counts, since the game image is over 0x26000000 bytes long. If the pointers *are* in sectors, however, the differences between pointers would allow for models, textures, etc. in many cases. Has anyone ever seen pointers at the end of a SLUS file before?

[Cyberman]

Much has been said about text pointers for translation hacking, but I've seen precious little about the pointers a game engine uses for other objects, such as character models and textures. I've been searching every nook and cranny of the Chrono Cross disc 1 game image for pointers to the battle and overworld character model data, but to no avail. I'm 99% confident that such pointers exist, because if I, say, overwrite a given character model with another model that has much more data and spills over the sector boundary at which the original model normally ends, the game freezes on me. A model swap iin which the new model is less than or equal to the file size of the original model works just fine, on the other hand.

So you need a pointer in the right direction?

The only place I can find pointers to the battle models is in the game image's Table of Contents, but it appears that the game engine does not "interact" with this giant pointer table in any meaningful way -- while it helps immensely in dumping all files from the CD, changing the Table of Contents to match file alterations does nothing to solve the problems associated with those alterations, and a major alteration to the Table of Contents absent any other changes to the game image causes no problems. If anyone's interested, there's some notes on Chrono Cross' Table of Contents here: http://www.chronocompendium.com/Forums/index.php/topic,5181.0.html

This might be a left over from game construction/compilation. As you know many Squaresoft games have internal tools to construct a game.  (Heck I have internal tools I use to construct calibration information icons etc. in my embedded system programs). This might be an artifact like the FIELD file in the FIELD directory in FF7.

Given the difficulty I've had in locating pointers to overworld and battle models, it is my great suspicion that those pointers may be embedded within the battle script and fieldscripts, which may in turn be LZSS compressed (hence my difficulty in finding the pointers straight-out). My question for the qhimm community is, has there been any investigation of the pointers to model data and other graphical objects in the Playstation versions of Xenogears, FF7, FF8 and FF9? I suppose FF9's pointer system would be most applicable to Chrono Cross given the overlapping development periods.

Also, is the expectation of finding pointers to various objects within uncompressed fieldscripts and battle scripts realistic? I figure if there's a command that says, "load model ()" it would make sense that an "oh, by the way, the models start here, here, here, and here..." list would directly follow. However, this is an aspect of game hacking to which I am completely new.

Well first let's look at similar games. Such as FF8 and FF9.
FF9 has sectors of data that are aligned on (SURPRISE) sectors. These are the actual field and other data. The same is likely with CC.  Models etc are more likely part of an even larger file that spans across sectors.
I suggest you find WHAT sector you are in when you find field or battle model data. Then search for that group of data in little or big endian format.
Sectors in the PS1 are split up into Minute:second:frame
There are 99 minutes 60 seconds and 75 frames.
You can find the minute second and frame number from the sectors header information (if you are looking at a raw ISO file).
This will give you the reference of files (which if I remember correctly already exists).  As for the offsets THOSE should be at the beginning of each data object.  It's likely that the field models for example are all lumped into one BIG file. I noticed a lot of redundancy in CC files. IE multiple textures of the same kind.  This would suggest that much of the data is organized by field FILE (FF7 does the same thing) the field files may be grouped by area.  IE what section of the map you are in.  I believe the indexes you so desire are likely not in a singular file but scattered throughout the field and battle data.

Cyb

[FaustWolf]

Sector numbers for the model and texture files I'm looking at are reported only in the Table of Contents (and in Little Endian), and this table must be a remnant of the production process as you say, Cyb.

What's really interesting about the battle models is that, while there's no "big file" header sitting right before the first model texture (textures are stored first in CC), the first model texture IS on the first sector boundary following the end of all the battle scripts. I should, perhaps, search for pointers relative to the beginning of the battle scripts, as they might be part of one big battle construct -- maybe.

The reason for the texture redundancy, to my understanding, is that duplicates of NPC models are stored in each "room" so the game can load them on an as-needed basis, helping to reduce load time I guess -- so it should be just the same as in FF7 judging from what you and halkun have told me. I can see where the pointers to those NPC models would be tucked within each individual room's fieldscript. I'm wondering if the pointers themselves would be compressed along with the fieldscripts; if I can't find any pointers in the field files, then I guess I have my answer.

BTW Cyb, do you happen to know if Square games typically use single-pass LZSS compression, or do they sometimes use multiple-pass compression?

[Cyberman]

Sector numbers for the model and texture files I'm looking at are reported only in the Table of Contents (and in Little Endian), and this table must be a remnant of the production process as you say, Cyb.

Artifact from the EVIL compilation engine they made. More seriously this may be loaded in when the game starts and kept in memory. Especially if it's < 128K (this isn't very large considering they have 2Megs of memory).

What's really interesting about the battle models is that, while there's no "big file" header sitting right before the first model texture (textures are stored first in CC), the first model texture IS on the first sector boundary following the end of all the battle scripts. I should, perhaps, search for pointers relative to the beginning of the battle scripts, as they might be part of one big battle construct -- maybe.

it would make sense if the encounter system (IE battle script system) worked via the field data. Each battle my guess is like FF7's battle scripts (or I should say similar in construction).  The bottom line is there are only so many ways to do things. It would make sense to load encounter data from the field script (IE you collide with something and an encounter occurs). The battle script is unique to the BATTLE number my guess. If the information is kept relative to battle script then likely all pointer references are somewhere in each battle script.  Each combatant behaves according to the script (special battles or regular battles both run according to a script).

The reason for the texture redundancy, to my understanding, is that duplicates of NPC models are stored in each "room" so the game can load them on an as-needed basis, helping to reduce load time I guess -- so it should be just the same as in FF7 judging from what you and halkun have told me. I can see where the pointers to those NPC models would be tucked within each individual room's fieldscript. I'm wondering if the pointers themselves would be compressed along with the fieldscripts; if I can't find any pointers in the field files, then I guess I have my answer.

It's simpler to sift through the pile in each room than to use numeric indexing like FF7 uses.  IE battle script might be relative to each room/field location (so for example when you go to the Komodo Isle and fight the baby Komodo's the mother komodo fight is in each field location that the baby Komodo's are in).

BTW Cyb, do you happen to know if Square games typically use single-pass LZSS compression, or do they sometimes use multiple-pass compression?

I could see no reason to doubly compress them. It wouldn't gain you anything really. In fact it would make the data larger I would think. If they used LZSS that is.  It's cheap and free.  If you can find enough data to determine if it's LZSS compressed or not that would be a start. It might take hand deciphering a few files (IE to see if they are compressed).

My guess is it might not be compressed at all if the bitmaps aren't.
The compression of bitmaps is quite significant with LZSS

Cyb

[FaustWolf]

Thanks again for the insight! 

All fieldscript and battle scripts in Chrono Cross are definitely LZSS compressed according to notes left by the French translation team. Interestingly, the scripts are tagged "SSZL," but maybe that's because the R3000 is reading them in Little Endian order?

The tools left by the French Translation team -- Terminus Traductions -- successfully decompress the dialogue text (also tagged "SSZL") but curiously their tools don't seem to recompress to the same degree that the files are originally compressed, which is why I inquired about the possibility of dual-pass compression. Doubly curious, on the other hand, is the fact that the game engine can perfectly read the dialogue even after (possibly flawed) recompression, which weirds me out. I'm going to spend some time figuring out the compression scheme manually as I did with the Front Mission 2 portrait graphics (I brought up that subject in an earlier thread, which you might find academically interesting: http://forums.qhimm.com/index.php?topic=7365.0), and hopefully acquire enough programming knowledge to write a compression/decompression utility. I gotta learn the tricks of the trade, and this should be a good exercise.

I did some investigation into Chrono Cross' room files, and to my sheer horror discovered that NPC models are far separated from their textures  . Until now I had thought that 3D model data and textures were always stored right next to each other, and the game engine simply knew to start with the texture and knew that the model would be the file on the following sector boundary. Since the models are stored first in the room files, then some intervening graphics, and then the model textures, I'm going to guess for now that the game engine looks for the models first, and then, perhaps, within the vicinity of the NPC models there's some pointers to the associated textures.

In any case, it seems more and more likely that each fieldscript and battle script may have its own set of pointers to character and enemy models. I thought that would be too redundant and space-wasting at first, but it would have some speed advantages, and then there's the fact that each room's dialogue script repeats the game's Debug Mode messages. I never thought about it when I was playing the game almost a decade ago, but now I know why I never saw a "now loading" screen in Chrono Cross.

[Cyberman]

Thanks again for the insight! 

Sometimes it's even useful

All fieldscript and battle scripts in Chrono Cross are definitely LZSS compressed according to notes left by the French translation team. Interestingly, the scripts are tagged "SSZL," but maybe that's because the R3000 is reading them in Little Endian order?

I think this is the true. It may be using the unix magic cookie type prefix to identify them, and yes since Sony uses little endian a lot it's likely that SSZL becomes LZSS to the system. If things are tagged that means that can be used as an ID and the original files may have been compressed and then converted to a file structure.  Packaging is the thing to think about.

The tools left by the French Translation team -- Terminus Traductions -- successfully decompress the dialogue text (also tagged "SSZL") but curiously their tools don't seem to recompress to the same degree that the files are originally compressed, which is why I inquired about the possibility of dual-pass compression. Doubly curious, on the other hand, is the fact that the game engine can perfectly read the dialogue even after (possibly flawed) recompression, which weirds me out. I'm going to spend some time figuring out the compression scheme manually as I did with the Front Mission 2 portrait graphics (I brought up that subject in an earlier thread, which you might find academically interesting: http://forums.qhimm.com/index.php?topic=7365.0), and hopefully acquire enough programming knowledge to write a compression/decompression utility. I gotta learn the tricks of the trade, and this should be a good exercise.

There are 2 systematic methods of compression. I believe you need to examine what method they used. Some methods yield significant improvements however (cough) this can only be determined by recompressing the original data and seeing if they are equivalent.

I did some investigation into Chrono Cross' room files, and to my sheer horror discovered that NPC models are far separated from their textures  . Until now I had thought that 3D model data and textures were always stored right next to each other, and the game engine simply knew to start with the texture and knew that the model would be the file on the following sector boundary. Since the models are stored first in the room files, then some intervening graphics, and then the model textures, I'm going to guess for now that the game engine looks for the models first, and then, perhaps, within the vicinity of the NPC models there's some pointers to the associated textures.

This would make sense if the game loads texture batches of data in chunks. IE an entire set of possible textures for a room is loaded instead of just a few. IE it's making texture packs I think.  Each texture is likely an enumerated type so part of the model information could be an enumerated type that is them used to look up the texture from a list of index information (into the TP).  It really depends on how they set up the battle engine structuring.

In any case, it seems more and more likely that each fieldscript and battle script may have its own set of pointers to character and enemy models. I thought that would be too redundant and space-wasting at first, but it would have some speed advantages, and then there's the fact that each room's dialogue script repeats the game's Debug Mode messages. I never thought about it when I was playing the game almost a decade ago, but now I know why I never saw a "now loading" screen in Chrono Cross.

Speed is an important factor.  It's likely the Chrono Cross kernel dumps the 'outside' and loads the battle scene engine then reloads the 'outside' much like FF7 does.
Cyb