Messages

1

General Discussion / Re: Chrono projects C&D'd (including technical research)

« on: 2009-05-12 18:49:56 »

I agree that the word "mod" should have been used, since "hack" makes us sound like we were trying to steal credit card numbers or something. It's probably my bad for not advising the core CE team more on the type of language to use. You see, the creators of CE were focused solely on this effort for years, and weren't quite steeped in the wider game modding culture; I got the feeling that they wanted to pour all their effort into one last project before moving on with their lives, whereas a lot of us have dreams of going further and modding other games.

However, I gave the phrase "ROM Hack" the benefit of the doubt since I also visit Romhacking.net frequently, and they don't seem to be attracting any unwanted attention despite the fact that they apply the same terminology to patching that we did. I'm rather sad that the admin of Romhacking.net is accusing the C&D of being fake. While badly worded for a C&D compared to what Chrono Trigger: Resurrection received years ago, it is indeed legitimate -- an admin did get through to SE legal to confirm, as explained over at RH.net -- and denying that legitimacy encourages the very confidence with which the Chrono Compendium forged ahead with its own projects.

2

General Discussion / Re: Chrono projects C&D'd (including technical research)

« on: 2009-05-12 02:28:52 »

The Chrono Compendium has never, doesn't, and will never, ever release ROMs. Everything was done through patches. Just a standalone mod, like what's been done elsewhere. Only an .IPS format patch would have been released.

I am curious, however, as to why you were under that impression grimmy.

3

General Discussion / Re: Chrono projects C&D'd (including technical research)

« on: 2009-05-11 03:52:23 »

Now, I don't mean to make everyone frightened and stop pursuing the art (and it is an art), especially with PSX-era work, which happens in far too few places. Consider it more of a reminder to back up your stuff or make contingency plans.

I imagine the legal repercussions of a C&D are nil as long as you comply with it. I think the Chrono Trigger: Resurrection folks went on to fine careers in their private lives. Now, if you wanted to hire a lawyer to fight it and lost, then it might go on your criminal record if you lost the case.

4

General Discussion / Re: Chrono projects C&D'd (including technical research)

« on: 2009-05-10 23:29:05 »

It seems obvious in retrospect of course, but the thing is, the Chrono Compendium had actually released a smaller-scale modification over a year ago -- also virally advertised -- without making waves, not to mention a retranslation before that, plus the demo for the work that ultimately garnered the C&D. I think the real kicker this time around was the existence of CT:DS; Square's apparent fixation on copy protection in the C&D order suggests to me that they thought we were modifying CT:DS (a recent product with heavy copy protection), whereas we were really doing an SNES-era modification. Crimson Echoes wasn't all that different from modifications that have been happening for a decade now, though it was certainly an ambitious modification.

Hah, and to think we even got our feet wet in PSX-era modification with Chrono Cross. Interesting that Square Enix didn't pick up on that specifically; maybe because we weren't really that far along, or maybe because some mole really did point CE specifically out to SE as suggested above.

5

General Discussion / Re: Chrono projects C&D'd (including technical research)

« on: 2009-05-10 19:33:46 »

In several forums we've encountered doubts that this is actually happening. ZeaLitY has posted the header of the email sent to him from Square Enix legal, if anyone wants to double-check.

6

General Discussion / Re: Chrono projects C&D'd (including technical research)

« on: 2009-05-10 17:58:18 »

Interesting, I wasn't aware of yaz0r also running into this, though I remember he had to suddenly stop KH-related technical research. Guess that explains it.

Up till the C&D the Compendium received, two playable, original, 3D Chrono projects started by other groups were canned by Square. A section of the Compendium forums was engaged in technical research very similar to what's discussed in the qhimm Technical board and at Romhacking.net. Although Crimson Echoes was intended to be a new game, complete with new story and various other features, it was really just modification of what was already in the ROM for the most part.

It's scary, because Square Enix is literally forbidding us from exploring the game engine and graphics & sound resources. If we so much as open a hex editor and trade information about file formats in Chrono games, we're violating the terms of Square Enix's C&D.

7

General Discussion / Chrono projects C&D'd (including technical research)

« on: 2009-05-10 04:59:54 »

For the admins here, I'm wondering if you'd care to receive a bunch of Chrono Cross PSX notes?
http://www.chronocompendium.com/Forums/index.php/topic,7396.msg159819.html#msg159819

Square Enix is really on the warpath. Hopefully it only extends to Chrono projects, but I figured I'd give you guys a heads-up in case it proves useful.

8

Scripting and Reverse Engineering / Re: Ehrgeiz Data Exploration!

« on: 2009-04-04 01:00:39 »

I used PVV3 as my VRAM viewer and pSX (psxfin.exe) as my VRAM dumper. The methodology is as follows:

PVV3 source:
http://www.romhacking.net/forum/index.php/topic,5092.msg89496.html#msg89496

To do a VRAM dump in pSX, you go to
*Debug > Monitor > r3000
*Debug > Break (to freeze the game's processes at the point you want stuff to stop moving in memory)
*GPU > Save VRAM


As for investigation of the raw file format, yaaargh!   I'd really hate to think that the textures are compressed (they weren't in FF8 or FF9 after all, and those were roughly contemporary to Ehrgeiz IIRC), but that may be the case because I'm unable to track down the uncompressed VRAM texture in the game image. Granted, that's difficult to do though because PVV's address lookup feature hasn't been implemented yet (I don't even know whether Creaothceann is still working on it), and it can only save to .BMP and .PNG, and not as raw data.

The best I've been able to do so far is compare segments of data in the .BMP capture of Sephiroth's texture to the raw VRAM data and find a match. That matched data does not exist as such in the game image. So the best I can do so far is inform you that the data "may be compressed." 

9

Scripting and Reverse Engineering / Re: Ehrgeiz Data Exploration!

« on: 2009-04-02 18:58:45 »

Oh, I got them from VRAM dumps. Since there's only a handful of characters in Ehrgeiz, I'd be happy to rip them all. Look for them sometime this weekend.   

Ripping the monsters from the dungeon sidegame would be more difficult though, since there's more of them. However, I recall that the end bosses were particularly beautiful. Are you guys interested in those textures too?

10

Scripting and Reverse Engineering / Re: FF8 Models, How did Qhimm rip them?

« on: 2009-02-15 18:38:14 »

Whoa, huge thanks JWP! Since I'm still dedicated to the Chrono Cross model project over at the Chrono Compendium for awhile, I'm going to reproduce what you wrote in a PM to me so everyone can see it:

This is what I've managed to gather so far:

S1: Skeleton
S2: Geometry
S3: Animation (there's a subheader with no. frames)
S4: Unknown
S5: Unknown
S6: Unknown
S7: Monster Info section (I decoded a lot of this, see below)
S8: Suspected of being some sort of Battle Script (it contains text that's displayed in the battle)
S9: Some sort of sound data
SA: Some sort of sound data
SB: Texture

By the way, I believe that the duplicate pointers are because a section is unused (ie. zero length) so the section after ends up with the same pointer.


struct Item {
   BYTE id;
   BYTE amount;
};

struct Action {
   BYTE type; //??? 0x02 = generic magic
   BYTE animscript; //???
   BYTE id; //???
   BYTE unk;
};

struct MonsterInfo {
   char name[24]; //FF8 encoded - length might be less than this

   BYTE HP[4]; //HP = (HP[0]*x*x)/20 + (HP[0] + 100*HP[2])*x + 10*HP[1] + 1000*HP[3], where x = monster lvl
   BYTE unk0[24];

   Action RosterL[16];
   Action RosterM[16];
   Action RosterH[16];

   BYTE Unk1[4];

   BYTE Card[3]; //1st Byte = Card Drop, 2nd Byte = Carded into, 3rd Byte = Rarely Carded into, 0xFF = no drop/can't card
   BYTE Devour[3]; //Low, Med and High lvl Devour effect id's

   BYTE unk2[6];

   USHORT LowLvlDraw[4];
   USHORT MedLvlDraw[4];
   USHORT HighLvlDraw[4];

   Item LowLvlMug[2];
   Item LowLvlRMug[2];
   Item MedLvlMug[2];
   Item MedLvlRMug[2];
   Item HighLvlMug[2];
   Item HighLvlRMug[2];

   Item LowLvlDrop[2];
   Item LowLvlRDrop[2];
   Item MedLvlDrop[2];
   Item MedLvlRDrop[2];
   Item HighLvlDrop[2];
   Item HighLvlRDrop[2];

   BYTE unk3[20];

   BYTE ElemRes[8]; //These are how resistant the monster is to each element 0 = very weak
               //0 - Fire
               //1 - Ice
               //2 - Thunder
               //3 - Earth
               //4 - Poison
               //5 - Wind
               //6 - Water
               //7 - Holy
   BYTE StatusRes[20]; //Status resistances 0xFF = immune
               //0 - Death
               //1 - Poison
               //2 - Petrify
               //3 - Darkness
               //4 - Silence
               //5 - Beserk
               //6 - Zombie
               //7 - Sleep
               //8 - Haste
               //9 - Slow
               //10 - Stop
               //11 - Regen
               //12 - Reflect
               //13 - Doom (red timer) - command/doomtrain
               //14 - Slow Petrify? (white timer) - cast using doomtrain
               //15 - Float
               //16 - Confuse
               //17 - Drain
               //18 - Huh
               //19 - Huh
};

11

Scripting and Reverse Engineering / Re: The Model Project (Currently FFVIII and FFIX)

« on: 2009-01-17 17:49:34 »

Oh my lord, that's beautiful. That's what I get for not checking the qhimm wiki first.

Has investigation of the FF8 model format been wikified anywhere? I didn't see it in the qhimm wiki when I checked last, but I could be wrong, maybe...

X-Dina, let's wait a bit and see if Zidane or Zande pop up. I know they had model viewing programs for FF9.

12

Scripting and Reverse Engineering / Re: FF8 Models, How did Qhimm rip them?

« on: 2009-01-11 04:38:28 »

Oh snap. Okay. Yeah, I can test these then, just a matter of nabbing enough free time. Once I finish some odds and ends from the Chrono Cross model project over at the Chrono Compendium, I'll move straight into this I guess.

Any qhimm regulars interested in helping us with a massive FF8 model project to make these babies viewable once and for all? For one thing, I think we'd need someone to code the model reader/Blender plugin because the person who did that for Chrono Cross is extremely strapped for time.

13

Scripting and Reverse Engineering / Re: FF8 Models, How did Qhimm rip them?

« on: 2009-01-11 00:29:23 »

Wowza!   Just did some some investigation of the FF8 PC (as in Personal Computer version) battle models, which appear to be labeled with the c0mXXX.dat convention, with "XXX" representing the character model number I suppose. I'm going to link to what I believe are three examples of battle models:

http://www.megaupload.com/?d=86TZWUV2
c0m120 would be Raijin's battle model for reference (the dark-skinned guy with the kickass beat-'em-up staff or whatever his weapon is called).

I'll go through what I've found out so far -- halkun probably knows way more because he used FF8 UV maps to teach me how to interpret the Chrono Cross UV Maps to begin with. 

The model header, read byte-by-byte (sorry, I'm a hexplorer, not a programmer ) appears thus:

Model Header: 0x34 bytes long.

#S #S #S #S S1 S1 S1 S1 - S2 S2 S2 S2 S3 S3 S3 S3
S4 S4 S4 S4 S5 S5 S5 S5 - S6 S6 S6 S6 S7 S7 S7 S7
S8 S8 S8 S8 S9 S9 S9 S9 - SA SA SA SA SB SB SB SB
EF EF EF EF

S1: Section 1 offset relative to header, Unknown
S2: Section 2 offset relative to header, Something suspected of containing a UV Map and Vert Pool
S3: Section 3 offset relative to header, Something suspected of containing a skeleton
S4: Section 4 offset relative to header, Unknown
S5: Section 5 offset relative to header, CAN BE DUPLICATIVE OF S4!? It's slightly possible that it's a UV map too.
S6: Section 6 offset relative to header, Unknown
S7: Section 7 offset relative to header, Unknown, but has an 8-byte header that seems to be an ASCII tag.
S8: Section 8 offset relative to header, Unknown but has a structured subheader.
S9: Section 9 offset relative to header, 8-byte header, then AKAO tag -- sound data!?
SA: Section 10 offset relative to header, leads directly into an AKAO tag or duplicative of SB.
SB: Section 11 offset relative to header, The Texture!!
EF: End of File!

So it appears that the model files themselves contain the texture(s) and also some sound data(!?). What really, really confuses me at the moment with regard to the model file header is that some pointers in the table are duplicative! Apparently, there's enough room for dec11 or 0x0B sections, but not all of them are always used. Sometimes there's a need for two AKAO (sound data) sections, for example, but if not, the slot normally attributed to the second sound section is just given to the texture section. Anyone know what's up with this? Is this a common practice?

I suppose I might as well dump what I know about the last section of the model -- it's a Classic .TIM, and I wouldn't be surprised if you ran it through PSicture and achieved readable results if you stripped off the header. Note that models use a number of 128x128 textures depending on how detailed they are I guess. Raijin uses 2 textures, but there's another model in the linked .RAR that appears to use 5(!!) textures.

Texture format:
Header of (4+#T*4) bytes

#T #T #T #T T1 T1 T1 T1 - T2 T2 T2 T2 ... ...
EF EF EF EF

Where...
 #T = Number of Textures applied to the model.
 T1 = Offset to the first texture relative to the section header
 T2 = Offset to the second texture relative to the section header
 ... etc
 EF = End of File! relative to the section header

Each .TIM seems to be either identical to or extremely reminiscent of what I call a "Classic .TIM" in this Compendium wiki.       

Big thanks to X-Dina for the models; he could provide way more for further investigation, probably.


Since I don't actually own FF8 PC, the best I can do is properly detail the section subheaders, so others will know what to avoid hitting accidently when they try trial-and-error experiments.

14

Scripting and Reverse Engineering / Re: FF8 Models, How did Qhimm rip them?

« on: 2009-01-09 20:48:21 »

I'm not sure if it was Qhimm, halkun, or Cyb who posted pics of some ripped FF8 models a couple years back, but it was done somehow -- I remember seeing a beautiful screenie of the Ifrit summon in a model editor. I imagine the PC version will be tons easier to work with than the PSX version in this regard.

I don't believe the models were ever completely figured out though, or there was otherwise some problem hindering progress. Reminds me that I need to get around to studying the Chrono Cross model animation format, heh heh...

15

Scripting and Reverse Engineering / Re: A simple question about FFIX...

« on: 2008-06-01 23:49:44 »

I, too, am a graphics fanatic without programming capability (seems you have more than I if you know a thing or two about C#).

My own lack of hacking experience relative to such gurus as halkun and Cyb leaves me somewhat myopic in directing you on where to start, but I'd have to recommend first ripping a FFIX game image with IsoBuster. The process I link to here from the Chrono Compendium should be equally applicable to FFIX, since I used it to view FFIX textures at one point I think. Free 600+MB of space on your hard drive, download IsoBuster, and pop the first disc into your CD drive to rip the game image in 2048 byte/sector format so that you have a perfect copy of the disc on your hard drive. Note that FFIX is actually somewhat kinder to the novice hacker than Chrono Cross because you could simply copy&paste the FF9.img file from the CD to your hard drive -- and in fact, I suspect this is the very file that some of the dumping tools laying around in this forum act on.

Whichever process you decide on (IsoBuster will give you the executable, or SLUS file, FF9.img and the .STR movie files in their assembled order as they appear on the game CD, whereas FF9.img will give you the meat of the game data only), search for PSicture and use it to scan the game image. I suspect all textures in FF9 are classic .TIM files, which means PSicture will allow you to view all of them. I believe this is the case with FF8 at least.

I would also recommend that you download Cygnus Hex Editor - Free Edition, which has the most powerful search features I've seen on a free hex editor. Arguably it is inferior to Hex Workshop, but I feel that all my own hacking needs are more than met by Cygnus.

Zidane2 provides some documentation on the FF9 model format in this thread. I'm unsure how complete the documentation on FF9 models is at the moment, but I am certainly interested on completing that knowledge at some point in the near future. However, my attention is primarily focused on Chrono Cross models at the moment; come to think of it, X-Dina, you may benefit extraordinarily from the knowledge halkun and Cyb already shared during the Chrono Cross Model Project. See everything from top to bottom here, and you can learn as I did:
http://www.chronocompendium.com/Forums/index.php/topic,4770.0.html

16

Scripting and Reverse Engineering / Re: A simple question about FFIX...

« on: 2008-06-01 05:11:12 »

Here's a list of recent topics that may interest you just in case you haven't seen them already:

http://forums.qhimm.com/index.php?topic=5481.0
http://forums.qhimm.com/index.php?topic=7285.0
http://forums.qhimm.com/index.php?topic=6968.0
http://forums.qhimm.com/index.php?topic=7168.0

Also try running a search on Zidane2, since both he and Zande are the current FF9 model viewing experts to my knowledge. I haven't kept up on the latest FF9 developments lately, so I'd be very interested in seeing where progress currently is on viewing those models. Has the animation format been decoded yet, for example?

X-Dina, let us know where your hacking abilities currently stand. If you've never seen the sheer, utter beauty of a Squaresoft model texture, I will have to introduce you to IsoBuster and PSicture -- the very first tools any young hacker should be introduced to, IMHO. They're free to boot.

17

Scripting and Reverse Engineering / Re: Let's talk about pointers in PSX Square games

« on: 2008-05-19 02:00:09 »

Thanks again for the insight! 

All fieldscript and battle scripts in Chrono Cross are definitely LZSS compressed according to notes left by the French translation team. Interestingly, the scripts are tagged "SSZL," but maybe that's because the R3000 is reading them in Little Endian order?

The tools left by the French Translation team -- Terminus Traductions -- successfully decompress the dialogue text (also tagged "SSZL") but curiously their tools don't seem to recompress to the same degree that the files are originally compressed, which is why I inquired about the possibility of dual-pass compression. Doubly curious, on the other hand, is the fact that the game engine can perfectly read the dialogue even after (possibly flawed) recompression, which weirds me out. I'm going to spend some time figuring out the compression scheme manually as I did with the Front Mission 2 portrait graphics (I brought up that subject in an earlier thread, which you might find academically interesting: http://forums.qhimm.com/index.php?topic=7365.0), and hopefully acquire enough programming knowledge to write a compression/decompression utility. I gotta learn the tricks of the trade, and this should be a good exercise.

I did some investigation into Chrono Cross' room files, and to my sheer horror discovered that NPC models are far separated from their textures  . Until now I had thought that 3D model data and textures were always stored right next to each other, and the game engine simply knew to start with the texture and knew that the model would be the file on the following sector boundary. Since the models are stored first in the room files, then some intervening graphics, and then the model textures, I'm going to guess for now that the game engine looks for the models first, and then, perhaps, within the vicinity of the NPC models there's some pointers to the associated textures.

In any case, it seems more and more likely that each fieldscript and battle script may have its own set of pointers to character and enemy models. I thought that would be too redundant and space-wasting at first, but it would have some speed advantages, and then there's the fact that each room's dialogue script repeats the game's Debug Mode messages. I never thought about it when I was playing the game almost a decade ago, but now I know why I never saw a "now loading" screen in Chrono Cross.

18

Scripting and Reverse Engineering / Re: Let's talk about pointers in PSX Square games

« on: 2008-05-16 00:54:14 »

Sector numbers for the model and texture files I'm looking at are reported only in the Table of Contents (and in Little Endian), and this table must be a remnant of the production process as you say, Cyb.

What's really interesting about the battle models is that, while there's no "big file" header sitting right before the first model texture (textures are stored first in CC), the first model texture IS on the first sector boundary following the end of all the battle scripts. I should, perhaps, search for pointers relative to the beginning of the battle scripts, as they might be part of one big battle construct -- maybe.

The reason for the texture redundancy, to my understanding, is that duplicates of NPC models are stored in each "room" so the game can load them on an as-needed basis, helping to reduce load time I guess -- so it should be just the same as in FF7 judging from what you and halkun have told me. I can see where the pointers to those NPC models would be tucked within each individual room's fieldscript. I'm wondering if the pointers themselves would be compressed along with the fieldscripts; if I can't find any pointers in the field files, then I guess I have my answer.

BTW Cyb, do you happen to know if Square games typically use single-pass LZSS compression, or do they sometimes use multiple-pass compression?

19

Scripting and Reverse Engineering / Re: Let's talk about pointers in PSX Square games

« on: 2008-05-15 20:41:53 »

  It's from a little-known strategy/RPG called Brigandine.

I've actually found what *might* just be a huge, general pointer table toward the end of Chrono Cross' SLUS file. Thing is, the pointers appear to have a two-byte stride -- not very promising, even if they're sector counts, since the game image is over 0x26000000 bytes long. If the pointers *are* in sectors, however, the differences between pointers would allow for models, textures, etc. in many cases. Has anyone ever seen pointers at the end of a SLUS file before?

20

Scripting and Reverse Engineering / Let's talk about pointers in PSX Square games

« on: 2008-05-15 18:49:05 »

Much has been said about text pointers for translation hacking, but I've seen precious little about the pointers a game engine uses for other objects, such as character models and textures. I've been searching every nook and cranny of the Chrono Cross disc 1 game image for pointers to the battle and overworld character model data, but to no avail. I'm 99% confident that such pointers exist, because if I, say, overwrite a given character model with another model that has much more data and spills over the sector boundary at which the original model normally ends, the game freezes on me. A model swap iin which the new model is less than or equal to the file size of the original model works just fine, on the other hand.

The only place I can find pointers to the battle models is in the game image's Table of Contents, but it appears that the game engine does not "interact" with this giant pointer table in any meaningful way -- while it helps immensely in dumping all files from the CD, changing the Table of Contents to match file alterations does nothing to solve the problems associated with those alterations, and a major alteration to the Table of Contents absent any other changes to the game image causes no problems. If anyone's interested, there's some notes on Chrono Cross' Table of Contents here: http://www.chronocompendium.com/Forums/index.php/topic,5181.0.html

Given the difficulty I've had in locating pointers to overworld and battle models, it is my great suspicion that those pointers may be embedded within the battle script and fieldscripts, which may in turn be LZSS compressed (hence my difficulty in finding the pointers straight-out). My question for the qhimm community is, has there been any investigation of the pointers to model data and other graphical objects in the Playstation versions of Xenogears, FF7, FF8 and FF9? I suppose FF9's pointer system would be most applicable to Chrono Cross given the overlapping development periods.

Also, is the expectation of finding pointers to various objects within uncompressed fieldscripts and battle scripts realistic? I figure if there's a command that says, "load model ()" it would make sense that an "oh, by the way, the models start here, here, here, and here..." list would directly follow. However, this is an aspect of game hacking to which I am completely new.

21

Scripting and Reverse Engineering / Re: Front Mission 2 (PSX) Character Portrait Compression

« on: 2008-03-01 15:28:07 »

Wow, thanks guys! I'll take a look at those LZS examples. When I get some time I'll post the full decompression algorithm as I understand it in English. The way the algorithm is set up, it would seem to involve a ton of "if, else" logic. The value of the control byte apparently specifies whether the following reference byte(s) are to be interpreted as RLE, LZS with one sliding-window reference byte, or LZS with two sliding-window reference bytes.

In short, here's what happens with the control bytes:

If the control byte is greater than or equal to 0x3D but less than 0x7E, the reference bytes represent RLE compression. The length of the uncompressed run will be (Control Byte - 0x3D) bytes long.

If the control byte is greater than or equal to 0x7E but less than 0xBE, only the next byte is a reference that points backward into the sliding window. The number of bytes to copy from the window equals (Control Byte - 0x7E).

If the control byte is greater than or equal to 0xBE, the next two bytes are references that point backward into the sliding window. The number of bytes to copy from the window equals (Control Byte - 0xBE).

I don't believe the control bytes specify where the compressed run ends because, for the middle type above, the length of the compressed runs is completely variable even if the beginning control byte is consistent among compressed runs. Some more "if, else" logic may be needed to identify the last byte, which should always be less than 0x3D. That last byte instructs the algorithm to simply write the next (last byte value) bytes, then it encounters another control byte, rinses, and repeats.

From what I've seen of Ficedula's notes, the LZS compression in FF7 was "consistent," whereas I would describe FM2's compression as "situational." This might mean that available LZS code would have to be modified to such an extent that I'll need to just code something from scratch. I've got a bunch of Python tutorials on hand now, and it appears I would be able to make use of that language's input-output functions. I've always wanted to learn a programming language, so this project will give me the incentive to buck up and do it.

EDIT: Heeey, look at this! Turns out a patent was filed in the US for a new compression scheme that combined single-pass LZSS and RLE compression in 1997 -- the very same year Front Mission 2 was released. Interesting. FM2 wasn't released in the US of course, but I wonder if the LZSS/RLE combination might appear in other Square games.
http://www.patentstorm.us/patents/5701125/description.html

22

Scripting and Reverse Engineering / Re: Front Mission 2 (PSX) Character Portrait Compression

« on: 2008-02-28 01:49:33 »

I'd definitely be interested in Cyb's observations. It does appear that the "sliding window" typical of LZS compression is coming into play with this graphics data. I don't think Ficedula referred to the method as a "sliding window," but I think that's what's described in his notes -- when the decompression algorithm reaches a section of compressed reference data, that reference data tells the algorithm to fish around in the uncompressed file up to that point and regurgitate a specific run of bytes that fall within the "window".

While tinkering with the compressed data and its expanded VRAM equivalent, I cobbled together a mock "freeze frame" of one of the portrait files in mid-decompression. Whatever decompression algorithm is used would encounter the reference bytes D4 03 08 05 at offset 0x1066 (0x1066 bytes had been decompressed up to this point), and these reference bytes magically transform into a BF followed by twenty-one 00s in the uncompressed file. Lo and behold, there are two instances earlier in the decompressed data of one BF followed by twenty-one 00s! Thus it is extremely likely that the reference bytes instruct the decompressor to re-create that previous run of bytes. It's a fascinating compression scheme.

For now I'll presume that the size of the "sliding window" is 4096 (0x1000) bytes as in Ficedula's FF7 notes. The next step will be to do some thought experiments to determine which earlier BF 00 00... is being targeted (both instances of these runs are within the sliding window, at addresses 0x326 and 0x863) and explore the possible ways in which bitwise operations could be used to get those four reference bytes to point at the target. How I enjoy logic games when there's a treasure at the end! 

EDIT: Oooh, I just realized I should do the thought experiment with the BF 00 00... run that occurs just prior to the one I've been looking at. That way I'll know that it's definitely a duplicate of the first one starting out, so I can cut the theoretical work in half. This is why typing-while-thinking is important.

I will report my findings for everyone's amusement. Hopefully compression won't be as hard to work with as I had originally feared. The key is probably just having the "before decompression" and "after decompression" examples on hand to look at, although really experienced hackers can just trace the decompression algorithms in an emulator and be done with it.

SUPER EDIT: Ooh! Ooh! I've got part of it now! And more imporantly, I didn't even accidently double-post in my excitement! 

Here's two situations to illustrate some of what's going on with the decompression algorithm.

Situation 1
The decompression algorithm has worked its way to offset 0x1066, where it encounters the reference instruction D4 03 08 05. Now the algorithm reads those middle two bytes in little endian and goes 0x803 bytes back into the sliding window. Guess what's at offset (x1066 - x803) = x863? Yes! It's the BF 00 00... run we're looking for.

Situation 2
Earlier, the decompression algorithm worked its way to offset 0x863, where it encountered the reference instruction D7 3D 05 29. 0x863 - 0x53D = 0x326, which is -- yee-haw! -- the offset of the first BF 00 00... run!

Wow, this is fun. Now to figure out what the first and last bytes in the reference instructions do, and why reference instructions may be odd or even lengths, and why they may even appear like straight-out RLE instructions at times. But I guess the use of a sliding window proves that this is an LZS derivative, eh?

Mega Edit: The last byte in a four-byte reference specifies the number of literal bytes following the reference, in essence pointing the algorithm to the next control byte. For example, 0x5 literal bytes follow the reference in Situation 1 above, and 0x29 literal bytes follow the reference in Situation 2.

This end-byte rule applies to all compression references, regardless of their length. The last byte always tells the algorithm how many literal bytes come after the reference. Now to figure out what the first (the "control") byte does, exactly...

For one thing, I've discovered that a three-byte reference can act as follows: Control; NegativeOffset; #LiteralsFollowing. In other words, at times only one byte is needed to specify the negative offset for copying because that offset is less than x100. The control must serve the following functions, at the very least...

*Specify the number of bytes to regurgitate from earlier in the uncompressing file.
*Specify the number of bytes needed for the NegativeOffset.

This is where bitwise operations will probably come into play. Time to look at some 1s and 0s!

Another EDIT:

Decomposing some control bytes into bits reveals the following pattern.

81: 10000001 The following backward reference is single (value less than or equal to 0xFF).
D7: 11010111 The following backward reference is a double (value greater than 0xFF).
D4: 11010100 The following backward reference is a double.
A0: 10100000 The following backward reference is single.
C1: 11000001 The following backward reference is a double.
9D: 10011101 The following backward reference is single.
41: 01000001 The following reference is a straight-out RLE reference
42: 01000010 The following reference is a straight-out RLE reference

The only thing that really stands out to me across the examples are the leftmost two bits. Question: I think these would actually be the *final* two bits fed into the decompression algorithm if the byte is being read in Little Endian order. Is that correct? 

If so, we might have the following as decompression "opcodes" of sorts, if this sort of thing is possible:
01: LZS decompression, next backward reference is a single byte.
11: LZS decompression, next bakcward reference is a pair of bytes.
10: RLE decompression, next reference is the byte to be repeated.

The number of bytes to copy, either for LZS or RLE decompression, must also be specified by the control byte. There's definitely a pattern across the control bytes for this as well, but the fact that 0xD7 and 0xD4 both specify a copied run of (Dec 22, x16) bytes throws things off kilter for the time being. Other than the relative positioning of the copied runs, nothing differs between the runs copied by these control bytes (these are the BF 00... runs). The control bytes should be identical, and yet they aren't.

Ha! Scratch that. The control byte 0xD7 in Situation 2 above corresponds to a run of one BF and dec 24 zero bytes. The control byte 0xD4 in Situation 1 above corresponds to a run of one BF and dec 21 zero bytes. Looks like the pattern holds!

I think I'm seriously close to fully cracking this decompression algorithm. Anyone happen to know of any open-source LZS decompressors? I'm wondering if I could simply modify existing code to handle the decompression at hand. One problem is that I've never seen LZS decompression being used side-by-side with RLE compression, so I'll have to figure out how to add that. Or maybe I'll just research python decompression, since python is the first programming language I intend to learn. Wish me luck!

23

Scripting and Reverse Engineering / Re: Front Mission 2 (PSX) Character Portrait Compression

« on: 2008-02-27 05:07:40 »

Shh, don't tell anyone!

This is merely a diversion I can chew on while I compile the necessary updates to the Chrono Cross File Structure wiki at the Compendium. I've never dealt with compression before, and the very nicely-drawn Front Mission 2 portraits give me great incentive to investigate this sort of thing.

According to the LZS format wiki, http://wiki.qhimm.com/FF7/LZS_format, a control byte precedes a block of data composed of literal bytes and two-byte references, and you can determine the composition of a block by decomposing the control byte into its bits. Thus a control byte 03 (binary 00000011) tells us that the following block of data is (2 pairs of references + 6 single literal) = 10 bytes long, or two pairs of reference bytes followed by six literal bytes.

Using this analogy with the second comparison example pic in my first post, the top portion (compressed data) starts with a control byte A0. Decomposed into bits, this would read 10100000, or six literal bytes, one reference pair, one literal byte, and one reference pair in order before the next control byte comes into play. However, it would appear that the compressed data is actually set up as follows:

Control byte 0xA0 -> one reference pair -> sixteen literal bytes -> next control byte 0xA0. Literal bytes would be the ones in red in the examples, and the controls and compression references would have to be the bytes in green in the sample data.

Since the analogy doesn't describe what I'm seeing, I'm either misinterpreting the control byte scheme presented in the LZS wiki or Front Mission 2 character portraits are compressed in some variation of LZS that is not identical to the compression scheme used in FF7. While I get back to work on the Cross model project, I'll be thinking about how/if the basic principles of LZS compression can be adapted to reproduce what's in the sample data I lifted from the FM2 game CD and its associated uncompressed content in VRAM.

24

Scripting and Reverse Engineering / Re: Front Mission 2 (PSX) Character Portrait Compression

« on: 2008-02-27 01:36:52 »

Yikes, I've developed a bad habit of double-postedness.

I shall do more research on LZS compression techniques; I'm aware of Ficedula's notes on the wiki now, thanks for pointing this possibility out halkun. Front Mission 2 and FF7 were released in the same year, but given the *apparent* presence of RLE runs seemingly at random, this is perhaps a variation on compression schemes used in other Square games.

For one thing, the compressed portraits each begin with the same bytes (PB.. in ASCII), which might just represent the uncompressed file length. I find it strange (probably just given my inexperience in compression) that all the uncompressed portrait files would have the same length in bytes. The portraits are all the same dimensions, but I figure each portrait will be unique in the amount of "blank space" surrounding the character's face, which would determine the ease with which the file can be compressed in the first place.

I can confirm that the *compressed* portrait files are all the same length, for what it's worth.

25

Scripting and Reverse Engineering / Re: Front Mission 2 (PSX) Character Portrait Compression

« on: 2008-02-25 22:01:45 »

There are other instances in the data where a run beginning with 0x41 is straight-RLE compressed. For example, there's a 41 00 04, which represents 00 00 00 00. This specific example isn't pictured, but it's interesting. Given that there's a 40 08 03 in one of the posted examples, and that uncompresses to 08 08 08, we have...

40: Tells whatever device is responsible for decompression that the next two bytes represent RLE compression with an uncompressed run of 3 bytes.

41: " " the next two bytes represent RLE compression with an uncompressed run of 4 bytes.

Could the first byte of any compressed run be an opcode of sorts, perhaps? The general magnitude of its value might determine the type of compression used, and its exact value indicates the uncompressed length as well. The bytes after the opcode determine the content of the uncompressed run. Anyone ever see a compression method like this before?

EDIT: Okay, I've found a file identifier for the compressed files. In ASCII, it's PB.., or in hex, 50 42 00 00. This header does not exist in the uncompressed equivalent.