Ehrgeiz Data Exploration!

[koral]

Allow me to introduce yet another game for Quimm-forum dissection!

I found my copy of the disk buried deep amidst the pile of ancient stuff I have, and its so badly scratched it probably wont ever play again 
But my PC still reads most of the files, so all is not lost

I have the PAL version and there are 32 files all laid out nicely for our convenience.

The files can be categorised by their extensions:

• XA - Streamed audio (8 files, typically 20MB+)
• STR - Videos (12 files, ranging from 5MB-35MB)
• BIN - Data of all kinds bundled up in these (5 files, ranging from 1MB-20MB)
• EXE - there are 5 of these files, all less than 1MB. Probably similar to FF7's engine modules, but for this game.
• 2 PSX disc files, CNF and SLES

The most promising file seems to be ALLBIN.BIN, roughly 20MB in size (the largest of the BIN files) and contains numerous audio chunks, TIM images and most probably also model data.
I dont know if these BIN files are the same as FF7's BIN files, but I do know that the various chunks which I have spotted (audio and TIM particularly) are uncompressed and plain as daylight.
I didn't see any obvious offsets near the beginning of the file though.

Yes, yes, I should be finishing the CrisesCore stuff before diving head-first into other things, but as always, we wont get anywhere unless somebody takes the first step. 

[Landarma]

For XA and STR, I don't think they have something unusual, although each XA file has at least 8 tracks(most of them are BGM, but some tracks are SFX, and at least one is voice clip for 'Quest mode'). 

[halkun]

The bins are probably lzs compressed. Look for strings of the same number, and if none are found, it's compressed. Compare to the bin format of FF7, they probably used the same library.

[koral]

I dont know if these BIN files are the same as FF7's BIN files, but I do know that the various chunks which I have spotted (audio and TIM particularly) are uncompressed and plain as daylight.

There are huge kilobyte chunks of zeros in here, not to mention long, long strings of the same character too, so I can safely say that this file is not compressed
And it is also not the same as FF7's BIN files, feels more like Crises-Core where the data is continually streamed in when and as required, with no obvious offsets anywhere to be found (probably in those EXE files).

But it is fairly easy to spot the different file-types inside though, and the most striking is one called "pBAV".
Maybe I am imagining things, but these may very well be modified p-model files 

These are what I had initially assumed to be Audio files, because the data towards the end seems very reminiscent of RIFF files.

The data itself seems to be structured as follows (based on data from a couple of pBAV's):

Code: [Select]

0x0000   32-byte header: magic = "pBAV"
0x0020   2048-bytes (2 KB!) long LIST of 16-byte chunks
0x2080   4608-bytes (4 KB!) long LIST of 32-byte chunks
0x1A20   1488 bytes of unknown data
0x1ff0   lotsa zeros (1376 bytes)
0x2550   HUGE list of 16-byte chunks, split occasionally with "wwwwww"s (multiple 0x77) padding bytes

The positions of the TIM files would help to prove or contradict the theory of these potentially being model data.
I am still looking for a solid way to seperate TIM files from everything else though.

[koral]

MrAdults pointed me to a tool he wrote a long time ago called "Timmy" which hunts down TIM files and converts them into BMP files.

So I tried it on ALLBIN.BIN and it gave me 637 TIM files!

You can see some of the images here:


But the strange thing is, there are no Character textures! I am sure that character's eyes and clothing were definitly textured in the game.
I tried running Timmy on the other files too (ignoring XA and STR files) and none of them contained any TIM textures.

So there can only be one conclusion: the Character models must be compressed!

It might have been difficult to determine precisely where they would have been, but for Timmy's clever file-naming scheme:
the filenames of the extracted TIM files are the actual offsets where they were found.

And coincidently enough, the TIM files seemed to be regularly spaced through the ALLBIN.BIN files, but there was a HUGE leap at one place:
from TIM at offset 2555904 (0x270000) to TIM at offset 8388660 (0x800034), approximatly 5.5 megabytes of data containing no TIM file

And a smaller (yet still unusually large) gap from TIM at offset 10862604 (0xA5C00C) to TIM at offset 12961848 (0xC5C838), approximatly 2 megabytes of TIM-less data

Either of which should contain Character or Level model data.
 

[ultima espio]

Korals on a roll:D haha  anyway characters are textured, i got Tifas using 3D ripper years ago, got  no geo with it though.

[Zande]

But it is fairly easy to spot the different file-types inside though, and the most striking is one called "pBAV".
Maybe I am imagining things, but these may very well be modified p-model files 

These are what I had initially assumed to be Audio files, because the data towards the end seems very reminiscent of RIFF files.

The data itself seems to be structured as follows (based on data from a couple of pBAV's):

Code: [Select]

0x0000   32-byte header: magic = "pBAV"
0x0020   2048-bytes (2 KB!) long LIST of 16-byte chunks
0x2080   4608-bytes (4 KB!) long LIST of 32-byte chunks
0x1A20   1488 bytes of unknown data
0x1ff0   lotsa zeros (1376 bytes)
0x2550   HUGE list of 16-byte chunks, split occasionally with "wwwwww"s (multiple 0x77) padding bytes


VAB = Banks of audio data, similar VAG. It's very easy to detect playstation audio as the actual data is split in block of 16 bytes whereas the last 14 bytes are 28 packed samples, first byte is predictor/shiftfactor and the 2nd byte is flag. The flag byte is usually 0x00 or 0x02, so every 16th byte is either a zero or a two. Also the flag for sample end is 0x07, and often the last block looks like this:

Code: [Select]

00 07 77 77 77 77 77 77 77 77 77 77 77 77 77 77

[Satoh]

MrAdults pointed me to a tool he wrote a long time ago called "Timmy" which hunts down TIM files and converts them into BMP files.

So I tried it on ALLBIN.BIN and it gave me 637 TIM files!

You can see some of the images here:


But the strange thing is, there are no Character textures! I am sure that character's eyes and clothing were definitly textured in the game.
I tried running Timmy on the other files too (ignoring XA and STR files) and none of them contained any TIM textures.

So there can only be one conclusion: the Character models must be compressed!

It might have been difficult to determine precisely where they would have been, but for Timmy's clever file-naming scheme:
the filenames of the extracted TIM files are the actual offsets where they were found.

And coincidently enough, the TIM files seemed to be regularly spaced through the ALLBIN.BIN files, but there was a HUGE leap at one place:
from TIM at offset 2555904 (0x270000) to TIM at offset 8388660 (0x800034), approximatly 5.5 megabytes of data containing no TIM file

And a smaller (yet still unusually large) gap from TIM at offset 10862604 (0xA5C00C) to TIM at offset 12961848 (0xC5C838), approximatly 2 megabytes of TIM-less data

Either of which should contain Character or Level model data.
 

PSicture can rip the character textures, so wherever they are, they are in standard TIM format. (Thanks to FaustWolf for ripping them on another forum...)

[MrAdults]

I don't implicitly trust TIMmy, as I did write it like 6 years ago, but it would be pretty odd for it to just magically miss every single character texture. Seems more likely that they are uncompressed, but not in TIM format.

[koral]

TIMmy is reliable enough. 
I tried PSicture too (at X-Dina's suggestion) and it gave me the exact same TIM files, 637 of them in ALLBIN.BIN, nothing in the others.
I also tried creating an ISO of the disc and looking for TIMs in that, but TIMmy crashed half way through, and PSicture gave me 607 TIMs at the end, missing 30 of them altogether.

X-Dina, could you tell me more about FaustWolf's attempts? I have been unable to find any more TIMs, so MrAdults theory of non-TIM textures might also be plausible.

And thanyou Zande for confirming those files as Audio, my initial impressions were correct then 

[Satoh]

Ah, now that you mention it, it was around the time we deciphered the modified TIM format that Chrono Cross uses... it could actually be similar to that instead then... I could have remembered incorrectly...

I'll see if I can't get in touch with him so he can shed light on this matter...

[FaustWolf]

Oh, I got them from VRAM dumps. Since there's only a handful of characters in Ehrgeiz, I'd be happy to rip them all. Look for them sometime this weekend.   

Ripping the monsters from the dungeon sidegame would be more difficult though, since there's more of them. However, I recall that the end bosses were particularly beautiful. Are you guys interested in those textures too?

[koral]

I am not so much interested in what I can extract, I am more interested in the how

So you dont have to upload any textures for me. I dont know about other people though.

This is a relatively old game so the models wont be anywhere as good as Crises-Core (etc) anyway.
I just want to get the Ehrgeiz model-format's details drilled for future reference.

The more info the Quimm wiki has (and the more files my viewer-tool can view), the happier I will be

Could you tell me more about the VRAM dumps you did? I assume the textures must have been in TIM format, probably after being uncompressed by the game.
Any idea about offsets?
And if you could give me texture-counts too then that would help speed up the process 

Thanks FaustWolf
 


Oh and I will just mention this before I forget

The first huge gap in the ALLBIN.BIN file actually does contain compressed data of some kind. There are no long strings, and everything seems like long nonsense of hex characacters.
But the compressed data is definitly chunked, I have seen regular uncompressed bytes (long strings of zeros mainly) inbetween at varying places.
Might just be program opcodes or general RAM stuff.

The second hugh gap has very long lists of things, and might actually be the mesh data itself. Or maybe not. 

[FaustWolf]

I used PVV3 as my VRAM viewer and pSX (psxfin.exe) as my VRAM dumper. The methodology is as follows:

PVV3 source:
http://www.romhacking.net/forum/index.php/topic,5092.msg89496.html#msg89496

To do a VRAM dump in pSX, you go to
*Debug > Monitor > r3000
*Debug > Break (to freeze the game's processes at the point you want stuff to stop moving in memory)
*GPU > Save VRAM


As for investigation of the raw file format, yaaargh!   I'd really hate to think that the textures are compressed (they weren't in FF8 or FF9 after all, and those were roughly contemporary to Ehrgeiz IIRC), but that may be the case because I'm unable to track down the uncompressed VRAM texture in the game image. Granted, that's difficult to do though because PVV's address lookup feature hasn't been implemented yet (I don't even know whether Creaothceann is still working on it), and it can only save to .BMP and .PNG, and not as raw data.

The best I've been able to do so far is compare segments of data in the .BMP capture of Sephiroth's texture to the raw VRAM data and find a match. That matched data does not exist as such in the game image. So the best I can do so far is inform you that the data "may be compressed." 

[Cyberman]

Depends on the era of the game. FF7 used extensively LZS compression on it's data. This included all the battle models and data. Interestingly the basic data set didn't change that much mostly the BATTLE data some of the field files and movies we what changed.  The battle models didn't change a whole lot.

Erstwhile, my guess is they would have used LSZ if at all. The main reason is that it doesn't have large memory requirements for decompressing data. Unlike other dictionary compression techniques.  It also does a good job of handling graphics data.

I've used a similar technique with RLE data that it uses with dictionary compression the difference being that RLE doesn't require buffering of any data so it's data space requirements are much smaller. (I had NO space to use hence the technique that used none was the only appropriate.)

If you can find pointer or LBA information in the data then you might be able to find the model blocks. If you already have found the model data you may just have to dig around in it to find the texture data.  My guess is they would only compress things that aren't used often. It depends on which takes longer, loading the data from the disk OR decompressing the data from the disk. My guess is the former (that's why they did it that way in FF7, they improved there engine design enough to make it easier for them in FF8 etc.)

Cyb

[finelamier]

Hi,
i try to do the same ,im using TMD_RIP but not really works. now i just extract same TOD file using TOD_rip and tim file its realy hard to find a good ripper for ps1 gamz but if it'll work it will be realy funy.

if you find new way  to do that help me plz

sorry for my english 


  If you're going to revive an old topic, you should be contributing something.  Warned.

[Bosola]

The bins are probably lzs compressed. Look for strings of the same number, and if none are found, it's compressed. Compare to the bin format of FF7, they probably used the same library.

Just spotted this, albeit a year later; nice tip.