Compression

[Lord_Skylark]

Okay...here's one of the many files in FF11 PS2 Japanese Beta... I'm wondering if anyone is able to know what type of compression the file uses. And I"m guessing all of the files will use it as well. It appears that this is actually more than one file compressed into a single file as well.

www.ffcompendium.com/~Skylark/ff11/12.DAT

Thanks,
~Sky

[DeadLajik]

Well I took a look at it but didn't get anywhere..

If this is from a PC release then maybe you can run it under a debugger.  You would want to set a breakpoint on the "CreateFileW" call. Make sure it is the correct call though. The first element to CreateFileW is a pointer to a character array. So you'd have to check the ESP register and then trace to the point of memory it is referring.

Remember that intel is little endian though so.. if the ESP says:

401BFFBF   it really means: BFFF1B40  so dump that memory and make sure it says something like "DATA\12.DAT" or something.

If this is from the PS2 then you'll need to get a MIPS dissambler and search for the system call the opens files.  However, on the MIPS the first 4 arguments to functions are passed in seperate registers $a0 through $a3. If there are more then 4 arguments the additional arguments are placed onto the stack.

At this point you'll most likely be reading assembly code, but it should give you enough information to figure out what some bytes in the file represent.

... and if you knew all this stuff about debugging.. sorry to explain it again.

[fa_pa]

what do you expect to be in that file?
because if you don´t know what this file contains its much harder to find the compression algorithm

edit: the file also has no real header. could it be possible that its just a part of a big file?

[Lord_Skylark]

Maybe it is a bigger file, I don't know. But I know in that file there are a bunch of compressed smaller files that are the names of the monsters. I can show you what one of the files would look like from the pc version uncompressed, and I image they would be the same. Would someone be able to check out the compression if I made an image and uploaded it? All the files are seperate. And oddly, this is a cd, not a dvd for PS2. It's about 500 megs total. If I uploaded, could someone check it then?

Thanks,
~Sky

[fa_pa]

the uncompressed file from the pc game would be helpful  
is this file from the ps2 version?
you say you have uncompressed files from the pc version why decompress the ps2?

Code: [Select]

ZONEPOINT_SD:¸;BS¹;
W´;COUƒvSD¹;
Bºv
W´;DEATH¯sƒTƒ“ƒhƒŠƒAAP:
ƒoƒXƒgƒD[ƒN„
ƒEƒBƒ$_ƒX„"3‹4CT:ƒ\Œ4„
‹4„"
„4
G„d Ž€–S—¦:
À{À{‰{
WORLD_POINƒPƒ‹P

‹Y4
ŽPAP_TOPƒK@GD‡
Wˆ"

‹"
‹^ˆB
‹BˆŸlˆ\
…sCƒjƒu)O

this part apears often in the file. it looks like a table or somthing like that.
after this follow much crap.

[Lord_Skylark]

Well, if you didn't read close enough - this is the beta version of the game and is quite different than the final version - so I want to check the files and view them from the beta version.

~Sky

[Lord_Skylark]

Well, as stated above, this is the BETA version of the game - and it is quite different than the final version. Thus, I want to be able to view the files from the beta version and actually test and see what all was changed and stuff.

~Sky

[mirex]

uhh file is 11MB ... thats too big, aint there any smaller file, like < 1MB ? i'd like to take a look, but 11mb is too big for my 64kbs ISDN

[DeadLajik]

Skylark,

I am writing a program that may help you.. What it does is analyze sequences of bytes, and writes how many times they are duplicated within the file. This might help to figure out some of the data structures..

I can probably finish it by friday..

[DeadLajik]

Probably pretty useless info for you now but I got the following from a program I am working on that is analyzing that file:

Code: [Select]


    DEATH appears 90 times
    IHDR appears 32 times
    IEND appears 32 times
    ZONEPOINT_SD appears 60 times
    menumap appears 62 times


Most of those don't mean much right now.. except the IHDR and IEND tags..

These are part of the PNG file format.

So inside 12.dat are 32 PNG files. The don't look to be encrypted either but I havn't written any tool to extract them yet. Maybe i'll do this later tonight.

[fa_pa]

yup there are png headers inside but i cant get the image displayd there is something wrong in the header.

Code: [Select]

00000000  89 50 4e 47 0d 0a 1a 0a 00 00 00 0d 49 48 44 52  .PNG........IHDR
00000010  83 08 01 58 00 04 02 08 03 83 12 04 60 dc 09 b5  ...X........`Ãœ.µ
00000020  83 19 05 04 67 41 4d 41 83 21 05 00 8b 25 60 4d  ....gAMA.!...%`M
00000030  83 29 08 20 63 48 52 4d ff ff ff 83 03 86 06 8c  .). cHRMÿÿÿ.....
00000040  0c 88 18 7f 46 80 4b 01 00 00 03 00 50 4c 54 45  ....F.K.....PLTE
00000050  13 0e 13 77 7a 7e 0d 0f 11 80 82 83 15 17 17 77  ...wz~.........w

edit:
here is the extracted png file. it doesn´t work because the header is not real png but i´m working on it.

http://wap.justanotherportal.de/folders/fa_pa/ff11_test.png

edit2:
there also headers of BMP files

edit3:
here is a header from a real png

Code: [Select]

00000000  89 50 4e 47 0d 0a 1a 0a 00 00 00 0d 49 48 44 52  .PNG........IHDR
00000010  00 00 02 58 00 00 00 db 04 03 00 00 00 43 f3 f2  ...X...Û.....Cóò
00000020  b0 00 00 00 18 50 4c 54 45 00 00 00 6d 6d 6d 49  Â°....PLTE...mmmI
00000030  49 49 b6 b6 b6 92 92 92 ff ff ff db db db 24 24  II¶¶¶’’’ÿÿÿÛÛÛ$$
00000040  24 1e d8 3e d6 00 00 20 00 49 44 41 54 78 da ed  $.Ø>Ö.. .IDATxÚí
00000050  bd cd 5b e2 ca f3 3e 1c 5e 02 5b 14 19 b7 18 44  Â½Ã[âÊó>.^.[..·.D


[DeadLajik]

I extracted a few of the PNG files but they all appear to be encrypted in some way, or at least the IHDR chunk of the file is.

According to the PNG specifcation byte 0x1A should be 0x00 since there is only one type of compression, not 0x12 which appears to be on most of the PNG files (all of them that I extracted).

Cracking the encryption on the HDR part could lead us to the encryption used on the entire 12.DAT file.

I'll try to decode this header but it will take some time since changing bytes in the header requires me to recompute the CRC of the file.

Edit:
The BMP files are also encrypted somehow.. The header for one BMP said its file size was >3500 MB which is most likely incorrect.

Edit2:

It's also sorta interesting that pretty much all the PNG files seem to have identical headers.

[DeadLajik]

Well, I looked over the PNG files some more, and they appear to be compressed also..At least the individual chunks inside each PNG file since they are actually missing bytes.. For example, each chunk has a CRC checksum at the end of it's section. The next 4 bytes represent the size of the next chunk..

Well in the PNG files from 12.DAT the IHDR CRC bytes go into the gAMA length bytes..

[Lord_Skylark]

Alright - you want me to upload the entire image tonight? It's about 500 megs. Can you email me at [email protected] for a link to it, because I don't want to post the link on a messageboard and use up lots of bandwidth having tons of people downloading it.

~Sky

[DeadLajik]

Skylark,

I wont be able to look at anything  for the next 5 days because I am going out of town..

However I don't think the whole image of the CD is necessary, so if you havn't uploaded it yet.. don't..  

If worse comes to worse I might need to take a look at the executable and maybe any DLL's that it links in, to determine what is going on.

[mirex]

Could you guys upload few of those compressed bmp's on the net ? I know structure of .bmp maybe i could find something usefull.
If you dont have where to upload them use http://bin.mypage.sk

[DeadLajik]

While there are many places where the "BM" header matches in that file I can not tell whether they are BMPs or not.. especially because the 4 bytes after (DWORD) that describe the file size have bad values.

I havn't really gotten anywhere on this encryption. I can only suggest taking a look at the PNG file that was posted earlier to decypher it.