Qhimm.com Forums

Miscellaneous Forums => Scripting and Reverse Engineering => Topic started by: koral on 2009-01-20 23:11:59

Title: FF7 Crisis Core - File Format and Data Investigation
Post by: koral on 2009-01-20 23:11:59
--------------------------------------------------------------------------------------
[time saver]

All the findings and knowledge is being wiki-fied here: http://wiki.qhimm.com/FF7:CC

Tools to view/extract Crises-Core data and Models:


Everything else in this first post is old and probably will become obsolete.
However I have kept it mostly intact because of nostalgic value, as it was my first post here at Quimms!  :lol:

HUGE thanks to everyone here on the Quimm forums for your support!
It's all for you guys, enjoy!  :-D

[/time saver]
--------------------------------------------------------------------------------------




Not much seems to be discussed about FF7:CC regarding its files and data, so I thought I'd look into things myself and maybe help to get the ball rolling  :-D

Thanks to G for his FF7CC extractor, I quickly browsed through the 8000 odd files it churned out and noticed some really interesting things; I'm not sure how the extractor works, but same file-types (having same or similar header data or possibly function) are grouped together via their filenames.

(EDIT: Get G's-Extractor from here: http://forums.qhimm.com/index.php?topic=7053.0 (http://forums.qhimm.com/index.php?topic=7053.0) also uploaded here if anyone can't get it from there: http://www.mediafire.com/?nwm24zynjmk (http://www.mediafire.com/?nwm24zynjmk) )

Taking G's extractor filenames as reference, here's what I've discovered [these are the files output by his extractor, eg "00001.raw"]:

(NOTE: these are my initial thoughts ONLY, I may be wrong on several accounts, which is why I'm posting here amongst all the elite FF r-engineers, quimm and mirex deserve special mention at this point  8-) )

(NOTE2: When I refer to "Header:" I don't mean actual byte values, just some common ASCII text contained within the first 16 or so bytes of the files)

(NOTE3: I didn't look through ALL the files, just randomly poking around and narrowing the boundries; I may have missed some clusters or mistyped some filename values)


00000 - 00005   possibly misc system files, nothing too important
00006 - 00062 Header: "PSMF001" - PSP format Movies / Video files
00063 - 01349 Header: "RIFF" - Audio files
01350 - 01417 Header: "SSCF" - more Audio files
01418 - 01442 Menu and DMW related stuff (I think)
 Mix of two types of files:
    Header: [seemingly random] - files contain bunch of indexed Long's or something, no idea
    Header: "IMG" - guessing they're image / Texture files
01443 - 01445  standard PNG images
01446              Now this is interesting - seems to contain a list of Locations and Mission Titles!
01447              Contains a long list of character names, probably used when talking to NPC's etc
01448 - 01455  No discernable header (pretty much random). Maybe mesh data?
01456              Header: "MBD" - Indices or Counts of something
01457 - 02011  Story progression and Event data
  Mix of two types of files:
    Header: "GT"  - Chapter End Images
    Header: "ATEL" - LOTS of these, contain various Textures, dialogue scripts, etc
02012 - 02457 Models: Characters, Monsters, NPCs and possibly also Locations
  Mix of three types of files:-
    Header: [seemingly random]
    Header: "!" - contain single Textures for each Model. Maybe mesh-data too, but haven't found any.
    Header: "&&"
02458 - 03352  Tiny files, no common header, no ascii info visible. No idea.
03353 - 08538  unknown, no common header, fairly small files <500 kb. Ditto
08539 - 08622  Monster / Boss Data!
   Header: "MOT_" - later also contain "SSCF" data later. Interesting...
   maybe "file-names", Event Scripts, Boss Battle scripts, Monster behaviour or something else. Its all there, but no idea what.
08623 - 08627  Header: "zack_" - later also contain "ATEL" data
08628 - 08690  Materia (Magic) effect files!
  Header: "MAGIC_W0"   
08671 - 08706  unknown, tiny files
08707 - 08725  Header: "PSMF0015" - more Videos


So there you have it, my first two cents for the community here.

I'm mainly interested in the Character Model data myself, having already written my own viewers for FF7 and FF8 field and battle models (now looking to fry bigger fish :-P ) but that doesn't mean I'll ignore everything else possibly important to understand the game better (Maps, Events, etc).

Anyone care to help?
 :-D






EDIT:
Here'll I'll try keep all discovered File-Formats listed in an organised way, with a description of what they are and how to extract their contents:

==========================================
FF7 CC File Format Information Library
==========================================
LAST UPDATED: 15 MAR 2009


See: http://wiki.qhimm.com/FF7:CC



Stuff below has yet to be archived (much of it incomplete):

---------------------------
ATEL = Event data, story and game progression
---------------------------
- RAW Files:
- Not much else known about , except that they contain various different File-Type chunks:
Code: [Select]
    MBD    - Script (dialogue) Font table
    FEP     - ?
    ANM    - ?
    TEX     - Textures (fully decoded)
    MDL     - ?
    VTL     - ?
    [ASCII]- ?
    &&      - ?
    SSCF   - Audio (dialogues and/or music)

ignitz
Quote
The scripts is in this files ATEL headers and some other I don't remember
Inside these files have headers MBD in after these have the scripts then in Japanese and American have a table of 4 bytes
Here this table in format Thingy
80000000=A
80000100=B
80000200=C
80000300=D
80000400=E
80000500=F
80000600=G
80000700=H
80000800=I
80000900=J
80000A00=K
80000B00=L
80000C00=M
80000D00=N
80000E00=O
80000F00=P
80001000=Q
80001100=R
80001200=S
80001300=T
80001400=U
80001500=V
80001600=W
80001700=X
80001800=Y
80001900=Z
80001A00=a
80001B00=b
80001C00=c
80001D00=d
80001E00=e
80001F00=f
80002000=g
80002100=h
80002200=i
80002300=j
80002400=k
80002500=l
80002600=m
80002700=n
80002800=o
80002900=p
80002A00=q
80002B00=r
80002C00=s
80002D00=t
80002E00=u
80002F00=v
80003000=w
80003100=x
80003200=y
80003300=z
80003400=0
80003500=1
80003600=2
80003700=3
80003800=4
80003900=5
80003A00=6
80003B00=7
80003C00=8
80003D00=9
80003E00=!
80003F00=?
80004000=
80004100="
80004200='
80004300=,
80004400=.
80004500=:
80004600=;
80004700=--
80004800=/
80004900=(
80004A00=)
80004B00=$
80004C00=<
80004D00=>
80004E00=%
80004F00=[
80005000=]
80005100=YEN
80005200=@Cruz
80005300=#
80005400=_
80005500=|
80005600=^
80005700=º
80005800=-
80005900=+
80005A00==


---------------------------
! = Model Data (only textures found so far)
---------------------------
- RAW Files: 02012 - 02457, maybe other places too
- So far only Textures have been decoded, mesh-data remains elusive

Code: [Select]
Header: 127 bytes
   0x00   6 bytes: magic hex values ( 00 00 00 00 21 27 )
   0x10   DWORD: offset to [unknown chunk 1]
   0x14   DWORD: offset to [unknown chunk 2]
   0x1C   DWORD: offset to [TEXTURE Chunk]
   0x34   DWORD: offset to [unknown chunk 3]
   0x3C   DWORD: offset to [unknown chunk 4]
   0x40   DWORD: offset to [unknown chunk 5]
   0x48   DWORD: offset to [unknown chunk 6]

[Unknown Chunk 1]
   (seems to always start at 0x80)

   for each data-chunk (32 byte-block):
   - 1 short    always: 00
   - 1 short    bone id?
   - 2 bytes   always: FF FF
   - 1 byte    sometimes: 00, sometimes: 01 -- maybe polygon type (quad / tri)
   - 1 byte    always: 00
   - 1 long    always: 00
   - 4 bytes    ???
   - 4 floats    polygon normal?

[Unknown Chunk 2]
   Possibly Vertex data

   Header: 32-bytes
   - 1 long (0x00)  = number of polygons (data-blocks) in [Unknown Chunk 1]
   - 1 long (0x04)  = offset to data start

   Data: ?


[TEXTURE Chunk]
   header: 16-bytes
0x00  = SHORT: image width
0x02  = SHORT: image height
0x08  = DWORD: offset to PALLETE-data start offset
0x0c  = DWORD: offset to PIXEL-data start offset

   PALLETE data:
255 4-byte [RGBA] chunks - Texture Colour Pallete [255 entries]

   PIXEL data:
bytes referencing to Pallete-Entry


[Unknown chunk 3]
   ?

[Unknown chunk 4]
   ?

[Unknown chunk 5]
   ?

[Unknown chunk 6]
   ?


==========================================
[eof]
==========================================

to be continued...
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: Zande on 2009-01-21 08:27:10
Only thing I can remember right now is that SSCF chunks are audio samples.
Seems I have a different naming of the files, so..


edit:
Quote
00063 - 01349
Header: "RIFF" - possibly Audio files
Oh yes, I remember those also.. they are in vaild IFF format, it's all the voice acting file thingies. The files are stored in Sony's ATRAC3 format (Minidisk etc.), if you rename the files to .wav and have an ATRAC3 codec they should play in your prefered mediaplayer (like winamp).
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: koral on 2009-01-21 14:14:20
Hey cool, thanks info Zande  :-D

Now why would your file-naming be different? I used the data from a US-region UMD, but I can't imagine why there would be differences between different region versions of the game, unless theres actually differences in game-content?!
Can anyone confirm that my file-lists? I may have botched up somewhere with the ISO creation or something, although I doubt that should make a difference.

I'm going to look into some of the "potentially mesh-data" files [02012 - 02457], see what I can dig-up.
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: Zande on 2009-01-21 14:45:15
Nono, there's no difference between the game data, it's just the way the files are extracted, and most precisly, the naming of the files extracted... The big pack of files doesn't have a filename list, hence all the numbers.. and my files are numbered slightly different (which is for I reason that's not very important). :)
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: koral on 2009-01-21 19:44:57
Ah, OK thanks for clearing that up :)

I've had a go at parsing some of the files which I believe contian vertex data, but I don't seem to be getting anywhere. Lots of files contain lots of chunked values, yet I've found nothing yet which might be vertices.
Files [03353 - 08538] might contain something of interest, thousands of files, could be anything.
Also some files seem to contain different File-Types embedded within the same file, i've noticed some "MDB" file-types too.

Do you have any sort of information on the "IMG" files / chunks (or some way to view these PSP format images )?
I know Playstation used to use TIM image files almost exclusively, but I've got no idea what PSP images tend to be (especially for this game).

I'm guessing that Meshes would usually be located near to their corresponding Textures (or in the same file) so if I know which Texture belongs to which character / model, I could check out nearby files more thoroughly
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: Zande on 2009-01-21 21:41:42
Quote
Do you have any sort of information on the "IMG" files / chunks (or some way to view these PSP format images )?
I know Playstation used to use TIM image files almost exclusively, but I've got no idea what PSP images tend to be (especially for this game).
The image palettes are 32bit format R8G8B8A8. Palettes for textures are fairly easy to notice because the alpha channel is pretty much all the time fully solid. For sprites it slightly harder as they use alot of transparency.
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: koral on 2009-01-22 11:11:02
OK, thanks again zande  :-D

I was just looking through one of those "Atel" files, and I believe it may be a character or something, lots of different file-types embedded within.

For reference, I'll list out the Offsets and the Header (file-types) here, as well as "regions" of data-flow changes:


File: 01488
Header: Atel
Chunk Offsets:
   0x2fc40    SSCF   - Sound files
   0x43da0   FEP
   0x46f50    - Event scripts?
   0x47090    MBD
   0x47450    MBD
   0x507f0   FEP
   0x52fd0    MDL
   0x5a2c0    MBD
   0x5a7d0   FEP
   0x5b560    MDL
   0x5b6c0    VTL
   0x5b770    - script?
   0x5b790    FEP
   0x60d50    ANM
   0x60e00   TEX
   0x63240    MDL
   0x632f0      - "false" !!
   0x63310      - "true" !!
   0x633a0    MDL
   0x63600    MDL
   0x63920    VTL
   0x639d0    - script
   0x63a20    FEP
   0x6a2f0    ANM
   0x6a3a0    ANM
   0x6a5d0    TEX
   0x6ca10    TEX
   0x6ee50    MDL
   0x6efb0    MDL
   0x6f210   MDL
   0x6f530    VTL
   0x6f5e0    - script
   0x6f640    FEP
   0x74c00    ANM
   0x74cb0    TEX
   0x770f0    MDL
   0x77250    MDL
   0x774b0    MDL
   0x777d0    VTL
   0x77880    - script
   0x778d0    FEP
   0x7f3d0    ANM
   0x7f480    TEX

        .... [to be continued?]


its a big file, and I haven't even covered half the file (it got boring, and tiriing), but its clear to see a pattern of distinct data blocks, most importantly (in order they seem to occur):

    FEP     - ?
    ANM    - animation data of some sort?
    TEX     - texture data of some sort?
    MDL     - model data? guessing maybe skeleton or skin data
    VTL     - ?
    [ASCII] - script of some kind?

Vertex/face data may have been covered within the first 0x2fc40 bytes, but that's open to speculation.

Next step: Look into Texture-data (IMG and now TEX files) to try get an idea of WHAT is located where (character, backgrounds, sprites, menus, etc, etc)
fun times :lol:
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: koral on 2009-01-22 17:19:53
(sorry for the double post, but I want to help differentiate the large content coming up from last time. won't happen again)


just been playing with an IMG file and have more-or-less got it figured, maybe around 50%

File: 01428

Contains 4 different IMG paletted textures

Header: 80 bytes

   0x4   - DWORD: offset to IMG-1 Pixel data
   0xC   - DWORD: offset to IMG-1 Pallete data

   0x14   - DWORD: offset to IMG-2 Pixel data
   0x1C   - DWORD: offset to IMG-2 Pallete data

   0x24   - DWORD: offset to IMG-3 Pixel data
   0x2C   - DWORD: offset to IMG-3 Pallete data

   0x34   - DWORD: offset to IMG-4 Pixel data
   0x3C   - DWORD: offset to IMG-4 Pallete data

Then starting with each "IMG" headed chunk:

        - 32 byte header (contains shorts for sizes etc)   [eg, for IMG-1, offset: 0x50]
        - actual Pixel-data (which pallete value)             [eg, for IMG-1, offset: 0x70]
        - pallete entries (4 bytes RGBA)                         [eg, for IMG-1, offset: 0x10070]


Textures aren't my strong point, and from what I've spotted so far, this all seems like menu-images or sprites or something, far too small and incoherent to be anything useful.

I'll have a look at the TEX data-chunks next, see how they match up with these IMG formats, maybe be able to get something useful from it.
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: koral on 2009-01-23 21:58:19
TEX files are just like IMG images, palletized bytes of RGBA colors, not packed or encrypted AFAIK.

I haven't been able to get any nice images/textures though, maybe there's some kind of interlacing going on or something else altogether. Textures just aren't my cup-of-tea and I'm not going to waste any more time on them.

Examples of my handiwork:
(http://img149.imageshack.us/img149/5711/01479rawtex0texra4.png)
(http://img207.imageshack.us/my.php?image=01505rawtex1texzw6.png)
(http://img91.imageshack.us/img91/9435/01524rawtex4texom4.png)


Here's the structure of TEX files as far as I can tell:

-------------------------------------
TEX File Format
-------------------------------------

Header: 48 bytes [upto 0x2f]

   0x00   magic "TEX"
   0x1C   DWORD: size Width
   0x20   DWORD: size height

Pixel Data: 0x30 offset

   (Width * Height) number of bytes, 1 byte per-pixel index (to pallete entry color, below)

Pallete data: (immedietly following Pixel data)

   16 bytes of mostly zeros (don't seem important)
   255 4-byte chunks [RGBA] (per pallete entry)


next, going to investigate those "GT" files
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: koral on 2009-01-24 14:28:16
GT Files are Textures too!  :|

---------------------------
GT File Format
---------------------------

Header: 16 bytes
   0x00   magic "GT"

Pallete Data: start offset = 0x10
   255 [RGBA] 4-byte chunks

Pixel Data: immedietly following Pallete [start offset = 0x410]
   256 * 256 bytes (index to pallete entry)


List of GT files I found:

   01458-01467
   01495
   01497-01498
   01563-01565
   01590-01592

Funnily enough, they were all 258 KB, 256x256 images. There might be another cluster of somewhere with larger or smaller sizes.

Definitly still something weird going on with how I'm trying to view them (image interpolation?), but IMO they look a LOT like SKY-BOX Textures, or background images of somesort.
Infact, I'd say they look more like TEXTURES than the TEX files seemed to be (which might have just been "effect sprites" or something).

Can someone please have a look into these GT files and see if they could interpret them properly? I'm VERY sure they're going to be proper images, but I totally suck with texture-data.

BTW, it really is lonely out here... I feel like I'm all on my own, even with the 200+ views this topic has had already  :?
Any help would be appreciated!
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: NameSpoofer on 2009-01-24 18:15:25
I would love to help you, because I also want to rip the textures and models from crisis core. The problem is I don't have an ISO file of crisis core so theres nothing I can do *sigh*. Anyways I believe people like zande, darkwings or shinra inc. are interested in these, maybe you should contact them.

Quote
File: 01488
Header: Atel
Chunk Offsets:
   0x2fc40    SSCF   - Sound files
   0x43da0   FEP
   0x46f50    - Event scripts?
   0x47090    MBD
   0x47450    MBD
   0x507f0   FEP
   0x52fd0    MDL
   0x5a2c0    MBD
   0x5a7d0   FEP
   0x5b560    MDL
   0x5b6c0    VTL
   0x5b770    - script?
   0x5b790    FEP
   0x60d50    ANM
   0x60e00   TEX
   0x63240    MDL
   0x632f0      - "false" !!
   0x63310      - "true" !!
   0x633a0    MDL
   0x63600    MDL
   0x63920    VTL
   0x639d0    - script
   0x63a20    FEP
   0x6a2f0    ANM
   0x6a3a0    ANM
   0x6a5d0    TEX
   0x6ca10    TEX
   0x6ee50    MDL
   0x6efb0    MDL
   0x6f210   MDL
   0x6f530    VTL
   0x6f5e0    - script
   0x6f640    FEP
   0x74c00    ANM
   0x74cb0    TEX
   0x770f0    MDL
   0x77250    MDL
   0x774b0    MDL
   0x777d0    VTL
   0x77880    - script
   0x778d0    FEP
   0x7f3d0    ANM
   0x7f480    TEX

Anway what type of file is this? It is like a model file or sth? I believe MDL is shortcut of model, ANM is shortcut of animation and TEX is shortcut of texture. I see good signs now. Thanks for doing this.

Edit: I might risk myself to download the ISO from bittorrent to help you since I am also interested. What version of crisis core do you have? Can you send me G's extractor as well?
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: Zande on 2009-01-24 20:08:39
The GT files are chapter images (http://the-world-is.mine.nu/cc/5066.jpg) or whatnot (I have no idea myself is I've never played CC but my friend said something like that when I asked him a while ago), the image data's slightly interleaved.
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: koral on 2009-01-24 22:32:52
wow, thanks for the support guys!

my goal is to try find and extract the characters mainly, but anything else is a bonus I suppose. There's so many darn files, its going to take a while, but we'll definitily find some interesting things along the way

don't worry about things too much namespoofer, I could always upload some files for you to have a look at

but yeah, I thought those were MoDeL, ANiMation, etc too, but the IMG files have been pretty small and I don't think those files will contain much useful. They might be small monsters or something though...


zande:

those are chapter images?!  :-o
your friend's image has helped me a lot, because I've realised where the header now contains the image width and height sizes, and also how close I got to extracting it.

Here's that very image [GT file 01495] showing how the data appears if read with no modifications (ie, read like scanlines from first x first y, going across and down):

(http://img185.imageshack.us/img185/5909/01495rawgttl6.png) (http://imageshack.us)

Its not really interlacing thats the problem, its how the pixels are organised for the PSP to show them efficiently. You can tell that the white pixels on the left are supposed to be the window behind Genesis, and roughly see Sephi's sword, but the pixels are totally confusing for me... :?

the GT file is here if anyone is interested: http://www.mediafire.com/?mohmdhz43zj


I'll update the first post to try organise my findings a little more, and any other information you guys discover or suggest.
I've already got a handful of extractors/viewers made for these files, but I'll upload them for everyone once they're perfect (not good enough yet, as you can tell by my screens).

I've stumbled across another potential group of character/model files which also contain palleted textures.
The pixel-data is probably organised in the same way everywhere, so if we can crack this, I'm sure ALL the other textures will also fall into place nicely too!

Title: Re: FF7 Crisis Core - File Format Investigation
Post by: Zande on 2009-01-24 23:27:22
I should've written also:

For the GT files, the 4th byte in the header tells the colour depth format, if this byte is 3 the palette have 16 colours and 1 nibble per pixel. If the byte have a value of 4, there's 256 colours and 1 byte per pixel. All GT files are the later format anyway (but not all textures).

The 5th byte in the header I'm unsure about but I think it's the pixel data format, but I haven't really look much at it. The only thing I know is that both the GT files and alot of other textures stores the image interleaved as smaller images, usually as 16x8 pixels (this would represent foramt 1).
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: koral on 2009-01-25 14:49:28
Ammended the first post, thanks again zande  :-D

I'm getting close to deciphering the interlacing with the images, now it looks something like this:
(http://img300.imageshack.us/img300/2531/01495rawgtfh3.png) (http://imageshack.us)
its a lot more like how it should be, with the two distinct halves of the image really helping to make things easier to figure out

they're definitly split into 16x8 blocks (as zande pointed out) but there's some additional interlacing going on between the blocks themselves. The image is still quite literally all over the place, just need to re-order the pixels some more

This is actually my first time trying to de-interlace anything, so if any video/image experts can give me a hand then it'll speed things up a bit!
 :lol:
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: dziugo on 2009-01-25 15:53:33
The way I think it is:
If you number the strips (16 pixels in width, 1 pixel in height) from left to right, and from up to down like that:
Code: [Select]
  0,   1,   2,   3,   4, ...  30,  31
 32,  33,  34,  35,  36, ...  62,  63
 64,  65,  66,  67,  68, ... 126, 127
128, ................................
.....................................

On the image it should be placed like this:
Code: [Select]
   0,   8,  16, ... 240, 248, | 256, 264, 272, ... 496, 504
   1,   9,  17, ... 241, 249, | 257, 265, 273, ... 497, 505
   2,  10,  18, ... 242, 250, | 258, 266, 274, ... 498, 506
   3,  11,  19, ... 243, 251, | 259, 267, 275, ... 499, 507
   4,  12,  20, ... 244, 252, | 260, 268, 276, ... 500, 508
   5,  13,  21, ... 245, 253, | 261, 269, 277, ... 501, 509
   6,  14,  22, ... 246, 254, | 262, 270, 278, ... 502, 510
   7,  15,  23, ... 247, 255, | 263, 271, 279, ... 503, 511

So basically, each 512x8 strip on the source image is a scrambled version of a 512x8 strip on the resulting image.

Edit: Or something like that :P
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: koral on 2009-01-25 20:51:12
I CRACKED IT!!

dziguo, your suggestion was close but it gave me weird results. It got me thinking on the right track though and I managed to solve it eventually.

The data is divided into 16x8 pixel-chunks, but is arranged in a strange sort of way:
Code: [Select]
ABCD  MNOP  YZ... 
EFGH  QRST   ...
IJKL  UVWX   ...

Then reading each chunk (in that order), grab rows of 16-pixels (like 8 scanlines, top to bottom of chunk) and plot them like so:
Code: [Select]
  1,  17,  33,  49,     5,  21,  37,  53,     9,  25,  41,  57,    13,  29,  45,  61,
  2,  18,  34,  50,     6,  22,  38,  54,    10,  26,  42,  58,    14,  30,  46,  62,
  3,  19,  35,  51,     7,  23,  39,  55,    11,  27,  43,  59,    15,  31,  47,  63,
  4   20,  36,  52,     8,  24,  40,  56,    12,  28,  44,  60,    16,  32,  48,  64


Here's some extracted images [the most important FF7 characters no-less]:

(http://img204.imageshack.us/img204/1264/01461rawgtnw3.th.png) (http://img204.imageshack.us/my.php?image=01461rawgtnw3.png)  (http://img127.imageshack.us/img127/9296/01462rawgtbd9.th.png) (http://img127.imageshack.us/my.php?image=01462rawgtbd9.png)  (http://img127.imageshack.us/img127/8016/01463rawgtzd2.th.png) (http://img127.imageshack.us/my.php?image=01463rawgtzd2.png)

And my [GT]-File-Viewer: http://www.mediafire.com/?2ez4jmmmlki

I'll update the first post sometime later, probably look into some of the other texture image files (TEX and IMG) to see how the Format-interlacing differs.

Enjoy! :wink:
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: NameSpoofer on 2009-01-25 20:53:43
theres no words to describe your greatness

keep up the work
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: dziugo on 2009-01-26 10:33:48
Yay. Glad you worked it out.
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: koral on 2009-01-26 22:28:47
YAHOO!!! [TEX] Files now decoded!

Examples:
(http://img238.imageshack.us/img238/3676/01479rawtex0texrb0.png) (http://imageshack.us)  (http://img98.imageshack.us/img98/2395/01479rawtex5texcx8.png) (http://imageshack.us)  (http://img98.imageshack.us/img98/7264/01500rawtex1texme8.png) (http://imageshack.us)  (http://img98.imageshack.us/img98/3979/01505rawtex1texad4.png) (http://imageshack.us)

I'm not sure what they represent exactly, possibly just misc textures used throughout the game, for various events, locations, etc. Nothing too characteristic of exactly WHAT they belong to, but still interesting nevertheless.

The pixel and pallete locations are different (see header data in first post) but the pixel packing is exactly the same as GT files (1 byte per pallete entry).

Interlacing is exactly the same, except that the interlacing in columns (how I numbered above: 1,17,33,49,   5,21,37,53, ... ) is determined by the width of the image.
So,
if width = 512, then you get 4 interlaced columns
if width = 256, then you get 2 interlaced columns
if width = 128, then you get 1 interlaced column
(ie, divide image width by 128)

Remember that TEX files are embedded within ATEL files (and possibly other kinds of files too) so they're not as easy to simply "view" like [GT] files were.

I'm working on a unified FF7-CC File Loading program right now which would be able to find and view ANY recognisable file-format from any of the thousands of FF7-ISO extracted data files.
When I get it going, I'll be sure to upload it for y'all.

I can't be bothered with IMG files though (GUI images with weird bit-palleting) so I'm going to jump the gun and check out some textures from possible character files.
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: Roxas on 2009-01-27 01:47:21
Wow!  :-o I'll definitely be watching how this progresses. 8-)
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: Jonnylossus on 2009-01-27 01:57:07
Great work!
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: koral on 2009-01-27 23:48:13
 :-D
As promised, I've uploaded the first public version of my FF7-CC File-Viewer proggy. Check out the first post, the link's there right on top.

I've browsed through some of the ATEL files [01457 - 02011] but haven't spotted anything tooo important. Seems like story-event related storage mainly, especially if you consider that the GT Chapter-End images were also mixed in.
But obviously I haven't been through ALL these ATEL files, so if anyone manages to spot anything REALLY interesting then do let me know here.

So far its only been Textures, but until I find the Character Textures I'm not going to get distracted with all the other chunks in those ATEL files (sound, mesh, animations?).
Unless someone wants to help out  :-P
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: ShinRa Inc on 2009-01-28 00:43:25
Anyways I believe people like zande, darkwings or shinra inc. are interested in these, maybe you should contact them.

Very interested, and in Dirge of Cerberus as well.  Unfortunately, my technical knowledge is somewhat lacking, so I'm not sure what I could do to help at this stage.  In the past I've mostly helped with testing out some of the viewers and editors that were developed here (Ultima, Ifalna, Leviathan, Zangan come to mind), and also with sorting/identifying the extracted data.

I'll try and keep up with what's going on as best I can here, but feel free to drop me a line on AIM if there's something you think I can help with.
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: koral on 2009-01-28 20:26:45
Thanks for offering ShinRa, so far things seem to be going smootly though, so feel free to just sit back and enjoy  :wink:


GOOD NEWS ALL!
I found the CHARACTERS!  :-D

Not just Characters, but most (if not all) the Models too, including Monsters, weapons, NPCs, some sky-dome images and other stuff (which came out garbled, probably just sprites or particles or something).

(http://img440.imageshack.us/img440/3787/thumbshq2.th.jpg) (http://img440.imageshack.us/my.php?image=thumbshq2.jpg)

There's a good 360 files, and what you see above is only a small selection  :-P


Some more information on how I got these:

Files: 02012 - 02457 = what I call [!]-Files

These files seem to contain a number of different chunks too, similar to ATEL files except not as obvious. I managed to decipher the offsets in the header and load the Texture-data from within each file.
So what you see above are ONLY the textures from within each of these [!] files, seemingly one texture per file.

[!] FILE
Code: [Select]
Header: 127 bytes
   0x00   6 bytes: magic hex values ( 00 00 00 00 21 27 )
   0x10   DWORD: offset to [unknown chunk 1]
   0x14   DWORD: offset to [unknown chunk 2]
   0x1C   DWORD: offset to [TEXTURE Chunk]
   0x34   DWORD: offset to [unknown chunk3]

Unknown Chunk 1
   14 (0xE) 32-byte chunks - vertex info? bones?

Unknown chunk 2
   header: 32-bytes
       0x02   DWORD: offset to data start
   data: 40 bytes chunks - vertex data? lots of data available

TEXTURE Chunk
   header: 16-bytes
0x00  = SHORT: image width
0x02  = SHORT: image height
0x08  = DWORD: offset to PALLETE-data start offset
0x0c  = DWORD: offset to PIXEL-data start offset

   PALLETE data:
255 4-byte [RGBA] chunks - Texture Colour Pallete [255 entries]

   PIXEL data:
bytes referencing to Pallete-Entry


Unknown chunk 3
mostly 0xff

The texture-data is exactly like [TEX] files (minus the TEX header) so there were no problems viewing those.
Mesh data HAS to be present in the same files too, just a matter of finding the correct chunk and understanding the face-indices to be able to render them.
Not long now :lol:
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: ultima espio on 2009-01-28 21:08:01
Thanks for offering ShinRa, so far things seem to be going smootly though, so feel free to just sit back and enjoy  :wink:


GOOD NEWS ALL!
I found the CHARACTERS!  :-D

Not just Characters, but most (if not all) the Models too, including Monsters, weapons, NPCs, some sky-dome images and other stuff (which came out garbled, probably just sprites or particles or something).

(http://img440.imageshack.us/img440/3787/thumbshq2.th.jpg) (http://img440.imageshack.us/my.php?image=thumbshq2.jpg)
The texture-data is exactly like [TEX] files (minus the TEX header) so there were no problems viewing those.
Mesh data HAS to be present in the same files too, just a matter of finding the correct chunk and understanding the face-indices to be able to render them.
Not long now :lol:

Did i ever mention that I love you? haha lol :-P :-P :-P
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: NameSpoofer on 2009-01-28 21:48:25
This is incredible Koral. Im sure when you fully decoded CC and created a "CC model viewer" people will start worshipping you  8-) . It might even revive this half-dead community. The combination of KH models and CC models in FF7 will result in an epic mod xD.
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: BlitzNCS on 2009-01-28 21:56:13
Uhh, for some reason, the Texture files close down the Viewer when you try to open them, in fact every single file apart from the ones that came with the .rar make it randomly close. any help? i REALLY wanna see those textures :D
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: spidergaby on 2009-01-28 22:02:10
I support you for this project !!
"CC model viewer" !

PS: we can see reno and rude...awesome  :mrgreen:

Title: Re: FF7 Crisis Core - File Format Investigation
Post by: koral on 2009-01-29 02:23:32
ultima espio: I love you too!  :-D
Thanks for (sortof) quoting my post, makes it easier to see what I said in the new page

NeoCloudstrife:
sorry about that, but that was the [GT] file viewer only which closed down automatically when it tried loading a non-[GT] file.
Use my newer FF7-CC-File-Viewer program instead (v0.2), which loads [GT] as well as [TEX] and [ATEL] files too: http://www.mediafire.com/?zowymgf3jmo
(has no-one checked the first-post?  :-P)

NameSpoofer & spidergaby:
I'm glad things are getting interesting again around here. I missed the "golden era" when FF7 file-format discoveries were going on, and things haven't really picked up much since. It'd be cool to get some of these models into a mega FF7-mod, the Turks would finally have their say ;)
There's also the near-perfect Sephiroth, Tifa (tour-guide) and Yuffie models too.

I have made a new version of the Viewer which loads [!] Texture files too (as demonstrated in my last post), but I haven't uploaded it yet.
I'll get around to it in a couple of days, fix a few things here and there.
 :wink:
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: BlitzNCS on 2009-01-29 03:40:18
NeoCloudstrife:
sorry about that, but that was the [GT] file viewer only which closed down automatically when it tried loading a non-[GT] file.
Use my newer FF7-CC-File-Viewer program instead (v0.2), which loads [GT] as well as [TEX] and [ATEL] files too: http://www.mediafire.com/?zowymgf3jmo
(has no-one checked the first-post?  :-P)


I have made a new version of the Viewer which loads [!] Texture files too (as demonstrated in my last post), but I haven't uploaded it yet.
I'll get around to it in a couple of days, fix a few things here and there.
 :wink:
thanks for that, and sorry, i thought they were the same thing :P

BTW, i'm REALLY looking forward to seeing textures and even getting these models!
keep up the good work!
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: ShinRa Inc on 2009-01-29 06:05:46


Here's some extracted images [the most important FF7 characters no-less]:

(http://img204.imageshack.us/img204/1264/01461rawgtnw3.th.png) (http://img204.imageshack.us/my.php?image=01461rawgtnw3.png)  (http://img127.imageshack.us/img127/9296/01462rawgtbd9.th.png) (http://img127.imageshack.us/my.php?image=01462rawgtbd9.png)  (http://img127.imageshack.us/img127/8016/01463rawgtzd2.th.png) (http://img127.imageshack.us/my.php?image=01463rawgtzd2.png)


I believe those are the chapter break images.
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: ultima espio on 2009-01-29 18:10:55
--------------------------------------------------------------------------------------
[time saver]

Grab the latest version of my FF7-CC File-Viewer here: http://www.mediafire.com/?2ez4jmmmlki

Latest version: v0.2 on 27 Jan 2009

Aww invalid file :-(
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: ignitz on 2009-01-29 19:40:26
--------------------------------------------------------------------------------------
[time saver]

Grab the latest version of my FF7-CC File-Viewer here: http://www.mediafire.com/?2ez4jmmmlki

Latest version: v0.2 on 27 Jan 2009

Aww invalid file :-(

[2]
This Works
http://www.mediafire.com/?zowymgf3jmo

The scripts is in this files ATEL headers and some other I don't remember

Inside these files have headers MBD in after these have the scripts then in Japanese and American have a table of 4 bytes

Here this table in format Thingy

Code: [Select]
80000000=A
80000100=B
80000200=C
80000300=D
80000400=E
80000500=F
80000600=G
80000700=H
80000800=I
80000900=J
80000A00=K
80000B00=L
80000C00=M
80000D00=N
80000E00=O
80000F00=P
80001000=Q
80001100=R
80001200=S
80001300=T
80001400=U
80001500=V
80001600=W
80001700=X
80001800=Y
80001900=Z
80001A00=a
80001B00=b
80001C00=c
80001D00=d
80001E00=e
80001F00=f
80002000=g
80002100=h
80002200=i
80002300=j
80002400=k
80002500=l
80002600=m
80002700=n
80002800=o
80002900=p
80002A00=q
80002B00=r
80002C00=s
80002D00=t
80002E00=u
80002F00=v
80003000=w
80003100=x
80003200=y
80003300=z
80003400=0
80003500=1
80003600=2
80003700=3
80003800=4
80003900=5
80003A00=6
80003B00=7
80003C00=8
80003D00=9
80003E00=!
80003F00=?
80004000=
80004100="
80004200='
80004300=,
80004400=.
80004500=:
80004600=;
80004700=--
80004800=/
80004900=(
80004A00=)
80004B00=$
80004C00=<
80004D00=>
80004E00=%
80004F00=[
80005000=]
80005100=YEN
80005200=@Cruz
80005300=#
80005400=_
80005500=|
80005600=^
80005700=º
80005800=-
80005900=+
80005A00==

but the Pointers I don't found.
Someone found please post because I want to translate to PT-BR(Portuguese-Brazil) this game and I don't have a PSP, my friend have.

---------------------------------------------------------------------------
00006 - 00062
Header: "PSMF001" - possibly PSP format Movies / Video files

This videos in the games when Zack's remind somebody o some thing in the DMW*

08707 - 08725
Header: "PSMF0015" - more Videos?

These are videos em CG and Real cutcenes. That not have none of realtime on PSP XP


These two files are PMF Video Files
Can play in Media Player Classic with AVC/H264 Decoder, the audio Not

Can play with audio with a homebrew in the PSP custom firmware

I don't know how to decode the AT3 in PMF files to reencode

-----------------------------------------------------------------------------
Problem with your program
I don't know what s this

(http://i464.photobucket.com/albums/rr10/ignitzhjfk/BUG-FF7CC-FileViewer.jpg)
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: Silver Wings on 2009-01-29 21:53:42
First of all, thank you so much for the great work :D
And, I think I know how to get the audio within a pmf file. You need MPlayer to extract the at3 from pmf file and use a homebrew called 'at3decode test' written by Z33 to decode it to wav.

*edit: here is the guide http://www.psp-hacks.com/forums/f134/guide-convert-pmf-to-avi-t159985
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: ignitz on 2009-01-29 22:18:39
First of all, thank you so much for the great work :D
And, I think I know how to get the audio within a pmf file. You need MPlayer to extract the at3 from pmf file and use a homebrew called 'at3decode test' written by Z33 to decode it to wav.

*edit: here is the guide http://www.psp-hacks.com/forums/f134/guide-convert-pmf-to-avi-t159985


Thanks for the tip
I doing this but
The PMF Ending AT3 it's brick my PSP because is to big.
And PSP FAT is hard to found. Can anyone interest to make a Windows decode? I'm not a excelent programer to make a decomrpession audio or video.

It's strange a Square games that not have a LZ compression like FFT FF7 FF8

Here's a DUMP of the first MBD 0000.raw
Code: [Select]
// 00000.raw CETRANSLATORS - Final Fantasy VII - Script Dump [$9F3-$CF0] - Thu Jan 29 15:03:00 2009
//Script extraído por IGNITZ
//www.cetranslators.com.br

// [$9F3]
<$00><NL>
<$84><$00><$00><$00><$0C><$00><$00><$00>Where do you want to go?<END>

// [$A64]
<$40><$05><$00><$08> Battle<END>

// [$A88]
 Ms. Ishibashi<END>

// [$AC4]
 Mr. Oka<END>

// [$AE8]
 Mr. Kitase<END>

// [$B18]
 Mr. Kunikata<END>

// [$B50]
 Mr. Shindo<END>

// [$B80]
 Mr. Terada<END>

// [$BB0]
 Mr. Maeda<END>

// [$BDC]
 Moriya kun<END>

// [$C0C]
<$40><$04><$00><$00>

How to find the first script?
 I dump the RAM and search for 3º MBD script and search in the ISO
XD

First the 1º script that I have dumped a very long time ago
Code: [Select]
Zack speaking.
Making progress,Zack?
What's going on, Angeal?
Why are we fighting Shinra troops?
They're Wutai troops in disguise.
Now, head to the open area above. Towards Sector 8? Yes.
But first, you have to clear a path.
Clear a path?
You'll see what I mean.
Be careful.
I can cut loose, right?
Use some discretion.
It's showtime!
Not too bad.
Piece of cake!
I'll make 1st in no time!
Go up the stairs at the end of the platform.
You got it!


Wutai troops have hijacked Shinra Express MK93 II.
The train is currently headed to Midgar Sector 8.
SOLDIER operative will be inserted to neutralize the situation.
We are to proceed as scheduled.
Commence mission in      
Commence mission in 3    
Commence mission in 3 2  
Commence mission in 3 2 1
Commence mission in 3 2 1 mark.
Begin mission.
CAOperative is ready for insertion.
The train has been overrun by Wutai troops.
Eliminate them and regain control of the train.
Oh yeah!
Get serious!
Zack!
Zack, focus!
here are no Shinra troops on this train.A CAUnderstood?
Hello to you, too!
Come and get it!
SOLDIER 2nd Class Zack, on the job!
Train MK93 II re--routed to Sector 1 Station near Mako Reactor 1.!
Proceed to mission phase 2 and change surveillance rank...
Control Zack using the directional buttons
or the analog stick.
Enemies in the direction that Zack is facing
will be targeted automatically.
Cycle through commands using the L or R button,
and press A GA to confirm.
Press A HA to guard, and A IA to dodge.

I found pointer in the MDB FIles of 0001.raw but of this first script (tat I don't remeber what file is) I'm not found

Here
http://i464.photobucket.com/albums/rr10/ignitzhjfk/TABELA.jpg
I found thias in the lost files
XD
I delete the header to make this table font to be visualizble

Sorry for my ENGLISH

-------------------------------------------------------------------------

I found a piece of file os the firs script and dumped
Code: [Select]
// 1 script CETRANSLATORS - Final Fantasy VII - Script Dump [$0-$1A7F] - Thu Jan 29 15:33:06 2009
//Script extraído por IGNITZ
//www.cetranslators.com.br

// [$0]
<$4D><$42><$44><$00><NL>
<$12><$00><$00><$00><$0C><$00><$00><$00> <END>

// [$18]
Zack speaking.<END>

// [$54]
<$40><$04><$00><$00><NL>
<$1A><$00><$00><$00><$0C><$00><$00><$00> <END>

// [$6C]
Making progress, Zack?<END>

// [$C8]
<$40><$04><$00><$00><NL>
<$3D><$00><$00><$00><$0C><$00><$00><$00>What's going on, Angeal?<END>

// [$13C]
Why are we fighting Shinra troops?<END>

// [$1C8]
<$40><$04><$00><$00><NL>
<$45><$00><$00><$00><$0C><$00><$00><$00>They're Wutai troops in disguise.<END>

// [$260]
Now, head to the open area above.<END>

// [$2E8]
<$40><$04><$00><$00><NL>
<$15><$00><$00><$00><$0C><$00><$00><$00> <END>

// [$300]
Towards Sector 8?<END>

// [$348]
<$40><$04><$00><$00><NL>
<$2B><$00><$00><$00><$0C><$00><$00><$00>Yes.<END>

// [$36C]
But first, you have to clear a path.<END>

// [$400]
<$40><$04><$00><$00><NL>
<$11><$00><$00><$00><$0C><$00><$00><$00> <END>

// [$418]
Clear a path?<END>

// [$450]
<$40><$04><$00><$00><NL>
<$25><$00><$00><$00><$0C><$00><$00><$00>You'll see what I mean.<END>

// [$4C0]
Be careful.<END>

// [$4F0]
<$40><$04><$00><$00><NL>
<$1F><$00><$00><$00><$0C><$00><$00><$00> <END>

// [$508]
So, I can cut loose, right?<END>

// [$578]
<$40><$04><$00><$00><NL>
<$18><$00><$00><$00><$0C><$00><$00><$00> <END>

// [$590]
Use some discretion.<END>

// [$5E4]
<$40><$04><$00><$00><NL>
<$12><$00><$00><$00><$0C><$00><$00><$00> <END>

// [$5FC]
It's showtime!<END>

// [$638]
<$40><$04><$00><$00><NL>
<$10><$00><$00><$00><$0C><$00><$00><$00> <END>

// [$650]
Not too bad.<END>

// [$684]
<$40><$04><$00><$00><NL>
<$2A><$00><$00><$00><$0C><$00><$00><$00>Piece of cake!<END>

// [$6D0]
I'll make 1st in no time!<END>

// [$738]
<$40><$04><$00><$00><NL>
<$30><$00><$00><$00><$0C><$00><$00><$00> <END>

// [$750]
Go up the stairs at the end of the platform.<END>

// [$804]
<$40><$04><$00><$00><NL>
<$0F><$00><$00><$00><$0C><$00><$00><$00> <END>

// [$81C]
You got it!<END>

// [$84C]
<$40><$04><$00><$00><NL>
<$36><$00><$00><$00><$0C><$00><$00><$00> <END>

// [$864]
Wutai troops have hijacked Shinra Express MK93 II.<END>

// [$930]
<$40><$04><$00><$00><NL>
<$36><$00><$00><$00><$0C><$00><$00><$00> <END>

// [$948]
The train is currently headed to Midgar Sector 8. <END>

// [$A14]
<$40><$04><$00><$00><NL>
<$45><$00><$00><$00><$0C><$00><$00><$00> <END>

// [$A2C]
A SOLDIER operative will be inserted to neutralize the situation.<END>

// [$B34]
<$40><$04><$00><$00><NL>
<$23><$00><$00><$00><$0C><$00><$00><$00> <END>

// [$B4C]
We are to proceed as scheduled.<END>

// [$BCC]
<$40><$04><$00><$00><NL>
<$1E><$00><$00><$00><$0C><$00><$00><$00> <END>

// [$BE4]
Commence mission in       <END>

// [$C50]
<$40><$04><$00><$00><NL>
<$1E><$00><$00><$00><$0C><$00><$00><$00> <END>

// [$C68]
Commence mission in 3     <END>

// [$CD4]
<$40><$04><$00><$00><NL>
<$1E><$00><$00><$00><$0C><$00><$00><$00> <END>

// [$CEC]
Commence mission in 3 2   <END>

// [$D58]
<$40><$04><$00><$00><NL>
<$1E><$00><$00><$00><$0C><$00><$00><$00> <END>

// [$D70]
Commence mission in 3 2 1 <END>

// [$DDC]
<$40><$04><$00><$00><NL>
<$23><$00><$00><$00><$0C><$00><$00><$00> <END>

// [$DF4]
Commence mission in 3 2 1 mark.<END>

// [$E74]
<$40><$04><$00><$00><NL>
<$12><$00><$00><$00><$0C><$00><$00><$00> <END>

// [$E8C]
Begin mission.<END>

// [$EC8]
<$40><$04><$00><$00><NL>
<$25><$00><$00><$00><$0C><$00><$00><$00> <END>

// [$EE0]
Operative is ready for insertion.<END>

// [$F68]
<$40><$04><$00><$00><NL>
<$5D><$00><$00><$00><$0C><$00><$00><$00>The train has been overrun by Wutai troops.<END>

// [$1028]
Eliminate them and regain control of the train.<END>

// [$10E8]
<$40><$04><$00><$00><NL>
<$0C><$00><$00><$00><$0C><$00><$00><$00> <END>

// [$1100]
Oh yeah!<END>

// [$1124]
<$40><$04><$00><$00><NL>
<$10><$00><$00><$00><$0C><$00><$00><$00> <END>

// [$113C]
Get serious!<END>

// [$1170]
<$40><$04><$00><$00><NL>
<$09><$00><$00><$00><$0C><$00><$00><$00> <END>

// [$1188]
Zack!<END>

// [$11A0]
<$40><$04><$00><$00><NL>
<$10><$00><$00><$00><$0C><$00><$00><$00> <END>

// [$11B8]
Zack, focus!<END>

// [$11EC]
<$40><$04><$00><$00><NL>
<$37><$00><$00><$00><$0C><$00><$00><$00>There are no Shinra troops on this train.<END>

// [$12A4]
Understood?<END>

// [$12D4]
<$40><$04><$00><$00><NL>
<$16><$00><$00><$00><$0C><$00><$00><$00> <END>

// [$12EC]
Hello to you, too!<END>

// [$1338]
<$40><$04><$00><$00><NL>
<$14><$00><$00><$00><$0C><$00><$00><$00> <END>

// [$1350]
Come and get it!<END>

// [$1394]
<$40><$04><$00><$00><NL>
<$27><$00><$00><$00><$0C><$00><$00><$00> <END>

// [$13AC]
SOLDIER 2nd Class Zack, on the job!<END>

// [$143C]
<$40><$04><$00><$00><NL>
<$44><$00><$00><$00><$0C><$00><$00><$00> <END>

// [$1454]
Train MK93 II re--routed to Sector 1 Station near Mako Reactor 1.<END>

// [$1558]
<$40><$04><$00><$00><NL>
<$3E><$00><$00><$00><$0C><$00><$00><$00> <END>

// [$1570]
Proceed to mission phase 2 and change surveillance rank...<END>

// [$165C]
<$40><$04><$00><$00><NL>
<$01><$01><$00><$00><$0C><$00><$00><$00>Control Zack using the directional buttons <END>

// [$171C]
or the analog stick.<END>

// [$1770]
 <END>

// [$1778]
Enemies in the direction that Zack is facing <END>

// [$1830]
will be targeted automatically.<END>

// [$18B0]
 <END>

// [$18B8]
Cycle through commands using the L or R button, <END>

// [$197C]
and press <$40><$06><$00><$00> to confirm.<END>

// [$19DC]
 <END>

// [$19E4]
Press <$40><$07><$00><$00> to guard, and <$40><$08><$00><$00> to dodge.<END>

// [$1A6C]
<$40><$04><$00><$00><$07><$14><$09><$05><$05><$24><$16><$05><$18><$20><$00><$00><$00><$00><$00><$00>

01000000=<NL>
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: Silver Wings on 2009-01-29 22:48:31
Quote
The PMF Ending AT3 it's brick my PSP because is to big.
I dont really get what your problem is.

Anyway, you CAN run the homebrew on PSP slim using Time Machine instead of looking for a FAT one. I think your psp has custom firmware installed so what you need is a Pandora battery which is easily created from a standard battery by homebrew.

Finally, as far as I know, there is no other way to decode the at3 to wav in Windows or any environments. It can be done only on PSP. In short, psp system has to decode the at3 first to play it and Z33's homebrew simply takes this moment to decode it to wav (of course, it's not as "simple" to complete such great work at all :D)

ps: I'm very sorry for being off the topic  :-)
To ignitz, if you have any question, feel free to pm me, I'll help you as far as I can :-)
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: ignitz on 2009-01-29 23:00:31
Here more Lost screenshots lost in my PC
XD

http://i464.photobucket.com/albums/rr10/ignitzhjfk/file004014.jpg

http://i464.photobucket.com/albums/rr10/ignitzhjfk/file004013.jpg

http://i464.photobucket.com/albums/rr10/ignitzhjfk/file004012.jpg

http://i464.photobucket.com/albums/rr10/ignitzhjfk/file004011.jpg

Extracted with xpert with plugins properply

file004002 ==== Table image
file004003 ==== Monster ???
file004004 ==== unknow... bear???
file004005 ==== Chocobo Image
file004006 ==== Cait image
file004007 ==== Moogle
file004008 ==== Goblin
file004009 ==== Não lembro
file004010 ==== ????????
file004011 ==== Summons
file004012 ==== Small Summons
file004013 ==== ??????????

The header that deforme the image.
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: koral on 2009-01-29 23:33:31
what'd I miss? What'd I miss?  :-o

Wowzers, that's some impressive work ignitz! is that your own CC File decoding program or someone elses, because it looks really cool.
I'm sorry my viewer doesn't work properly for you, the problem is definitly todo with Irrlicht's font-encoding system which is probably incompatible with your OS Language settings. I'll try using MSCVC or something else in the future to handle the GUI for the viewer (or maybe try get Japanese fonts working or something)

MDB files contain tables for script-characters? the [ATEL] files should have those dialogue-scripts within them after the MDB data, I can't imagine them being anywhere else.

Silver Wings: thanks for the video and audio conversion info, :-)

ultima espio
sorry about all the wrong link, it gets confusing sometimes  :-P


I've uploaded the latest version of the viewer here (v0.25): http://www.mediafire.com/?dac2zmtkzjo

The layout has changed a bit to make it easier to find and select files. Just dump all your FF7-CC files into the "CCDATA" folder and the program should list them all automatically.
The biggest change is that it allows you to view the textures inside those yummy [!] files.
I've included samples of the 3 different File-Types with the program so you can immedietly test it.

IMO this File-Format business is getting a little out of hand. I wish we could have a Wiki or something so we can document all our findings there in an organised way  :|
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: ultima espio on 2009-01-30 00:16:18
How about a feature that lets us save the texture so that we dont have to printscreen it? It would be nice to see the texture transparency too. Great Job! :-D :-D
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: ignitz on 2009-01-30 00:22:36
Wowzers, that's some impressive work ignitz! is that your own CC File decoding program or someone elses, because it looks really cool.
I'm sorry my viewer doesn't work properly for you, the problem is definitly todo with Irrlicht's font-encoding system which is probably incompatible with your OS Language settings. I'll try using MSCVC or something else in the future to handle the GUI for the viewer (or maybe try get Japanese fonts working or something)

No not mine, the credits is for some user of this site
http://www.alucard.cc/

He found the grafics. and compaired the Chinese Hack and JAP

Me to compaired but chinese not MOD the pointers.

About At3 audio files decode is true
but you have to decode (With codec installed) in the Windows Media Player or WInamp, Media Player Class don't work
Crystal Tile
I'm not found the web site because the old link is dead

Oh
and thanks for the Silver Wings
I'm going take a PSP and will try to reencode the ending with my subtitles that I will have to remake
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: BlitzNCS on 2009-01-30 01:09:24
Hey, koral,thanks a lot for the awesome new texture viewer, this is EPIC!!!!!

one rather small question though:
How come the files that you supply with the rar work, and files that other people send me work, however, when i dump the files myself, it can't tell the filetype, everything is blank, and i cant view anything. all the files are the right size, so i'm wondering if its vista, trying to compress the files annoyingly or something. any help?
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: ignitz on 2009-01-30 01:34:25
Me Vista too that's the problem  :-D. I try to search in Irrlicht Engine site but not found a fix.
It's worked fine in XP.
I will use the Virtual Machine with XP intall to use this tool.

--------------------------------------------------------

I plus more in my script Dumper fo Crisis Core.

Code: [Select]
// 01911.raw CETRANSLATORS - CRISIS CORE Final Fantasy VII - Script Dump [$27AC0-$27E7F] - Thu Jan 29 18:00:35 2009
//Script extracted by IGNITZ
//www.cetranslators.com.br

// [$27AC0]
<MDB><STRING>
<25TEXT>Materia has leveled up!<END>

// [$27B34]
<STRING>
<27TEXT>Nothing more to do, kupo!<END>

// [$27BAC]
<STRING>
<14TEXT>Load failed.<END>

// [$27BF0]
<STRING>
<16TEXT>Load canceled.<END>

// [$27C3C]
<STRING>
<9TEXT><CHOICE>: Open<END>

// [$27C6C]
<STRING>
<9TEXT><CHOICE>: Push<END>

// [$27C9C]
<STRING>
<10TEXT><CHOICE>: Go up<END>

// [$27CD0]
<1TEXT><???TEXT><CHOICE>: Go down<END>

// [$27D0C]
<STRING>
<17TEXT><CHOICE>: Use elevator<END>

// [$27D5C]
<STRING>
<16TEXT><CHOICE>: Use gondola<END>

// [$27DA8]
<STRING>
<11TEXT><CHOICE>: Rotate<END>

// [$27DE0]
<STRING>
<9TEXT><CHOICE>: Read<END>

// [$27E10]
<STRING>
<9TEXT><CHOICE>: Move<END>

// [$27E40]
<1TEXT><???TEXT><CHOICE>: Examine<END>

// [$27E7C]
<$B5><$5B><$C6><$9E>

<XXXTEXT> Is the number of chars that will be displayed.
but the pointers NOT YET

I founded the Tile Map in the Atel Files.
That will be a reference to file and scenario that you be look.
But it's been a one year and I don't have ane screenshot abnd dont remember XD
but to help to decode the Crisis Core Format, I will try.
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: BlitzNCS on 2009-01-30 17:21:27
Hey, just another bit of constuctive criticism about the viewer.

It's very picky. and in actual fact, half the time, it wont switch between files. it says the filename you want it to, but with the previous filesize.

for instance:

I click on "02177.raw", which is The file containing Cissnei's Regular Texture. It shows that the texture file is X bytes big. i click on the texture and it displays it fine.

However, When i click on "02178.raw", it says that i have indeed clicked on "02178.raw", but the filesize is still X bytes, and hasn't changed. If i then click on the Texture file, it doesn't change, and still displays the texture that is in "02177.raw"

Another time, i'll Do the exact same thing, but the filesize will Change, and i'm able to view "02178.raw"

it seems to do this almost at complete random, is this just a vista problem still, or is it some glitch that you can fix.
Sorry if i didnt explain it too well, i'm bad at that...
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: ultima espio on 2009-01-30 17:22:48
Same thing happens to me NeoCloudstrife, so it's not just you. :-)
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: spidergaby on 2009-01-30 17:29:47
I want help you so here's what i've found :  http://jordansg57.googlepages.com/ look please :)
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: koral on 2009-01-30 18:01:15
whoops, I haven't got vista so I haven't been able to test it, sorry peeps  :-P

NeoCloud, you don't have to explain anymore, I know the problem exactly
the files should work fine with Vista, so maybe the ISO dumper/extractor is doing things a little differently in Vista?
The viewer GUI is really ugly yet its supposed to work fine on Vista. but then again irrlicht is supposed to mainly be used for rendering stuff, it just doesn't hold up as well for GUIs.
(So Ya, its not a criticism, its a fact  :wink: )

I know about those annoying problems when you click on something, and it doesn't change, so you click again and again until it does.
Its weird. Its annoying. But its the best I can do for now, sorry  :oops:

I will have a go and try converting it into a pure win32 application with forms, but i really find MS dependencies annoying. I'll see how things go.


Ignitz, you made a script-dumper? thanks for sharing all this information!  :-D
I'll have a go at adding Script-viewing capabilities into the Viewer, just for the sake of completeness.

I wonder if its possible to edit scripts then re-pack all the data into an ISO and play it? Translations could be done, but otherwise there probably isn't any point really because these days Squenix do a great job with translations


spidergaby
 :-o wow, that viewer's actually running on the PSP?! I've heard about LTE but I never thought of actually viewing the raw data from the PSP itself.

I've been posting all the information I find about the different file-formats etc, so feel free to try make your own viewer! Mine is really quite crap (the GUI especially) and I only made it to help make it easier for ME to find and extract the characters but i don't mind sharing it with everyone :lol:

But thanks for offering to help, just post here to let us all know how you're doing and what you find, the more the merrier   :-D



Nothing new on the model-data unfortunatly

I can't figure out the mesh-data at all, its all a little chaotic inside those [!] files.

[unknown chunk 1] doesn't appear to be anything useful, because most of the data consists of zeros, and not enough shorts (which usually correspond with vertex positions). Maybe its Bone and Skin data combined?

[unknown chunk 2] must contain all the Vertex AND all the Face data because there is no change in the data-flow within that chunk. But it feels a little random because its hard to tell how big each "data-chunk" is, sometimes appearing to be 64-byte chunks, sometimes 48, sometimes 32. It could just be a long list of Vertices (not indexed) but there still should be a more obvious pattern as to where one ends and the next begins.

I remember something about FF8 battle-models having UV's seperated from the the vertex data, maybe its similar to that and [unknown chunk 1] actually contains a list of UV's?

If anyone has ideas then please do spit them out!  :-)
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: ultima espio on 2009-01-30 18:04:32
How about a feature that lets us save the texture so that we dont have to printscreen it? It would be nice to see the texture transparency too. Great Job! :-D :-D

If anyone has ideas then please do spit them out!  :-)

I did lol :wink: :-P
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: spidergaby on 2009-01-30 18:39:27
So "koral" , sorry but i'm french, what was this link ? A model viewer ? You are currently doing a cc model viewer ?

PS: .raw file...I want open them !!! :-D
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: koral on 2009-01-30 23:10:04
espio, I meant ideas regarding the mesh-data 
but I will add the export option in the next version, promise   :wink:

spidergaby: the link is in the first post, but to save you some time: http://www.mediafire.com/?dac2zmtkzjo
its not a model-viewer yet, it can only open some kinds of files (mainly Textures actually) but you have to know what files because most files will be empty (the viewer ignores files it can't understand).



I've found a pattern in the [unknown chunk 1] portion of the [!]-Files which suggest that its Polygon data:

Code: [Select]
[Unknown Chunk 1]
(seems to always start at 0x80)

Possibly Polygon data

for each data-chunk (32 byte-block):

- 1 short always: 00
- 1 short bone id?
- 2 bytes always: FF FF
- 1 byte sometimes: 00, sometimes: 01 -- maybe polygon type (quad / tri)
- 1 byte always: 00
- 1 long always: 00
- 4 bytes ???
- 4 floats polygon normal?

its still a bit weird because there aren't any definite indices, and a lot of the time the entire block consists of zeros (even the 'normals').
Maybe I've made a mistake with the types, but this is the cleanest (readable) structure I can conjour up, and is easier to explain than super-huge or super-negative longs and shorts.

[Unknown Chunk 2] has a header attached with it
Code: [Select]
[Unknown Chunk 2]
Possibly Vertex data

Header: 32-bytes
- 1 long (@0x00)  = number of polygons (data-blocks) in [Unknown Chunk 1]
- 1 long (@0x04)  = offset to data start

Data: ?

the actual data-part remains as mysterious to me as ever, so I'm at a bit of a dead-end right now  :|

If anyone could help me figure this out, I'll be really greatful!
:-D
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: ShinRa Inc on 2009-01-31 00:15:35
Having spent most of the day just trying to browse through the textures, a couple of notes;

I'm using Windows XP and seem to be having the same issue as the vista users on clicking the file and it not loading;  I've found that trying to switch files too quickly is what does that, mostly.  Taking a slower, deliberate approach to select better results.

Something of a dumb question, but is it possible that there can be multiple texture files inside some of the Model .raws?  Some of the larger, more detailed textures seem to only have half the character model on them.

Also, it seems that their's a few environment models/textures around 02400, then it moves back into enemy models.
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: BlitzNCS on 2009-01-31 01:57:57
those "half-character textures" are just high qualiy face textures. that's for when the lips actualy move (close-ups) and isnt just 2 face textures alternating for speech. they dont need the rest of the body because its already as high detailed as needed. i've come across this kinda stuff before in my KHII modding days. to be honest, those model files are probably only the head.

@Koral, if you really need help, i know my way round a hex editor pretty well, enough to search for patterns in code like this. if you want, i suppose i wont mind taking a look for ya.
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: ShinRa Inc on 2009-01-31 03:31:18
those "half-character textures" are just high qualiy face textures. that's for when the lips actualy move (close-ups) and isnt just 2 face textures alternating for speech. they dont need the rest of the body because its already as high detailed as needed. i've come across this kinda stuff before in my KHII modding days. to be honest, those model files are probably only the head.

I thoght so too, but there's a few headless texture sets, too;
(http://img136.imageshack.us/img136/2525/lazardhirezxv9.png)
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: BlitzNCS on 2009-01-31 13:28:05
probably for the HQ models that already have HQ head textures. thats all i can guess at...
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: koral on 2009-01-31 16:07:06
yeesh, I just updated my first post with all the new knowledge that has been posted, and it wasn't easy. Practically working blind not being able to preview how the formatting stayed.
I might have missed some important bits, or confused other things and dumped them into the wrong section, so please be nice  :oops:

Thankyou to everyone who has helped contribute something!  :-D


ShinRa, yes the viewer has an awkard and bulky user-interface and I apologise if its too much of a pain  :-(
I'm trying to work with MSVC's Forms, but Irrlicht isn't behaving very well under the new environment, so I don't know if I will ever be able to fully get away from the irrlicht's GUI.
Just keep clicking something if it doesn't update. There are so many files I can understand the pain of unknowingly missing a few

About those "split-textures", NeoCloudstrife is right they are just higher-quality textures (and models) of the head and torso split like this to make it more efficient. No point loading a super high-quality model if only the head will be shown, right?

And I am absolutely certain that there are only the one Texture chunk inside those [!] files. None of the offsets in the header have led me to multiple textures yet


I'm beginning to believe that the Mesh data is not inside those [!] files, but maybe stored elsewhere. I can't spot any vertices at all inside [unknown-chunk-2] even though there is a lot of data present.
The other alternative is that the data is compressed, encrypted or "interpolated" for PSP-rendering-efficiency.

I wont be able to decipher this one easily, that's for sure.  :|

Maybe its time to look into other files I haven't even considered yet, and wait until I can get some solid leads before returning to these [!] files again
 :-D
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: spidergaby on 2009-01-31 16:32:41
I believe in you courage !!!  :-)
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: ShinRa Inc on 2009-01-31 18:44:03
Not complaining really, Koral, just trying to give some useful feedback.  I'm generally just happy to see some work being done on the newer games (:
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: koral on 2009-01-31 23:17:22
thanks spidergaby, and don't worry about it ShinRa I know you weren't complaining  :-D

ignitz can I ask you something?

I've added in the character-codes to read those [MBD] scripts and everything is working absolutely fine. But its all in one line and I can't figure out what other codes mean.
In your dumps, you used tags like "<NL>" (which obviously mean newline) and "<END>" (end of that person talking?) but I'm not sure what codes they are, how many different ones, etc, etc.

Do you have a list of those "OS" codes?
I think they all start with "40" rather than "80" but I might be wrong.

If you don't then it wont matter, I will just have to use trial-and-error to work out the important ones, so don't go out of your way or anything  :lol:



On a side note, I found that those [ATEL] files also contain [GT] and [!] file-chunks! I totally wasn't expecting them, but they are there, plain as daylight (almost).
the [GT] files are a little different from the Chapter-End images though, and look like they have a different kind of format (I think zande hinted this a LONG time ago).
[!] files will be tricky to load, because I dont think I can make the viewer view chunks in a chunk

Lots of other files might have these kinds of different files too, so I have changed how the viewer handles unknown files now. Instead of ignoring them completely, the viewer will try to find all known file-type chunk headers and list them out regardless. It might help to discover some more interesting things!

I'll upload the new version as soon as I can get these [MBD] scripts showing correctly, which should be in a day or two I think :wink:
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: ignitz on 2009-02-01 07:29:39
Hi Koral.
Sorry to wait.
here my my table now
Code: [Select]
0C000000=<???TEXT>
01000000=<STRING>\n
40020000=<NL>\n
40030000=<UNK1>
40040000=<END>\n\n
40050001=<M-CHOICE1>
40050002=<M-CHIOCE2>
40050003=<M-CHOICE3>
40050004=<M-CHOICE4>
40050005=<M-CHOICE5>
40050006=<M-CHOICE6>
40050007=<M-CHOICE7>
40050008=<M-CHOICE8>
40060000=<CROSS-PAD>
40070000=<CROSS-SQUARE>
402B0000=<VAR1>
402C0000=<VAR2>
402D0000=<VAR3>
40110000=<VAR4>
40120000=<VAR5>
4D424400=<MDB>
40100000=<CHOICE1>
40050101=<CHOICE2>
40360000=<FUNCTION1>
80000000=A
80000100=B
80000200=C
80000300=D
80000400=E
80000500=F
80000600=G
80000700=H
80000800=I
80000900=J
80000A00=K
80000B00=L
80000C00=M
80000D00=N
80000E00=O
80000F00=P
80001000=Q
80001100=R
80001200=S
80001300=T
80001400=U
80001500=V
80001600=W
80001700=X
80001800=Y
80001900=Z
80001A00=a
80001B00=b
80001C00=c
80001D00=d
80001E00=e
80001F00=f
80002000=g
80002100=h
80002200=i
80002300=j
80002400=k
80002500=l
80002600=m
80002700=n
80002800=o
80002900=p
80002A00=q
80002B00=r
80002C00=s
80002D00=t
80002E00=u
80002F00=v
80003000=w
80003100=x
80003200=y
80003300=z
80003400=0
80003500=1
80003600=2
80003700=3
80003800=4
80003900=5
80003A00=6
80003B00=7
80003C00=8
80003D00=9
80003E00=!
80003F00=?
80004000=
80004100="
80004200='
80004300=,
80004400=.
80004500=:
80004600=;
80004700=--
80004800=/
80004900=(
80004A00=)
80004B00=$
80004C00=<
80004D00=>
80004E00=%
80004F00=[
80005000=]
80005100=YEN
80005200=@Cruz
80005300=#
80005400=_
80005500=|
80005600=^
80005700=º
80005800=-
80005900=+
80005A00==
000000000C000000=<0TEXT>
010000000C000000=<1TEXT>
020000000C000000=<2TEXT>
030000000C000000=<3TEXT>
040000000C000000=<4TEXT>
050000000C000000=<5TEXT>
060000000C000000=<6TEXT>
070000000C000000=<7TEXT>
080000000C000000=<8TEXT>
090000000C000000=<9TEXT>
0A0000000C000000=<10TEXT>
0B0000000C000000=<11TEXT>
0C0000000C000000=<12TEXT>
0D0000000C000000=<13TEXT>
0E0000000C000000=<14TEXT>
0F0000000C000000=<15TEXT>
100000000C000000=<16TEXT>
110000000C000000=<17TEXT>
120000000C000000=<18TEXT>
130000000C000000=<19TEXT>
140000000C000000=<20TEXT>
150000000C000000=<21TEXT>
160000000C000000=<22TEXT>
170000000C000000=<23TEXT>
180000000C000000=<24TEXT>
190000000C000000=<25TEXT>
1A0000000C000000=<26TEXT>
1B0000000C000000=<27TEXT>
1C0000000C000000=<28TEXT>
1D0000000C000000=<29TEXT>
1E0000000C000000=<30TEXT>
1F0000000C000000=<31TEXT>
200000000C000000=<32TEXT>
210000000C000000=<33TEXT>
220000000C000000=<34TEXT>
230000000C000000=<35TEXT>
240000000C000000=<36TEXT>
250000000C000000=<37TEXT>
260000000C000000=<38TEXT>
270000000C000000=<39TEXT>
280000000C000000=<40TEXT>
290000000C000000=<41TEXT>
2A0000000C000000=<42TEXT>
2B0000000C000000=<43TEXT>
2C0000000C000000=<44TEXT>
2D0000000C000000=<45TEXT>
2E0000000C000000=<46TEXT>
2F0000000C000000=<47TEXT>
300000000C000000=<48TEXT>
310000000C000000=<49TEXT>
320000000C000000=<50TEXT>
330000000C000000=<51TEXT>
340000000C000000=<52TEXT>
350000000C000000=<53TEXT>
360000000C000000=<54TEXT>
370000000C000000=<55TEXT>
380000000C000000=<56TEXT>
390000000C000000=<57TEXT>
3A0000000C000000=<58TEXT>
3B0000000C000000=<59TEXT>
3C0000000C000000=<60TEXT>
3D0000000C000000=<61TEXT>
3E0000000C000000=<62TEXT>
3F0000000C000000=<63TEXT>
400000000C000000=<64TEXT>
410000000C000000=<65TEXT>
420000000C000000=<66TEXT>
430000000C000000=<67TEXT>
440000000C000000=<68TEXT>
450000000C000000=<69TEXT>
460000000C000000=<70TEXT>
470000000C000000=<71TEXT>
480000000C000000=<72TEXT>
490000000C000000=<73TEXT>
4A0000000C000000=<74TEXT>
4B0000000C000000=<75TEXT>
4C0000000C000000=<76TEXT>
4D0000000C000000=<77TEXT>
4E0000000C000000=<78TEXT>
4F0000000C000000=<79TEXT>
500000000C000000=<80TEXT>
510000000C000000=<81TEXT>
520000000C000000=<82TEXT>
530000000C000000=<83TEXT>
540000000C000000=<84TEXT>
550000000C000000=<85TEXT>
560000000C000000=<86TEXT>
570000000C000000=<87TEXT>
580000000C000000=<88TEXT>
590000000C000000=<89TEXT>
5A0000000C000000=<90TEXT>
5B0000000C000000=<91TEXT>
5C0000000C000000=<92TEXT>
5D0000000C000000=<93TEXT>
5E0000000C000000=<94TEXT>
5F0000000C000000=<95TEXT>
600000000C000000=<96TEXT>
610000000C000000=<97TEXT>
620000000C000000=<98TEXT>
630000000C000000=<99TEXT>
640000000C000000=<100TEXT>
650000000C000000=<101TEXT>
660000000C000000=<102TEXT>
670000000C000000=<103TEXT>
680000000C000000=<104TEXT>
690000000C000000=<105TEXT>
6A0000000C000000=<106TEXT>
6B0000000C000000=<107TEXT>
6C0000000C000000=<108TEXT>
6D0000000C000000=<109TEXT>
6E0000000C000000=<110TEXT>
6F0000000C000000=<111TEXT>
700000000C000000=<112TEXT>
710000000C000000=<113TEXT>
720000000C000000=<114TEXT>
730000000C000000=<115TEXT>
740000000C000000=<116TEXT>
750000000C000000=<117TEXT>
760000000C000000=<118TEXT>
770000000C000000=<119TEXT>
780000000C000000=<120TEXT>
790000000C000000=<121TEXT>
7A0000000C000000=<122TEXT>
7B0000000C000000=<123TEXT>
7C0000000C000000=<124TEXT>
7D0000000C000000=<125TEXT>
7E0000000C000000=<126TEXT>
7F0000000C000000=<127TEXT>
800000000C000000=<128TEXT>
810000000C000000=<129TEXT>
820000000C000000=<130TEXT>
830000000C000000=<131TEXT>
840000000C000000=<132TEXT>
850000000C000000=<133TEXT>
860000000C000000=<134TEXT>
870000000C000000=<135TEXT>
880000000C000000=<136TEXT>
890000000C000000=<137TEXT>
8A0000000C000000=<138TEXT>
8B0000000C000000=<139TEXT>
8C0000000C000000=<140TEXT>
8D0000000C000000=<141TEXT>
8E0000000C000000=<142TEXT>
8F0000000C000000=<143TEXT>
900000000C000000=<144TEXT>
910000000C000000=<145TEXT>
920000000C000000=<146TEXT>
930000000C000000=<147TEXT>
940000000C000000=<148TEXT>
950000000C000000=<149TEXT>
960000000C000000=<150TEXT>
970000000C000000=<151TEXT>
980000000C000000=<152TEXT>
990000000C000000=<153TEXT>
9A0000000C000000=<154TEXT>
9B0000000C000000=<155TEXT>
9C0000000C000000=<156TEXT>
9D0000000C000000=<157TEXT>
9E0000000C000000=<158TEXT>
9F0000000C000000=<159TEXT>
A00000000C000000=<160TEXT>
A10000000C000000=<161TEXT>
A20000000C000000=<162TEXT>
A30000000C000000=<163TEXT>
A40000000C000000=<164TEXT>
A50000000C000000=<165TEXT>
A60000000C000000=<166TEXT>
A70000000C000000=<167TEXT>
A80000000C000000=<168TEXT>
A90000000C000000=<169TEXT>
AA0000000C000000=<170TEXT>
AB0000000C000000=<171TEXT>
AC0000000C000000=<172TEXT>
AD0000000C000000=<173TEXT>
AE0000000C000000=<174TEXT>
AF0000000C000000=<175TEXT>
B00000000C000000=<176TEXT>
B10000000C000000=<177TEXT>
B20000000C000000=<178TEXT>
B30000000C000000=<179TEXT>
B40000000C000000=<180TEXT>
B50000000C000000=<181TEXT>
B60000000C000000=<182TEXT>
B70000000C000000=<183TEXT>
B80000000C000000=<184TEXT>
B90000000C000000=<185TEXT>
BA0000000C000000=<186TEXT>
BB0000000C000000=<187TEXT>
BC0000000C000000=<188TEXT>
BD0000000C000000=<189TEXT>
BE0000000C000000=<190TEXT>
BF0000000C000000=<191TEXT>
C00000000C000000=<192TEXT>
C10000000C000000=<193TEXT>
C20000000C000000=<194TEXT>
C30000000C000000=<195TEXT>
C40000000C000000=<196TEXT>
C50000000C000000=<197TEXT>
C60000000C000000=<198TEXT>
C70000000C000000=<199TEXT>
C80000000C000000=<200TEXT>
C90000000C000000=<201TEXT>
CA0000000C000000=<202TEXT>
CB0000000C000000=<203TEXT>
CC0000000C000000=<204TEXT>
CD0000000C000000=<205TEXT>
CE0000000C000000=<206TEXT>
CF0000000C000000=<207TEXT>
D00000000C000000=<208TEXT>
D10000000C000000=<209TEXT>
D20000000C000000=<210TEXT>
D30000000C000000=<211TEXT>
D40000000C000000=<212TEXT>
D50000000C000000=<213TEXT>
D60000000C000000=<214TEXT>
D70000000C000000=<215TEXT>
D80000000C000000=<216TEXT>
D90000000C000000=<217TEXT>
DA0000000C000000=<218TEXT>
DB0000000C000000=<219TEXT>
DC0000000C000000=<220TEXT>
DD0000000C000000=<221TEXT>
DE0000000C000000=<222TEXT>
DF0000000C000000=<223TEXT>
E00000000C000000=<224TEXT>
E10000000C000000=<225TEXT>
E20000000C000000=<226TEXT>
E30000000C000000=<227TEXT>
E40000000C000000=<228TEXT>
E50000000C000000=<229TEXT>
E60000000C000000=<230TEXT>
E70000000C000000=<231TEXT>
E80000000C000000=<232TEXT>
E90000000C000000=<233TEXT>
EA0000000C000000=<234TEXT>
EB0000000C000000=<235TEXT>
EC0000000C000000=<236TEXT>
ED0000000C000000=<237TEXT>
EE0000000C000000=<238TEXT>
EF0000000C000000=<239TEXT>
F00000000C000000=<240TEXT>
F10000000C000000=<241TEXT>
F20000000C000000=<242TEXT>
F30000000C000000=<243TEXT>
F40000000C000000=<244TEXT>
F50000000C000000=<245TEXT>
F60000000C000000=<246TEXT>
F70000000C000000=<247TEXT>
F80000000C000000=<248TEXT>
F90000000C000000=<249TEXT>
FA0000000C000000=<250TEXT>
FB0000000C000000=<251TEXT>
FC0000000C000000=<252TEXT>
FD0000000C000000=<253TEXT>
FE0000000C000000=<254TEXT>
FF0000000C000000=<255TEXT>

As you can see, after MBD header must have 01 00 00 00 hex code.
This is the start of a string. 01000000=<STRING>\n
An example in 01469.raw
in MBD files have 01000000 0E000000 0C000000
01000000 = Start String
E0000000 = Number of chars EX: 0x0e = 14 decimal
then
MBD  01000000            0E000000               0C000000
        STRING CODE          14 CHAR             UNK CODE (ALL TIME HAVE THIS SAME CODE)
      S              a            v               e       <SPACE>        f             a            i                 l            e             d              {.}       <NL>       <END>
80001200 80001A00 80002F00 80001E00 80004000 80001F00 80001A00 80002200 80002500 80001E00 80001D00 80004400 40020000 40040000
1             2              3             4             5             6             7              8             9             10             11             12             13             14

----------------------------------------------------------------
<MDB><STRING>
<14TEXT>Save failed.<NL>
<END>

----------------------------------------------------------------
01000000                 09000000               0C000000
STRING CODE             9 CHAR               UNK CODE (ALL TIME HAVE THIS SAME CODE)

<VARIABLE>      {:}      <SPACE>       O              p              e           n            <NL>     <END>
40110000     80004500 80004000 80000E00 80002900 80001E00 80002700 40020000 40040000
       1                2             3             4               5             6            7              8             9

----------------------------------------------------------------
<STRING>
<9TEXT><VAR4>: Open<NL>
<END>
----------------------------------------------------------------

In the string is this with TableLib dumper with my table.

Code: [Select]
// 01469.raw CRISISCORE - Final Fantasy VII - Script Dump [$297A0-$29AFF] - Fri Jan 30 04:10:32 2009
//Script extracted by IGNITZ
//www.cetranslators.com.br

// [$297A0]
<MDB><STRING>
<14TEXT>Save failed.<NL>
<END>

// [$297E8]
<STRING>
<9TEXT><VAR4>: Open<NL>
<END>

// [$29818]
<1TEXT><???TEXT><VAR4>: Pick up<NL>
<END>

// [$29854]
<1TEXT><???TEXT><VAR4>: Examine<NL>
<END>

// [$29890]
<STRING>
<9TEXT><VAR4>: Talk<NL>
<END>

// [$298C0]
<STRING>
<17TEXT>Obtained <VAR4> x <VAR5>!<NL>
<END>

// [$29910]
<STRING>
<17TEXT>Obtained <VAR4> gil!<NL>
<END>

// [$29960]
<STRING>
<27TEXT>SP increased by <VAR4> points!<NL>
<END>

// [$299D8]
<STRING>
<23TEXT><CROSS-PAD><FUNCTION1><FUNCTION1>: Save<NL>
<CROSS-SQUARE><FUNCTION1><FUNCTION1>: Delete<NL>
<END>

// [$29A40]
<STRING>
<24TEXT>New mission available!<NL>
<END>

// [$29AAC]
<STRING>
<18TEXT>You've got mail!<NL>
<END>


If you want I can send my files XD with tools, but with this post I cleared the struct of files MBD.
The pointers I don't find yet, it's been maybe in the initial struct Atel file
Or maybe be compressed in before MBD file

TIP: Multiple x4 for the count char to take the size of string.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: spidergaby on 2009-02-01 16:25:32
...oh noooo....what's that with the viewer ?  :-P

(http://img517.imageshack.us/img517/3294/aaaoz6.th.png) (http://img517.imageshack.us/my.php?image=aaaoz6.png)
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-02-01 21:19:50
ignitz, THANKYOU!!!  :lol:
I have it all working properly!

spider, are you using Vista too?
It might be because of the ugly OpenGL support that Vista has, so I have now changed the renderer to Sofware instead of OGL. This should work on all OS's the same.


And here is the latest version (v0.30): http://www.mediafire.com/?y3emvodmyzi

New features and changes:

- Now shows [MBD] scripts as simple textfiles (there is no scroll-bar, but you can use the arrow keys to scroll up or down, or ctrl+A and paste into notepad)
- Export out RAW-data chunks or Textures as PNG images
- Tried to make the User-Interface a little easier on the eyes


I think file "00000.raw" is special, here is a copy of the script:
Code: [Select]
<TEXT>
Where do you want to go?
<M-CHOICE>
 Battle
 Ms. Ishibashi
 Mr. Oka
 Mr. Kitase
 Mr. Kunikata
 Mr. Shindo
 Mr. Terada
 Mr. Maeda
 Moriya kun
<END>

Its a menu which has elements named after the game producers! Maybe its a Debug-Menu?

I'm not sure what I should do next  :|
Models would be nice, but it's useless without vertices

I will just wait I suppose, step down a gear, wait until someone can spot something worth looking into.

Have fun!  :-)
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: BlitzNCS on 2009-02-01 21:37:46
I'm not sure what I should do next  :|
Models would be nice, but it's useless without vertices

I will just wait I suppose, step down a gear, wait until someone can spot something worth looking into.

Have fun!  :-)

Model's would be great. i'll help as much as i can  :wink:
shouldnt be long before we find something!
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: ultima espio on 2009-02-01 22:17:32
i would love to help you look for the vertex data but i dont know what i'm looking for...if you kow what i mean. If someone could help me there i'd appreciate it. :-)
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: ignitz on 2009-02-02 05:06:53
The pointer of MBD files maybe be in teh opcodes or compressed in before of MBD header. I give up.
Without PSP is so hard.

Koral thanks for the tools. Be very cool, you make a nice JOB. :mrgreen:
The fonts work perfect now with your tool, thanks.

Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: ultima espio on 2009-02-02 14:56:55
I was browsing through the text files, and i happened to find this in 01859.raw. The bold bits are what you should take note of.

<TEXT>
Materia has leveled up!
<END>

<TEXT>
Nothing more to do, kupo!
<END>

<TEXT>
Load failed.
<END>

<TEXT>
Load canceled.
<END>

<TEXT>
<CHOICE1>: Open
<END>

<TEXT>
<CHOICE1>: Push
<END>

<TEXT>
<CHOICE1>: Go up
<END>

<TEXT>
<TEXT>
<CHOICE1>: Go down
<END>

<TEXT>
<CHOICE1>: Use elevator
<END>

<TEXT>
<CHOICE1>: Use gondola
<END>

<TEXT>
<CHOICE1>: Rotate
<END>

<TEXT>
<CHOICE1>: Read
<END>

<TEXT>
<CHOICE1>: Move
<END>

<TEXT>
<TEXT>
<CHOICE1>: Examine
<END>
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: BlitzNCS on 2009-02-02 15:03:36
Been doing some looking in the [!] files.

Seems characters models are in the second chunk, that being the largest. i've found vertex data there that is *almost* identical to that of KHII models.
Seems very familliar, particularly because of the "3F80"'s that keep appearing.

I can pretty much confirm that i have definately found 3D model data:

(http://i182.photobucket.com/albums/x74/beefpwnage/comparison-3.png)
if picture is too small, click here (http://i182.photobucket.com/albums/x74/beefpwnage/comparison-3.png)

On the left is some of the data in an MDLX model from KHII, although, what's important is the right side.
now, i'm not a genious, but i have done vertex hacking before, and i THINK i've figured this out...

on the right, we have the crisis core [!] file for tifa open at the model section.
Now, i could be wrong, but this looks very promising.

Section "A" in the tifa model appears to contain information regarding parent/child/sibling verticies and what have you. we can see this by looking at their functions. they both appear to be "LUI"ing "$c123", however, the line above that is "LUI"ing something different. this brings me to (almost) understand the first twelve bytes of Code:

First four bytes: Parent vertex.
Second four bytes: Child Vertex
Third Four bytes: Sibling vertex

now, that would make sense, right? seeing as the picture you can see it at the very first of the datablocks, and it would have no parent verticies.

Section B is completely unknown to me. i have no idea what it is, i only understand that it varies quite a lot. possibly its some sort of "normal" type thing. only speculation though. Apparently, it could also be some sort of rotation information.

If anything contains coordinate information, its Section C. there are THREE two-byte things going on. X, Y, Z. c'mon, looks pretty obvious, wouldnt you say so?

So, to conclude, these datablocks DO vary a little from time to time, with random padding and such, but other than that, i appear to have somewhat deciphered this. If it's no good, koral, it'll be back to the drawing board XD

I REALLY hope this helps you render those models koral :D
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-02-02 18:11:30
 :-D :-D
you're right neocloud, there is definitly an obvious pattern there with 4-dwords and 4-shorts (last short always a zero)
I just tried reading a block of them, but after 10 or so "groups" the data messes up because of the unexplained padding.

But you've confirmed it: mesh data (aka vertices) are most definitly present within the [!] files in [Uknown Chunk 2]

To call them "character files" would be a bit of an understatement, because I've seen simple things like barrels too. Bones are probably optional though, but maybe what they really are Animated Model Files.

I noticed that the [&&] files-chunks (also inside [ATEL] files) are almost identical to the [!] files except the header is smaller and contains less inner-sections. So those are most likely Static-Mesh Model files (maybe the model of the Location Map itself?).

No need to go back to the drawing-board, if we can figure out how the padding seems to work then that should allow us to get have enough vertices to render out.
i think there might be some sort of tag or code (like in [MBD] files) which determines how the current or next "group" should be read.

Or maybe they are Polygons? Then the multiple-dwords could contain indices for more than 3 vertices, explaining for the strange grouping.
Or maybe the groups are intentionally split based on the data from [unknown chunk 1], linked with the bones that they're attached with?

lots of possibilities! But at least you've given me something to think about  :-P


ultimo espio I noticed that too, those MBD scripts are in almost all [ATEL] files!
For some reason, the developers decided to use a Moogle and give it politely accurate scripts?!

I can only guess that its linked with the Debug-Menu thing, an event which can optionally be switched "on" but is off by default.
I wonder if there might be a keycode associated with it, or if its something which has to run during the game bootup. Could we somehow excecute the 00000.raw MBD manually?

Interesting possibilities right there
 :lol:
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-02-02 20:18:58
okay, I've just stumbled across a possible 4-byte "vertex" tag in [unknown-chunk-2] :
(hex) 00 00 80 3f 
(dec) 00 00 128 63 

The size of these groupings tend to vary, but there is definitly something going on here

for example, file 02192.raw (Tifa), [unknown-chunk-2] bytes read 4 at a time:
(tag shown 'underlined')

Code: [Select]
0, 0, 0, 0,
1, 0, 12, 0,
0, 0, 1, 2,
35, 7, 0, 0,
0, 0, 3, 0,
24, 0, 0, 0,
0, 0, 128, 63,
-----------------------  1 
25, 91, 1, 63,
35, 193, 67, 63,
134, 34, 0, 0,
133, 246, 42, 79,
179, 254, 0, 0,
0, 0, 128, 63,
-----------------------  2 
130, 6, 16, 63,
35, 193, 67, 63,
189, 52, 163, 0,
0, 249, 154, 79,
130, 250, 0, 0,
0, 0, 128, 63,
-----------------------  3 
57, 241, 15, 63,
15, 218, 78, 63,
172, 34, 169, 0,
83, 248, 191, 75,
40, 249, 0, 0,
0, 0, 128, 63,
-----------------------  4 
57, 241, 15, 63,
15, 218, 78, 63,
84, 34, 169, 0,
173, 7, 191, 75,
40, 249, 0, 0,
0, 0, 128, 63,
-----------------------  5 
207, 170, 37, 63,
15, 218, 78, 63,
0, 37, 135, 0,
0, 0, 123, 75,
83, 247, 0, 0,
0, 0, 128, 63,
-----------------------  6 
130, 6, 16, 63,
35, 193, 67, 63,
67, 52, 163, 0,
0, 7, 154, 79,
130, 250, 0, 0,
0, 0, 128, 63,
-----------------------  7 
252, 50, 69, 62,
88, 11, 157, 62,
61, 146, 14, 0,
226, 254, 67, 68,
124, 254, 0, 0,
0, 0, 128, 63,
-----------------------  8 
163, 201, 85, 62,
108, 110, 153, 62,
0, 134, 222, 0,
0, 0, 179, 69,
218, 250, 0, 0,
0, 0, 128, 63,
-----------------------  9 
252, 50, 69, 62,
72, 197, 150, 62,
195, 146, 14, 0,
30, 1, 67, 68,
124, 254, 0, 0,
0, 0, 128, 63,
-----------------------  10 
161, 32, 66, 62,
198, 58, 239, 62,
61, 146, 14, 0,
226, 254, 67, 68,
124, 254, 0, 0,
0, 0, 128, 63,
-----------------------  11 
155, 32, 66, 62,
38, 112, 244, 62,
195, 146, 14, 0,
30, 1, 67, 68,
124, 254, 0, 0,
0, 0, 128, 63,
-----------------------  12 
4, 232, 82, 62,
4, 199, 241, 62,
0, 151, 71, 0,
0, 0, 108, 69,
165, 2, 0, 0,
0, 0, 128, 63,
-----------------------  13 
57, 241, 15, 63,
15, 218, 78, 63,
84, 34, 169, 0,
173, 7, 191, 75,
40, 249, 0, 0,
0, 0, 128, 63,
-----------------------  14 
130, 6, 16, 63,
35, 193, 67, 63,
67, 52, 163, 0,
0, 7, 154, 79,
130, 250, 0, 0,
0, 0, 128, 63,
-----------------------  15 
25, 91, 1, 63,
35, 193, 67, 63,
122, 34, 0, 0,
123, 9, 42, 79,
179, 254, 0, 0,
0, 0, 128, 63,
-----------------------  16 
130, 6, 16, 63,
35, 193, 67, 63,
189, 52, 163, 0,
0, 249, 154, 79,
130, 250, 0, 0,
0, 0, 128, 63,
-----------------------  17 
207, 170, 37, 63,
15, 218, 78, 63,
0, 37, 135, 0,
0, 0, 123, 75,
83, 247, 0, 0,
0, 0, 128, 63,
-----------------------  18 
57, 241, 15, 63,
15, 218, 78, 63,
172, 34, 169, 0,
83, 248, 191, 75,
40, 249, 0, 0,
0, 0, 128, 63,
-----------------------  19 
130, 6, 16, 63,
35, 193, 67, 63,
67, 52, 163, 0,
0, 7, 154, 79,
130, 250, 0, 0,
0, 0, 128, 63,
-----------------------  20 
207, 170, 37, 63,
15, 218, 78, 63,
0, 37, 135, 0,
0, 0, 123, 75,
83, 247, 0, 0,
0, 0, 128, 63,
-----------------------  21 
207, 170, 37, 63,
35, 193, 67, 63,
0, 48, 139, 0,
0, 0, 204, 79,
144, 249, 0, 0,
0, 0, 128, 63,
-----------------------  22 
207, 170, 37, 63,
35, 193, 67, 63,
0, 48, 139, 0,
0, 0, 204, 79,
144, 249, 0, 0,
0, 0, 128, 63,
-----------------------  23 
207, 170, 37, 63,
15, 218, 78, 63,
0, 37, 135, 0,
0, 0, 123, 75,
83, 247, 0, 0,
0, 0, 128, 63,
-----------------------  24 
130, 6, 16, 63,
35, 193, 67, 63,
189, 52, 163, 0,
0, 249, 154, 79,
130, 250, 0, 0,
1, 0, 16, 0,
0, 0, 3, 2,
35, 135, 0, 0,
0, 0, 3, 0,
1, 0, 5, 0,
2, 0, 6, 0,
48, 0, 0, 0,
0, 0, 0, 0,
102, 102, 102, 63,
205, 204, 204, 61,
42, 207, 154, 62,
235, 142, 87, 63,
133, 29, 255, 0,
41, 249, 108, 86,
161, 0, 0, 0,
0, 0, 0, 0,
51, 51, 115, 63,
205, 204, 76, 61,
203, 247, 156, 62,
39, 8, 74, 63,
186, 21, 153, 0,
110, 251, 143, 86,
177, 252, 0, 0,
0, 0, 128, 63,
-----------------------  25 
0, 0, 0, 0,
0, 0, 0, 0,
74, 97, 186, 62,
6, 24, 76, 63,
189, 52, 163, 0,
0, 249, 154, 79,
130, 250, 0, 0,
0, 0, 128, 63,
-----------------------  26 
0, 0, 0, 0,
0, 0, 0, 0,
74, 97, 186, 62,
6, 24, 76, 63,
67, 52, 163, 0,
0, 7, 154, 79,
130, 250, 0, 0,
0, 0, 128, 63,
-----------------------  27 
0, 0, 0, 0,
0, 0, 0, 0,
74, 97, 186, 62,
236, 114, 61, 63,
0, 48, 139, 0,
0, 0, 204, 79,
144, 249, 0, 0,
0, 0, 0, 0,
51, 51, 115, 63,
205, 204, 76, 61,
203, 247, 156, 62,
39, 8, 74, 63,
70, 21, 153, 0,
146, 4, 143, 86,
177, 252, 0, 0,
0, 0, 128, 63,
-----------------------  28 
0, 0, 0, 0,
0, 0, 0, 0,
74, 97, 186, 62,
92, 22, 98, 63,
87, 14, 90, 0,
223, 6, 162, 78,
63, 4, 0, 0,
0, 0, 0, 0,
102, 102, 102, 63,
205, 204, 204, 61,
42, 207, 154, 62,
235, 142, 87, 63,
123, 29, 255, 0,
215, 6, 108, 86,
161, 0, 0, 0,
0, 0, 0, 0,
102, 102, 102, 63,
205, 204, 204, 61,
74, 132, 137, 62,
57, 65, 98, 63,
73, 253, 103, 0,
148, 4, 147, 87,
249, 4, 0, 0,
0, 0, 0, 0,
102, 102, 102, 63,
205, 204, 204, 61,
42, 207, 154, 62,
235, 142, 87, 63,
133, 29, 255, 0,
41, 249, 108, 86,
161, 0, 0, 0,
0, 0, 128, 63,
-----------------------  29 
0, 0, 0, 0,
0, 0, 0, 0,
74, 97, 186, 62,
6, 24, 76, 63,
189, 52, 163, 0,
0, 249, 154, 79,
130, 250, 0, 0,
0, 0, 128, 63,
-----------------------  30 
0, 0, 0, 0,
0, 0, 0, 0,
74, 97, 186, 62,
120, 24, 86, 63,
134, 34, 0, 0,
133, 246, 42, 79,
179, 254, 0, 0,
0, 0, 128, 63,
-----------------------  31 
0, 0, 0, 0,
0, 0, 0, 0,
74, 97, 186, 62,
92, 22, 98, 63,
169, 14, 90, 0,
33, 249, 162, 78,
63, 4, 0, 0,
0, 0, 0, 0,
102, 102, 102, 63,
205, 204, 204, 61,
42, 207, 154, 62,
235, 142, 87, 63,
133, 29, 255, 0,
41, 249, 108, 86,
161, 0, 0, 0,
0, 0, 128, 63,
-----------------------  32 
0, 0, 0, 0,
0, 0, 0, 0,
74, 97, 186, 62,
120, 24, 86, 63,
134, 34, 0, 0,
133, 246, 42, 79,
179, 254, 0, 0,
0, 0, 0, 0,
51, 51, 115, 63,
205, 204, 76, 61,
203, 247, 156, 62,
39, 8, 74, 63,
70, 21, 153, 0,
146, 4, 143, 86,
177, 252, 0, 0,
0, 0, 128, 63,
-----------------------  33 
0, 0, 0, 0,
0, 0, 0, 0,
74, 97, 186, 62,
236, 114, 61, 63,
0, 48, 139, 0,
0, 0, 204, 79,
144, 249, 0, 0,
0, 0, 0, 0,
51, 51, 115, 63,
205, 204, 76, 61,
65, 168, 157, 62,
238, 114, 61, 63,
0, 0, 130, 0,
0, 0, 147, 86,
4, 252, 0, 0,
0, 0, 128, 63,
-----------------------  34 
0, 0, 0, 0,
0, 0, 0, 0,
74, 97, 186, 62,
120, 24, 86, 63,
122, 34, 0, 0,
123, 9, 42, 79,
179, 254, 0, 0,
0, 0, 0, 0,
102, 102, 102, 63,
205, 204, 204, 61,
42, 207, 154, 62,
235, 142, 87, 63,
123, 29, 255, 0,
215, 6, 108, 86,
161, 0, 0, 0,
0, 0, 128, 63,
-----------------------  35 
0, 0, 0, 0,
0, 0, 0, 0,
74, 97, 186, 62,
92, 22, 98, 63,
87, 14, 90, 0,
223, 6, 162, 78,
63, 4, 0, 0,
0, 0, 128, 63,
-----------------------  36 
0, 0, 0, 0,
0, 0, 0, 0,
74, 97, 186, 62,
6, 24, 76, 63,
67, 52, 163, 0,
0, 7, 154, 79,
130, 250, 0, 0,
0, 0, 0, 0,
51, 51, 115, 63,
205, 204, 76, 61,
203, 247, 156, 62,
39, 8, 74, 63,
70, 21, 153, 0,
146, 4, 143, 86,
177, 252, 0, 0,
0, 0, 0, 0,
102, 102, 102, 63,
205, 204, 204, 61,
42, 207, 154, 62,
235, 142, 87, 63,
123, 29, 255, 0,
215, 6, 108, 86,
161, 0, 0, 0,
0, 0, 128, 63,
-----------------------  37 
0, 0, 0, 0,
0, 0, 0, 0,
74, 97, 186, 62,
3, 98, 111, 63,
0, 255, 126, 0,
0, 0, 67, 78,
207, 5, 0, 0,
0, 0, 0, 0,
102, 102, 102, 63,
205, 204, 204, 61,
74, 132, 137, 62,
57, 65, 98, 63,
73, 253, 103, 0,
148, 4, 147, 87,
249, 4, 0, 0,
0, 0, 0, 0,
51, 51, 115, 63,
205, 204, 76, 61,
34, 19, 149, 62,
3, 98, 111, 63,
0, 250, 126, 0,
0, 0, 145, 87,
222, 5, 0, 0,
0, 0, 0, 0,
51, 51, 115, 63,
205, 204, 76, 61,
65, 168, 157, 62,
238, 114, 61, 63,
0, 0, 130, 0,
0, 0, 147, 86,
4, 252, 0, 0,
0, 0, 128, 63,
-----------------------  38 
0, 0, 0, 0,
0, 0, 0, 0,
74, 97, 186, 62,
236, 114, 61, 63,
0, 48, 139, 0,
0, 0, 204, 79,
144, 249, 0, 0,
0, 0, 0, 0,
51, 51, 115, 63,
205, 204, 76, 61,
203, 247, 156, 62,
39, 8, 74, 63,
186, 21, 153, 0,
110, 251, 143, 86,
177, 252, 0, 0,
0, 0, 128, 63,
-----------------------  39 
0, 0, 0, 0,
0, 0, 0, 0,
74, 97, 186, 62,
120, 24, 86, 63,
122, 34, 0, 0,
123, 9, 42, 79,
179, 254, 0, 0,
0, 0, 128, 63,
-----------------------  40 
0, 0, 0, 0,
0, 0, 0, 0,
74, 97, 186, 62,
6, 24, 76, 63,
67, 52, 163, 0,
0, 7, 154, 79,
130, 250, 0, 0,
0, 0, 0, 0,
102, 102, 102, 63,
205, 204, 204, 61,
42, 207, 154, 62,
235, 142, 87, 63,
123, 29, 255, 0,
215, 6, 108, 86,
161, 0, 0, 0,
0, 0, 0, 0,
51, 51, 115, 63,
205, 204, 76, 61,
34, 19, 149, 62,
3, 98, 111, 63,
0, 250, 126, 0,
0, 0, 145, 87,
222, 5, 0, 0,
0, 0, 0, 0,
102, 102, 102, 63,
205, 204, 204, 61,
74, 132, 137, 62,
57, 65, 98, 63,
183, 253, 103, 0,
108, 251, 147, 87,
249, 4, 0, 0,
0, 0, 128, 63,
-----------------------  41 
0, 0, 0, 0,
0, 0, 0, 0,
74, 97, 186, 62,
3, 98, 111, 63,
0, 255, 126, 0,
0, 0, 67, 78,
207, 5, 0, 0,
0, 0, 0, 0,
51, 51, 115, 63,
205, 204, 76, 61,
203, 247, 156, 62,
39, 8, 74, 63,
186, 21, 153, 0,
110, 251, 143, 86,
177, 252, 0, 0,
0, 0, 128, 63,
-----------------------  42 
0, 0, 0, 0,
0, 0, 0, 0,
74, 97, 186, 62,
236, 114, 61, 63,
0, 48, 139, 0,
0, 0, 204, 79,
144, 249, 0, 0,
0, 0, 128, 63,
-----------------------  43 
0, 0, 0, 0,
0, 0, 0, 0,
74, 97, 186, 62,
6, 24, 76, 63,
189, 52, 163, 0,
0, 249, 154, 79,
130, 250, 0, 0,
0, 0, 0, 0,
102, 102, 102, 63,
205, 204, 204, 61,
74, 132, 137, 62,
57, 65, 98, 63,
183, 253, 103, 0,
108, 251, 147, 87,
249, 4, 0, 0,
0, 0, 0, 0,
102, 102, 102, 63,
205, 204, 204, 61,
42, 207, 154, 62,
235, 142, 87, 63,
133, 29, 255, 0,
41, 249, 108, 86,
161, 0, 0, 0,
0, 0, 128, 63,
-----------------------  44 
0, 0, 0, 0,
0, 0, 0, 0,
74, 97, 186, 62,
92, 22, 98, 63,
169, 14, 90, 0,
33, 249, 162, 78,
63, 4, 0, 0,
0, 0, 128, 63,
-----------------------  45 
0, 0, 0, 0,
0, 0, 0, 0,
74, 97, 186, 62,
3, 98, 111, 63,
0, 255, 126, 0,
0, 0, 67, 78,
207, 5, 0, 0,
0, 0, 0, 0,
102, 102, 102, 63,
205, 204, 204, 61,
74, 132, 137, 62,
57, 65, 98, 63,
183, 253, 103, 0,
108, 251, 147, 87,
249, 4, 0, 0,
0, 0, 128, 63,
-----------------------  46 
0, 0, 0, 0,
0, 0, 0, 0,
74, 97, 186, 62,
92, 22, 98, 63,
169, 14, 90, 0,
33, 249, 162, 78,
63, 4, 0, 0,
0, 0, 128, 63,
-----------------------  47 
0, 0, 0, 0,
0, 0, 0, 0,
74, 97, 186, 62,
92, 22, 98, 63,
87, 14, 90, 0,
223, 6, 162, 78,
63, 4, 0, 0,
0, 0, 0, 0,
102, 102, 102, 63,
205, 204, 204, 61,
74, 132, 137, 62,
57, 65, 98, 63,
73, 253, 103, 0,
148, 4, 147, 87,
249, 4, 0, 0,
0, 0, 128, 63,
-----------------------  48 
0, 0, 0, 0,
0, 0, 0, 0,
74, 97, 186, 62,
3, 98, 111, 63,
0, 255, 126, 0,
0, 0, 67, 78,
207, 5, 0, 0,
1, 0, 14, 0,
0, 0, 2, 2,
35, 71, 0, 0,
0, 0, 3, 0,
1, 0, 76, 0,
3, 0, 0, 0,
0, 0, 128, 63,
-----------------------  49 
0, 0, 0, 0,
57, 241, 15, 63,
15, 218, 78, 63,
172, 34, 169, 0,
83, 248, 191, 75,
40, 249, 0, 0,
0, 0, 0, 0,
0, 0, 128, 63,
-----------------------  50 
25, 91, 1, 63,
28, 63, 90, 63,
132, 11, 237, 0,
193, 244, 171, 70,
122, 254, 0, 0,
0, 0, 128, 63,
-----------------------  51 
0, 0, 0, 0,
25, 91, 1, 63,
35, 193, 67, 63,
134, 34, 0, 0,
133, 246, 42, 79,
179, 254, 0, 0,


this tag is also present in other [!] files, leading to similar breaks in the data, so its definitly not a coincidence (is anything a coincidence?  :-P)

i would have thought that the first few bytes after the tag would suggest something about the length of the data-block to come (something like a header) but there doesn't appear to be any kind of differentiating data until after 6 lines (24-bytes), when the tag doesn't appear again

could someone (neoCloud?) try to examine the bytes within these tag-groups to see if they can spot an obvious pattern of data?
I will try too, but it would help if I had other possible patterns to compare against, it would definitly help speed things up!  :wink:
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-02-02 22:28:59
yay, I think I found out roughly how the vertex data might be stored like!

Starting from the beggining of [Unknown-Chunk-2]:  (again taking 02192.raw [Tifa] as reference)
Code: [Select]
Header: 24-bytes
  - 0x12  :  SHORT = number of Vertices to follow

Vertices: 28-byte blocks (times number in header)

  - LONG      = vertex-Tag (00 00 80 3f)

  - SHORT  1  = ?
  - SHORT  2  = X
  - SHORT  3  = Y
  - SHORT  4  = Z
  - BYTE   1  = U
  - BYTE   2  = V

  - LONG     = ? 4-bytes all zeros


Other data: ?? no idea


For the Tifa model, the header only gives 24 vertices initially, so it really isn't much to play with.
Here are the exact numbers out of the file: (vertex-tags hidden)

Code: [Select]

0, 0,   1, 12,    0, 0 ,3, 0,   24, 0, 0, 0                                // 24-byte header
23321, 16129,   -16093, 16195,    179, 254 ,0, 0,   1, 0, 16, 0            // vertices: ?, X,Y,Z, U,V, ? (shown as 4 bytes instead of long)
1666, 16144,   -16093, 16195,    130, 250 ,0, 0,   0, 0, 0, 0
-3783, 16143,   -9713, 16206,    40, 249 ,0, 0,   0, 0, 0, 0
-3783, 16143,   -9713, 16206,    40, 249 ,0, 0,   0, 0, 0, 0
-21809, 16165,   -9713, 16206,    83, 247 ,0, 0,   0, 0, 0, 0
1666, 16144,   -16093, 16195,    130, 250 ,0, 0,   0, 0, 0, 0
13052, 15941,   2904, 16029,    124, 254 ,0, 0,   0, 0, 0, 0
-13917, 15957,   28268, 16025,    218, 250 ,0, 0,   0, 0, 0, 0
13052, 15941,   -15032, 16022,    124, 254 ,0, 0,   0, 0, 0, 0
8353, 15938,   15046, 16111,    124, 254 ,0, 0,   0, 0, 0, 0
8347, 15938,   28710, 16116,    124, 254 ,0, 0,   0, 0, 0, 0
-6140, 15954,   -14588, 16113,    165, 2 ,0, 0,   0, 0, 0, 0
-3783, 16143,   -9713, 16206,    40, 249 ,0, 0,   0, 0, 0, 0
1666, 16144,   -16093, 16195,    130, 250 ,0, 0,   0, 0, 0, 0
23321, 16129,   -16093, 16195,    179, 254 ,0, 0,   0, 0, 0, 0
1666, 16144,   -16093, 16195,    130, 250 ,0, 0,   0, 0, 0, 0
-21809, 16165,   -9713, 16206,    83, 247 ,0, 0,   0, 0, 0, 0
-3783, 16143,   -9713, 16206,    40, 249 ,0, 0,   0, 0, 0, 0
1666, 16144,   -16093, 16195,    130, 250 ,0, 0,   0, 0, 0, 0
-21809, 16165,   -9713, 16206,    83, 247 ,0, 0,   0, 0, 0, 0
-21809, 16165,   -16093, 16195,    144, 249 ,0, 0,   0, 0, 0, 0
-21809, 16165,   -16093, 16195,    144, 249 ,0, 0,   0, 0, 0, 0
-21809, 16165,   -9713, 16206,    83, 247 ,0, 0,   0, 0, 0, 0
1666, 16144,   -16093, 16195,    130, 250 ,0, 0,   1, 0, 16, 0
35, 135 ,0, 0,   0, 0, 3, 0                                      // data becomes messy afterwards, new header?
2, 0 ,6, 0,   48, 0, 0, 0
102, 102 ,102, 63,   205, 204, 204, 61
235, 142 ,87, 63,   133, 29, 255, 0
161, 0 ,0, 0,   0, 0, 0, 0
205, 204 ,76, 61,   203, 247, 156, 62
186, 21 ,153, 0,   110, 251, 143, 86


Funnily enough, I couldn't get a clean output from any of the other [!] model files. A lot of stuff is still rather fishy with my findings, but its a start  :wink:

The XYZ numbers don't feel right to me though, maybe too large and slightly rectangular.
I'll try to render these vertexes and see what they throw out. For all I know, it might just be gobbledegook  :lol:
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: BlitzNCS on 2009-02-02 22:54:15
Hey, i've got a question:
Me and a friend are trying to hack Crisis core. and with all this info, it'd be easy. only problem is, the file dumper dumps files in .raw format. this isnt the format that they originally are in the PKG. for instance, 02123.raw, which is zack's [!] file, isn't in the actual game as 02123.raw, or 02123.[!] or anything. does anyone know what they'd be called in the actual PKG file?
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: G on 2009-02-03 09:22:02
Quote
does anyone know what they'd be called in the actual PKG file?
I think, they are known only to Square developers.

Anyway, I can set proper extensions/filenames for files, if anyone would provide details, which file should be which, or better propose:
I would create text map file, where there would be possibility to set them by simply modification of it, for example:
0001 - "system.dat"
0002 - "text.bin"
...
and so on...

Current numeration order is based of course on offsets order.

Besides - great work on formats, koral!
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-02-03 09:57:17
bahahaha *spongebob impersonation* bahahaha  :-D

I was wrong! The data isn't vertex positions, its UVs!

Code: [Select]
FLOAT, FLOAT, (8 unknown bytes making up 24-byte block)
File: 02192.raw
Code: [Select]
0.000000, 0.000000,  0, 0 ,3, 0,   24, 0, 0, 0           // header
0.505296, 0.764666,  179, 254 ,0, 0,   1, 0, 16, 0       // 2 floats = UV
0.562599, 0.764666,  130, 250 ,0, 0,   0, 0, 0, 0
0.562275, 0.808015,  40, 249 ,0, 0,   0, 0, 0, 0
0.562275, 0.808015,  40, 249 ,0, 0,   0, 0, 0, 0
0.647138, 0.808015,  83, 247 ,0, 0,   0, 0, 0, 0
0.562599, 0.764666,  130, 250 ,0, 0,   0, 0, 0, 0
0.192577, 0.306727,  124, 254 ,0, 0,   0, 0, 0, 0
0.208777, 0.299671,  218, 250 ,0, 0,   0, 0, 0, 0
0.192577, 0.294474,  124, 254 ,0, 0,   0, 0, 0, 0
0.189578, 0.467245,  124, 254 ,0, 0,   0, 0, 0, 0
0.189578, 0.477418,  124, 254 ,0, 0,   0, 0, 0, 0
0.205963, 0.472221,  165, 2 ,0, 0,   0, 0, 0, 0
0.562275, 0.808015,  40, 249 ,0, 0,   0, 0, 0, 0
0.562599, 0.764666,  130, 250 ,0, 0,   0, 0, 0, 0
0.505296, 0.764666,  179, 254 ,0, 0,   0, 0, 0, 0
0.562599, 0.764666,  130, 250 ,0, 0,   0, 0, 0, 0
0.647138, 0.808015,  83, 247 ,0, 0,   0, 0, 0, 0
0.562275, 0.808015,  40, 249 ,0, 0,   0, 0, 0, 0
0.562599, 0.764666,  130, 250 ,0, 0,   0, 0, 0, 0
0.647138, 0.808015,  83, 247 ,0, 0,   0, 0, 0, 0
0.647138, 0.764666,  144, 249 ,0, 0,   0, 0, 0, 0
0.647138, 0.764666,  144, 249 ,0, 0,   0, 0, 0, 0
0.647138, 0.808015,  83, 247 ,0, 0,   0, 0, 0, 0
0.562599, 0.764666,  130, 250 ,0, 0,   1, 0, 16, 0
35, 135 ,0, 0,   0, 0, 3, 0
2, 0 ,6, 0,   48, 0, 0, 0
102, 102 ,102, 63,   205, 204, 204, 61
...

vertex positions probably folllow afterwards as another block of data until UV's start again. But now we know that there would be the same number of XYZs as UVs, so it should be a lot easier to solve!

G, this wouldn't have been possible without your extractor in the first place!  :-D

NeoCloud, just call them whatever you want (like G said before in the other thread, .RAW is there just for the sake of having an extension). As long as we agree on using the same type-extensions everywhere, it shouldn't cause a problem .
I called the [!] files "exp-files" (.exp) and [&&] "and-files" (.and) inside my program, the other file-types are simple enough to call .ATEL etc

it doesn't matter what the original developers called them as long as we know what they are  :lol:
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: BlitzNCS on 2009-02-03 15:01:12
 :-o
UV's!? wow :D
well at least we're gonna be able to apply textures easily now without having to re-map everything :wink:
This is going a LOT quicker than i had once thought it might :D

so, if i'm right, once the pattern i found stops being recognisable as the same pattern, (i.e once the UV maps finish), we're probably onto the vertex positions?
so i'm guessing its not going to be months and months before we're able to render these models? :D

*very happy*
alright, i'll do my best to keep looking!
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-02-03 15:08:51
wow, I'm wrong again!  :oops:
(sorry to dash your hopes so soon neocloud)

I think my first guess was right, because when I open other [!]-files i get "better" results (still not what I expect to find, but definitly not gobbledegook).

There really shouldn't be any FLOATS at all, SHORTS are far more compact (2 bytes instead of 4).
Or maybe its using 2-Byte FLOATS?! Its worth looking into. How were vertices stored in KH2? SHORTS or FLOATS?

The simplest [!]-File I could find was File: 02450.raw, which looks like it might be a barrel.
It has 84 vertices though, which feels like too much vertices or faces if its supposed to be only a simple barrel.
But there is no doubt that the 16-byte (20 including tag) is wrong at all.

Here is the complete dump of the SHORTS making up [unknown-chunk-2] in 02450.raw:
(## = vertex-tag 3f800000)

Code: [Select]
0, 0,   1, 12,    0, 3,   84, 0 ##
2907, 16127,   -24692, 16011,    -8118, 0,   0, 0 ##
-15208, 15156,   -17924, 16127,    0, 0,   0, 0 ##
2907, 16127,   -17924, 16127,    0, 0,   0, 0 ##
2907, 16127,   -24710, 16011,    8118, 0,   0, 0 ##
-15208, 15156,   -17942, 16127,    0, 0,   0, 0 ##
2907, 16127,   -17944, 16127,    0, 0,   0, 0 ##
4361, 16177,   -24576, -17783,    11481, 0,   0, 0 ##
19232, 16210,   3848, 15834,    8118, 0,   0, 0 ##
4361, 16177,   -20260, 16187,    -11481, 0,   0, 0 ##
4361, 16177,   -20260, 16187,    -11481, 0,   0, 0 ##
19232, 16210,   3848, 15834,    8118, 0,   0, 0 ##
4361, 16177,   -24576, -17783,    11481, 0,   0, 0 ##
12793, 16175,   -24576, -17783,    11481, 0,   0, 0 ##
-2074, 16141,   3848, 15834,    8118, 0,   0, 0 ##
12793, 16175,   -20260, 16187,    -11481, 0,   0, 0 ##
12788, 16175,   -20260, 16187,    -11481, 0,   0, 0 ##
-2079, 16141,   3848, 15834,    8118, 0,   0, 0 ##
12788, 16175,   -24576, -17783,    11481, 0,   0, 0 ##
-15208, 15156,   -14022, 16254,    11481, 0,   0, 0 ##
2907, 16127,   -14022, 16254,    11481, 0,   0, 0 ##
2907, 16127,   -5578, 16185,    8118, 0,   0, 0 ##
-15208, 15156,   -5578, 16185,    8118, 0,   0, 0 ##
-15208, 15156,   -14022, 16254,    11481, 0,   0, 0 ##
2907, 16127,   -5578, 16185,    8118, 0,   0, 0 ##
-15208, 15156,   -24692, 16011,    -8118, 0,   0, 0 ##
2907, 16127,   -24692, 16011,    -8118, 0,   84, 0 ##
2907, 16127,   -17152, 15216,    -11481, 0,   904, 665 ##
-15208, 15156,   -17152, 15216,    -11481, 0,   688, 665 ##
-15208, 15156,   -24692, 16011,    -8118, 0,   472, 665 ##
2907, 16127,   -17152, 15216,    -11481, 0,   472, 665 ##
-16067, 15156,   -14023, 16254,    -11481, 0,   472, 665 ##
2907, 16127,   -14022, 16254,    -11481, 0,   472, 665 ##
2907, 16127,   -5587, 16185,    -8118, 0,   472, 665 ##
-15208, 15156,   -5587, 16185,    -8118, 0,   31, 0 ##
-15208, 15156,   -14023, 16254,    -11481, 0,   -25968, 665 ##
2907, 16127,   -5587, 16185,    -8118, 0,   -27404, 66 ##
-15208, 15156,   -18688, 15216,    11481, 0,   1, 0 ##
-15208, 15156,   -24710, 16011,    8118, 0,   20, 0 ##
2907, 16127,   -18688, 15216,    11481, 0,   31, 0 ##
-15208, 15156,   -24710, 16011,    8118, 0,   -24016, 665 ##
2907, 16127,   -24710, 16011,    8118, 0,   -27404, 66 ##
2907, 16127,   -18688, 15216,    11481, 0,   1, 0 ##
-15208, 15156,   -5578, 16185,    8118, 0,   20, 0 ##
2907, 16127,   -17924, 16127,    0, 0,   31, 0 ##
-15208, 15156,   -17924, 16127,    0, 0,   24160, 665 ##
-15208, 15156,   -5587, 16185,    -8118, 0,   -27404, 66 ##
2907, 16127,   -17944, 16127,    0, 0,   1, 0 ##
-15208, 15156,   -17942, 16127,    0, 0,   20, 0 ##
3699, 16224,   27664, 16059,    0, 0,   31, 0 ##
19232, 16210,   10801, 16160,    -8118, 0,   31488, 665 ##
19232, 16210,   3848, 15834,    8118, 0,   0, 0 ##
19232, 16210,   10801, 16160,    -8118, 0,   0, 0 ##
4361, 16177,   -20260, 16187,    -11481, 0,   0, 0 ##
19232, 16210,   3848, 15834,    8118, 0,   0, 0 ##
19232, 16210,   10801, 16160,    -8118, 0,   0, 0 ##
19232, 16210,   3848, 15834,    8118, 0,   0, 0 ##
4361, 16177,   -20260, 16187,    -11481, 0,   0, 0 ##
3699, 16224,   27664, 16059,    0, 0,   0, 0 ##
19232, 16210,   3848, 15834,    8118, 0,   0, 0 ##
19232, 16210,   10801, 16160,    -8118, 0,   0, 0 ##
13455, 16128,   27664, 16059,    0, 0,   0, 0 ##
-2074, 16141,   10794, 16160,    -8118, 0,   0, 0 ##
-2074, 16141,   3848, 15834,    8118, 0,   0, 0 ##
-2074, 16141,   10794, 16160,    -8118, 0,   0, 0 ##
12793, 16175,   -20260, 16187,    -11481, 0,   0, 0 ##
-2074, 16141,   3848, 15834,    8118, 0,   0, 0 ##
-2079, 16141,   10794, 16160,    -8118, 0,   0, 0 ##
-2079, 16141,   3848, 15834,    8118, 0,   0, 0 ##
12788, 16175,   -20260, 16187,    -11481, 0,   0, 0 ##
13455, 16128,   27664, 16059,    0, 0,   0, 0 ##
-2079, 16141,   3848, 15834,    8118, 0,   0, 0 ##
-2079, 16141,   10794, 16160,    -8118, 0,   0, 0 ##
-15208, 15156,   -24710, 16011,    8118, 0,   0, 0 ##
-15208, 15156,   -17944, 16127,    0, 0,   0, 0 ##
2907, 16127,   -24710, 16011,    8118, 0,   0, 0 ##
2907, 16127,   -5587, 16185,    -8118, 0,   0, 0 ##
2907, 16127,   -17944, 16127,    0, 0,   0, 0 ##
-15208, 15156,   -5587, 16185,    -8118, 0,   0, 0 ##
-15208, 15156,   -24692, 16011,    -8118, 0,   0, 0 ##
-15208, 15156,   -17924, 16127,    0, 0,   0, 0 ##
2907, 16127,   -24692, 16011,    -8118, 0,   0, 0 ##
2907, 16127,   -5578, 16185,    8118, 0,   31360, 665 ##
2907, 16127,   -17924, 16127,    0, 0,   7, 0 ##

The numbers feel all wrong for vertex-positions. They are too similar and far too large.
I wonder if there is a byte or two which should be ignored within each vertex-group? That would offset the data so the SHORTS would pair up with other neighbours.
I'll give it shot, but any observations and options heartily welcome!  :lol:
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: BlitzNCS on 2009-02-03 16:44:01
Don't worry about the Hope-dashing :P i'm very used to it xD

The simplest [!]-File I could find was File: 02450.raw, which looks like it might be a barrel.
It has 84 vertices though, which feels like too much vertices or faces if its supposed to be only a simple barrel.

about this, i tried to recreate this "barrel" and i found out that 84 verticies is actually very reasonable for a barrel
i used the simplest cylinder i could create, and applied the texture.
(http://i182.photobucket.com/albums/x74/beefpwnage/Barrel.jpg)
the one from that actual game, i would guess, would be FAR more advanced than this, so i doubt there'd be LESS verticies.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-02-04 12:15:44
I think i wasn't wrong in the first place, the first 4 shorts really might just be 2-Float UVs!

(http://img201.imageshack.us/img201/986/notbarreljq4.jpg) (http://imageshack.us)

the data goes something like this:
Code: [Select]
read bytes until find tag (3f800000)
go back to beginning of vertex-group bytes and read:
- FLOAT  (i assumed U)
- FLOAT  (i assumed V)
- SHORT
- SHORT  (i assumed Y)
- BYTE
- SHORT  (i assumed X)
- SHORT
- SHORT  (i assumed Z)

Here is a video showing how these vertices look when plotted in space: http://www.mediafire.com/?henmwyytmdu

Apparantly not a barrel, but some kind of a crate? Or maybe my assumed XYZs are all wrong  :-P
But definitly on the right track now because there are obvious cylindrical shapes everywhere.

There only seem to be about 30 vertices visible, so the other 50 or so must be repeats, so maybe these are simple unindexed vertices to plot one after another?
I did try rendering them as triangles and quads, but they give gobbledegook, so maybe a mixture of the two determined by other tags or somthing.

here are the actual values I read in:  (the first ones just a "header" with all zeros)
Code: [Select]
VERTEX: 0:   START: 440, SIZE: 0  -->    SHORT:     BYTE:    SHORT: 
VERTEX: 1:   START: 444, SIZE: 20  -->    FLOAT:  0.498134, 0.272702,   SHORT:  155, 180,    BYTE:  74,   SHORT:  -32, 19071, 224,
VERTEX: 2:   START: 468, SIZE: 20  -->    FLOAT:  0.002758, 0.499466,   SHORT:  131, 239,    BYTE:  39,   SHORT:  211, 0, 0,
VERTEX: 3:   START: 492, SIZE: 20  -->    FLOAT:  0.498134, 0.499466,   SHORT:  131, 17,    BYTE:  39,   SHORT:  -45, 127, 0,
VERTEX: 4:   START: 516, SIZE: 20  -->    FLOAT:  0.498134, 0.272701,   SHORT:  101, 76,    BYTE:  182,   SHORT:  -225, -18817, 31,
VERTEX: 5:   START: 540, SIZE: 20  -->    FLOAT:  0.002758, 0.499465,   SHORT:  125, 17,    BYTE:  217,   SHORT:  44, 0, 0,
VERTEX: 6:   START: 564, SIZE: 20  -->    FLOAT:  0.498134, 0.499465,   SHORT:  125, 239,    BYTE:  217,   SHORT:  -212, 127, 0,
VERTEX: 7:   START: 588, SIZE: 20  -->    FLOAT:  0.691666, -0.001050,   SHORT:  32512, 0,    BYTE:  0,   SHORT:  -256, -9857, 44,
VERTEX: 8:   START: 612, SIZE: 20  -->    FLOAT:  0.821459, 0.106474,   SHORT:  32512, 0,    BYTE:  182,   SHORT:  -225, -18817, 31,
VERTEX: 9:   START: 636, SIZE: 20  -->    FLOAT:  0.691666, 0.733167,   SHORT:  32512, 0,    BYTE:  0,   SHORT:  -256, 10111, 211,
VERTEX: 10:   START: 660, SIZE: 20  -->    FLOAT:  0.691666, 0.733167,   SHORT:  32512, 0,    BYTE:  0,   SHORT:  -256, 10111, 211,
VERTEX: 11:   START: 684, SIZE: 20  -->    FLOAT:  0.821459, 0.106474,   SHORT:  32512, 0,    BYTE:  74,   SHORT:  -32, -18817, 31,
VERTEX: 12:   START: 708, SIZE: 20  -->    FLOAT:  0.691666, -0.001050,   SHORT:  32512, 0,    BYTE:  0,   SHORT:  -256, -9857, 44,
VERTEX: 13:   START: 732, SIZE: 20  -->    FLOAT:  0.684356, -0.001050,   SHORT:  -32512, 0,    BYTE:  0,   SHORT:  0, -9984, 44,
VERTEX: 14:   START: 756, SIZE: 20  -->    FLOAT:  0.554564, 0.106474,   SHORT:  -32512, 0,    BYTE:  74,   SHORT:  224, -18944, 31,
VERTEX: 15:   START: 780, SIZE: 20  -->    FLOAT:  0.684356, 0.733167,   SHORT:  -32512, 0,    BYTE:  0,   SHORT:  0, 9984, 211,
VERTEX: 16:   START: 804, SIZE: 20  -->    FLOAT:  0.684356, 0.733167,   SHORT:  -32512, 0,    BYTE:  0,   SHORT:  0, 9984, 211,
VERTEX: 17:   START: 828, SIZE: 20  -->    FLOAT:  0.554564, 0.106474,   SHORT:  -32512, 0,    BYTE:  182,   SHORT:  31, -18944, 31,
VERTEX: 18:   START: 852, SIZE: 20  -->    FLOAT:  0.684356, -0.001050,   SHORT:  -32512, 0,    BYTE:  0,   SHORT:  0, -9984, 44,
VERTEX: 19:   START: 876, SIZE: 20  -->    FLOAT:  0.002758, 0.995258,   SHORT:  239, 125,    BYTE:  0,   SHORT:  0, -9984, 44,
VERTEX: 20:   START: 900, SIZE: 20  -->    FLOAT:  0.498134, 0.995258,   SHORT:  17, 125,    BYTE:  0,   SHORT:  -256, -9857, 44,
VERTEX: 21:   START: 924, SIZE: 20  -->    FLOAT:  0.498134, 0.726230,   SHORT:  180, 101,    BYTE:  74,   SHORT:  -32, -18817, 31,
VERTEX: 22:   START: 948, SIZE: 20  -->    FLOAT:  0.002758, 0.726230,   SHORT:  155, 76,    BYTE:  74,   SHORT:  224, -18944, 31,
VERTEX: 23:   START: 972, SIZE: 20  -->    FLOAT:  0.002758, 0.995258,   SHORT:  239, 125,    BYTE:  0,   SHORT:  0, -9984, 44,
VERTEX: 24:   START: 996, SIZE: 20  -->    FLOAT:  0.498134, 0.726230,   SHORT:  180, 101,    BYTE:  74,   SHORT:  -32, -18817, 31,
VERTEX: 25:   START: 1020, SIZE: 20  -->    FLOAT:  0.002758, 0.272702,   SHORT:  180, 155,    BYTE:  74,   SHORT:  224, 18944, 224,
VERTEX: 26:   START: 1044, SIZE: 20  -->    FLOAT:  0.498134, 0.272702,   SHORT:  155, 180,    BYTE:  74,   SHORT:  -32, 19071, 224,
VERTEX: 27:   START: 1068, SIZE: 20  -->    FLOAT:  0.498134, 0.003673,   SHORT:  239, 131,    BYTE:  0,   SHORT:  -256, 10111, 211,
VERTEX: 28:   START: 1092, SIZE: 20  -->    FLOAT:  0.002758, 0.003673,   SHORT:  17, 131,    BYTE:  0,   SHORT:  0, 9984, 211,
VERTEX: 29:   START: 1116, SIZE: 20  -->    FLOAT:  0.002758, 0.272702,   SHORT:  180, 155,    BYTE:  74,   SHORT:  224, 18944, 224,
VERTEX: 30:   START: 1140, SIZE: 20  -->    FLOAT:  0.498134, 0.003673,   SHORT:  239, 131,    BYTE:  0,   SHORT:  -256, 10111, 211,
VERTEX: 31:   START: 1164, SIZE: 20  -->    FLOAT:  0.002758, 0.995258,   SHORT:  17, 131,    BYTE:  0,   SHORT:  0, 9984, 211,
VERTEX: 32:   START: 1188, SIZE: 20  -->    FLOAT:  0.498134, 0.995258,   SHORT:  239, 131,    BYTE:  0,   SHORT:  -256, 10111, 211,
VERTEX: 33:   START: 1212, SIZE: 20  -->    FLOAT:  0.498134, 0.726229,   SHORT:  76, 155,    BYTE:  182,   SHORT:  -225, 19071, 224,
VERTEX: 34:   START: 1236, SIZE: 20  -->    FLOAT:  0.002758, 0.726229,   SHORT:  101, 180,    BYTE:  182,   SHORT:  31, 18944, 224,
VERTEX: 35:   START: 1260, SIZE: 20  -->    FLOAT:  0.002758, 0.995258,   SHORT:  17, 131,    BYTE:  0,   SHORT:  0, 9984, 211,
VERTEX: 36:   START: 1284, SIZE: 20  -->    FLOAT:  0.498134, 0.726229,   SHORT:  76, 155,    BYTE:  182,   SHORT:  -225, 19071, 224,
VERTEX: 37:   START: 1308, SIZE: 20  -->    FLOAT:  0.002758, 0.003673,   SHORT:  239, 125,    BYTE:  0,   SHORT:  0, -9984, 44,
VERTEX: 38:   START: 1332, SIZE: 20  -->    FLOAT:  0.002758, 0.272701,   SHORT:  76, 101,    BYTE:  182,   SHORT:  31, -18944, 31,
VERTEX: 39:   START: 1356, SIZE: 20  -->    FLOAT:  0.498134, 0.003673,   SHORT:  17, 125,    BYTE:  0,   SHORT:  -256, -9857, 44,
VERTEX: 40:   START: 1380, SIZE: 20  -->    FLOAT:  0.002758, 0.272701,   SHORT:  76, 101,    BYTE:  182,   SHORT:  31, -18944, 31,
VERTEX: 41:   START: 1404, SIZE: 20  -->    FLOAT:  0.498134, 0.272701,   SHORT:  101, 76,    BYTE:  182,   SHORT:  -225, -18817, 31,
VERTEX: 42:   START: 1428, SIZE: 20  -->    FLOAT:  0.498134, 0.003673,   SHORT:  17, 125,    BYTE:  0,   SHORT:  -256, -9857, 44,
VERTEX: 43:   START: 1452, SIZE: 20  -->    FLOAT:  0.002758, 0.726230,   SHORT:  155, 76,    BYTE:  74,   SHORT:  224, -18944, 31,
VERTEX: 44:   START: 1476, SIZE: 20  -->    FLOAT:  0.498134, 0.499466,   SHORT:  131, 17,    BYTE:  39,   SHORT:  -45, 127, 0,
VERTEX: 45:   START: 1500, SIZE: 20  -->    FLOAT:  0.002758, 0.499466,   SHORT:  131, 239,    BYTE:  39,   SHORT:  211, 0, 0,
VERTEX: 46:   START: 1524, SIZE: 20  -->    FLOAT:  0.002758, 0.726229,   SHORT:  101, 180,    BYTE:  182,   SHORT:  31, 18944, 224,
VERTEX: 47:   START: 1548, SIZE: 20  -->    FLOAT:  0.498134, 0.499465,   SHORT:  125, 239,    BYTE:  217,   SHORT:  -212, 127, 0,
VERTEX: 48:   START: 1572, SIZE: 20  -->    FLOAT:  0.002758, 0.499465,   SHORT:  125, 17,    BYTE:  217,   SHORT:  44, 0, 0,
VERTEX: 49:   START: 1596, SIZE: 20  -->    FLOAT:  0.875220, 0.366059,   SHORT:  32512, 0,    BYTE:  217,   SHORT:  -212, 127, 0,
VERTEX: 50:   START: 1620, SIZE: 20  -->    FLOAT:  0.821459, 0.625644,   SHORT:  32512, 0,    BYTE:  182,   SHORT:  -225, 19071, 224,
VERTEX: 51:   START: 1644, SIZE: 20  -->    FLOAT:  0.821459, 0.106474,   SHORT:  32512, 0,    BYTE:  182,   SHORT:  -225, -18817, 31,
VERTEX: 52:   START: 1668, SIZE: 20  -->    FLOAT:  0.821459, 0.625644,   SHORT:  32512, 0,    BYTE:  182,   SHORT:  -225, 19071, 224,
VERTEX: 53:   START: 1692, SIZE: 20  -->    FLOAT:  0.691666, 0.733167,   SHORT:  32512, 0,    BYTE:  0,   SHORT:  -256, 10111, 211,
VERTEX: 54:   START: 1716, SIZE: 20  -->    FLOAT:  0.821459, 0.106474,   SHORT:  32512, 0,    BYTE:  182,   SHORT:  -225, -18817, 31,
VERTEX: 55:   START: 1740, SIZE: 20  -->    FLOAT:  0.821459, 0.625644,   SHORT:  32512, 0,    BYTE:  74,   SHORT:  -32, 19071, 224,
VERTEX: 56:   START: 1764, SIZE: 20  -->    FLOAT:  0.821459, 0.106474,   SHORT:  32512, 0,    BYTE:  74,   SHORT:  -32, -18817, 31,
VERTEX: 57:   START: 1788, SIZE: 20  -->    FLOAT:  0.691666, 0.733167,   SHORT:  32512, 0,    BYTE:  0,   SHORT:  -256, 10111, 211,
VERTEX: 58:   START: 1812, SIZE: 20  -->    FLOAT:  0.875220, 0.366059,   SHORT:  32512, 0,    BYTE:  39,   SHORT:  -45, 127, 0,
VERTEX: 59:   START: 1836, SIZE: 20  -->    FLOAT:  0.821459, 0.106474,   SHORT:  32512, 0,    BYTE:  74,   SHORT:  -32, -18817, 31,
VERTEX: 60:   START: 1860, SIZE: 20  -->    FLOAT:  0.821459, 0.625644,   SHORT:  32512, 0,    BYTE:  74,   SHORT:  -32, 19071, 224,
VERTEX: 61:   START: 1884, SIZE: 20  -->    FLOAT:  0.500802, 0.366059,   SHORT:  -32512, 0,    BYTE:  39,   SHORT:  211, 0, 0,
VERTEX: 62:   START: 1908, SIZE: 20  -->    FLOAT:  0.554564, 0.625643,   SHORT:  -32512, 0,    BYTE:  74,   SHORT:  224, 18944, 224,
VERTEX: 63:   START: 1932, SIZE: 20  -->    FLOAT:  0.554564, 0.106474,   SHORT:  -32512, 0,    BYTE:  74,   SHORT:  224, -18944, 31,
VERTEX: 64:   START: 1956, SIZE: 20  -->    FLOAT:  0.554564, 0.625643,   SHORT:  -32512, 0,    BYTE:  74,   SHORT:  224, 18944, 224,
VERTEX: 65:   START: 1980, SIZE: 20  -->    FLOAT:  0.684356, 0.733167,   SHORT:  -32512, 0,    BYTE:  0,   SHORT:  0, 9984, 211,
VERTEX: 66:   START: 2004, SIZE: 20  -->    FLOAT:  0.554564, 0.106474,   SHORT:  -32512, 0,    BYTE:  74,   SHORT:  224, -18944, 31,
VERTEX: 67:   START: 2028, SIZE: 20  -->    FLOAT:  0.554564, 0.625643,   SHORT:  -32512, 0,    BYTE:  182,   SHORT:  31, 18944, 224,
VERTEX: 68:   START: 2052, SIZE: 20  -->    FLOAT:  0.554564, 0.106474,   SHORT:  -32512, 0,    BYTE:  182,   SHORT:  31, -18944, 31,
VERTEX: 69:   START: 2076, SIZE: 20  -->    FLOAT:  0.684356, 0.733167,   SHORT:  -32512, 0,    BYTE:  0,   SHORT:  0, 9984, 211,
VERTEX: 70:   START: 2100, SIZE: 20  -->    FLOAT:  0.500802, 0.366059,   SHORT:  -32512, 0,    BYTE:  217,   SHORT:  44, 0, 0,
VERTEX: 71:   START: 2124, SIZE: 20  -->    FLOAT:  0.554564, 0.106474,   SHORT:  -32512, 0,    BYTE:  182,   SHORT:  31, -18944, 31,
VERTEX: 72:   START: 2148, SIZE: 20  -->    FLOAT:  0.554564, 0.625643,   SHORT:  -32512, 0,    BYTE:  182,   SHORT:  31, 18944, 224,
VERTEX: 73:   START: 2172, SIZE: 20  -->    FLOAT:  0.002758, 0.272701,   SHORT:  76, 101,    BYTE:  182,   SHORT:  31, -18944, 31,
VERTEX: 74:   START: 2196, SIZE: 20  -->    FLOAT:  0.002758, 0.499465,   SHORT:  125, 17,    BYTE:  217,   SHORT:  44, 0, 0,
VERTEX: 75:   START: 2220, SIZE: 20  -->    FLOAT:  0.498134, 0.272701,   SHORT:  101, 76,    BYTE:  182,   SHORT:  -225, -18817, 31,
VERTEX: 76:   START: 2244, SIZE: 20  -->    FLOAT:  0.498134, 0.726229,   SHORT:  76, 155,    BYTE:  182,   SHORT:  -225, 19071, 224,
VERTEX: 77:   START: 2268, SIZE: 20  -->    FLOAT:  0.498134, 0.499465,   SHORT:  125, 239,    BYTE:  217,   SHORT:  -212, 127, 0,
VERTEX: 78:   START: 2292, SIZE: 20  -->    FLOAT:  0.002758, 0.726229,   SHORT:  101, 180,    BYTE:  182,   SHORT:  31, 18944, 224,
VERTEX: 79:   START: 2316, SIZE: 20  -->    FLOAT:  0.002758, 0.272702,   SHORT:  180, 155,    BYTE:  74,   SHORT:  224, 18944, 224,
VERTEX: 80:   START: 2340, SIZE: 20  -->    FLOAT:  0.002758, 0.499466,   SHORT:  131, 239,    BYTE:  39,   SHORT:  211, 0, 0,
VERTEX: 81:   START: 2364, SIZE: 20  -->    FLOAT:  0.498134, 0.272702,   SHORT:  155, 180,    BYTE:  74,   SHORT:  -32, 19071, 224,
VERTEX: 82:   START: 2388, SIZE: 20  -->    FLOAT:  0.498134, 0.726230,   SHORT:  180, 101,    BYTE:  74,   SHORT:  -32, -18817, 31,
VERTEX: 83:   START: 2412, SIZE: 20  -->    FLOAT:  0.498134, 0.499466,   SHORT:  131, 17,    BYTE:  39,   SHORT:  -45, 127, 0,

If anyone can spot a better possibility for the XYZ values (any one of those shorts could be X, Y or Z) then please please do let me know!
 :-D
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-02-04 19:14:47
Tada!

(http://img183.imageshack.us/img183/7905/canba9.th.jpg) (http://img183.imageshack.us/my.php?image=canba9.jpg)  (http://img183.imageshack.us/img183/8350/barrelos0.th.jpg) (http://img183.imageshack.us/my.php?image=barrelos0.jpg)

Simple bone-less [!] files now understood enough to render!
I still haven't figured out where Characters and other larger models store their XYZs, so they show up as junk with the loader.

After hours of banging my head against the wall and lots of different XYZ combinations, I decided to try the obvious thing and set the last 3 shorts as XYZ and everything clicked together!  :-D

The vertex format so far is:
Code: [Select]
(after first vertex-tag)
FLOAT, FLOAT, 4-bytes (unknown), SHORT, SHORT, SHORT, 3f800000  =  U, V, (????), X, Y, Z, vertex-tag
rinse and repeat

Render as a simple triangle-(primitive)-list, no headaches over face indices or quads or anything else

Characters soon!
 :lol:
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: ultima espio on 2009-02-04 19:18:55
 :-o :-o :-o

Wow! Cant wait for characters lol.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-02-04 19:28:23
I know espio, I can't wait either  :-P

Here is what Moogle-san looks like right now:
(http://img240.imageshack.us/img240/2390/moogletf8.th.jpg) (http://img240.imageshack.us/my.php?image=moogletf8.jpg)

i know I'm close, but there is something horrible going on somewhere.

Almost there  :-D
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: BlitzNCS on 2009-02-04 19:46:46
*cries*

Wow. i REALLY am amazed at this :D

great job koral :D
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-02-04 21:30:29
a collage to tease you some more:
(http://img3.imageshack.us/img3/4531/collagecy3.th.jpg) (http://img3.imageshack.us/my.php?image=collagecy3.jpg)

Looking through the other [!] files I noticed a lot of them are almost fully intact, just like the barrel and that can (box) thing. Most machine monsters and machine bosses, small objects etc.

Characters are still iffy.
I managed to get rid of a lot of random vertices and things are looking better now, although still far from perfect:
(http://img502.imageshack.us/img502/773/moogle2ea1.th.jpg) (http://img502.imageshack.us/my.php?image=moogle2ea1.jpg)

more updates as they come  :wink:
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Jonnylossus on 2009-02-04 22:13:16
Thats really great.^^
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Silver Wings on 2009-02-05 06:30:56
Oh my godness!!!
I dont know how to say :touched:
Haizz, be ashame of being helpless to you guys
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-02-05 19:25:54
its cool, you don't need to feel ashamed :)

I haven't made much progress with the characters, but I have noticed a few things.

There are two types of Vertices:

- those grouped as 24-byte chunks (including tag)
- those not grouped as 24-byte chunks (ranging from 28 to 300 bytes)

What you see in the above screenshots are only those 24-byte vertices, so the only thing to figure out now is how those non-24-byte vertices are stored.

Now inside those non-24-byte-vertices there are also a handful of other "tags" of somesort, in varying order.

The ones I've spotted so far:
Code: [Select]
- 9a 99 99 3e  [short: 3e99999a]
- 33 33 33 3f  [short: 3f333333]
- 66 66 66 3f  [short: 3f666666]
- cd cc cc 3d  [short: 3dcccccd]

I have no idea what they are, possibly some kind of PSP op-codes?

I know there aren't any Normals present because in FF7-CC they haven't used any real lighting (for improved rendering speeds I would guess), but what other kind of additional data could possibly be squeezed within vertex-triangle-lists?
 :|
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: ignitz on 2009-02-06 07:29:48
I found a thing maybe be important in RAM dumper os PSP
In MDL and others files and other things have a memory pointers, I don't know will help with info.

MDL
4D444C00 01000A00 0011E098 01000000 01000000 B03E1609
01000000 C83E1609 00000000 FFFFFFFF 27000000 00060A00
00000000 FFFFFFFF 31000000 103F1609


Like This
B03E1609 (09163EB0)
C83E1609 (09163EC8)
103F1609 (09163F10)

This is a RAM "alocker" Have a Lots of this.
With a RAM PSP have 24MB then dump the RAM and you will be this way

B03E1609 then 0x09163EB0 then

0x09163EB0 - 0x8800000 = 0x963EB0 (this will be offset real on the RAM in HexEditor)

With this info you can info you can found a way to dissasembly the files i thing with this.

---------------------------------------------------------------------------------
MDL
4D444C00 01000A00 0011E098 01000000
01000000 B03E1609 // This point Lead to A

01000000 C83E1609  //This lead to B

A
00000000 FFFFFFFF 27000000 00060A00 00000000 FFFFFFFF


B
31000000 103F1609



PS: Maybe I will be wrong, I Thing this system will be work only Spanish Dump and found in USA GAme too

 0x8800000  this will be a Initial RAM for PSP
----------------------------------------------------------------------------------------

In files don't have any pointer.
In RAM this pointers tranforms in this pointers RAM

http://www.megaupload.com/pt/?d=GMESQM3F

Link to the DUMP SPANISH and DUMP USA "(2)"
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-02-07 21:32:23
um, I'm not sure what you're talking about ignitz  :oops:

usually there are a bunch of offsets in the header (first 0x80 bytes or so) pointing to different places in the file. I've been using a hex-editor to see if which number point to another section in the file, and which are just plain numbers.

But I don't know much about MDL files though, so I will have a look at that ram-dump to see what its about  :wink:

I would be interested to hear how you managed to dump the ram in the first place, it might help to discover what gets executed when the game boots up, because there might be something todo with the debug-menu in there!



Nothing new with the [!] files sadly, I'm still struggling to understand the non-24-byte vertex chunks  :-(

I gave them names though  :-P
- Type-1 Vertex data = 24-bytes
- Type-2 Vertex data = more than 24-bytes

Type-2 vertex-chunks actually contain multiple Type-1 vertices inside them, but they seem to be randomly located within the chunks.
The "extended" tags (0x333333f, 0xcdcccc3d, etc) look like they're padding, or mark the beginning and end of padding-bytes in some way. I haven't been able to spot an obvious pattern though, because they get used diffferently in different files.

so close yet so far  :|
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: spidergaby on 2009-02-08 09:15:54
Yeah good job, continue !!!!  :wink:
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: spidergaby on 2009-02-12 16:38:57
Any news ???  :cry:
PLease continue  :cry: :cry:
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: spidergaby on 2009-02-13 17:23:39
Hey why not use 3Dx ripper ??
Have you loojed at the Yazor models viewer ?
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: ultima espio on 2009-02-13 17:28:32
Will you pack it in triple posting?? It's really really annoying to us watching the topic to find your pointless posts.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-02-14 09:35:05
sorry everyone, life finally caught up with me again and I haven't had a chance   :-(

the project will be continuing until I can get characters rendered properly.
Everything else like map-meshes, animations, monsters and all the other stuff, I don't know. Maybe I will, maybe I wont.

I should get some time over the weekend to try solve this stupid vertex stuff, fingers crossed
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: BlitzNCS on 2009-02-14 11:09:41
I guess I'll try and help out too :D
Thanks for your commitment to this koral!
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: spidergaby on 2009-02-15 14:49:14
sorry for the "triple post"  :|
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: NameSpoofer on 2009-02-16 04:11:41
Hey Koral

Good to see you doing a great job

Its almost there. Thumbs up mate :wink:
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-02-16 19:24:55
A little update of what I did over the weekend, and what I'm trying now

Things are almost falling into place:
(http://img261.imageshack.us/img261/2528/moogle3cw0.jpg) (http://imageshack.us)

Moogle-san's head now has all the correct vertices, but hundreds of new rogue vertices also seemed to get detected.
I have been working on a special sort of "data parser" to handle the Type-2 vertices, because there really isn't any logical way to find the Type-1 vertices without reading through and ignoring all the random padding "tags"

For example, here is the breakdown of the first Type-2 vertex in file 02373 [moogle]:
Code: [Select]
  vertex 66, start: 0xd30,  end 0xdd0,   size: 0x100

0x00 00 00 80 3f *** TYPE-1 VERTEX tag

0x04   TYPE-1 VERTEX

0x18 01 00 0e 00 ? 24-byte padding start tag [2nd and 4th bytes always zero]
0x1c     20-byte padding

0x2c 54 00 00 00 ? 4-byte padding [2nd, 3rd and 4th bytes always zero]
0x30 66 66 66 3f   4-byte padding
0x34 cd cc cc 3d   4-byte padding

0x38   TYPE-1 VERTEX

0x4C 33 33 33 3f   4-byte padding
0x50 9a 99 99 3e   4-byte padding

0x54   TYPE-1 VERTEX

0x68 66 66 66 3f   4-byte padding
0x6c cd cc cc 3d   4-byte padding

0x70   TYPE-1 VERTEX

0x84 33 33 33 3f   4-byte padding
0x88 9a 99 99 3e   4-byte padding

0x8c   TYPE-1 VERTEX

There were 5 Type-1 vertices packed into this Type-2 chunk, and the number varies a lot.
The most difficult thing is trying to determine a UV-FLOAT (0-1) from either of 3 tag (padding) possibilities:

- 00 00 00 00 = pops up in a lot of other files, usually following other padding-tags
- 1st byte non-zero, 3rd byte non-zero, 2nd and 4th bytes zeros = 24-byte padding
- 1st byte non-zero, rest zeros = 4-byte padding

But its all most definitly based on 4-byte (2-SHORT, 1-FLOAT) reads, although not neccessarily after 20-byte chunks (size of Type-1 vertices).
Its a long haul and a tough cookie, but I wont give up until its over  :wink:
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: spidergaby on 2009-02-21 17:32:50
It looks ....strange  :-o
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: BlitzNCS on 2009-02-21 22:03:34
Indeed it is strange...
...Koral, how come the latest screenshot of the moogle appears to have a Completely messed up Body below the head, whereas the screenshot before that looks almost normal in terms of the body? In fixing the Head, have you somehow ruined the body? That's really quite weird.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-02-22 16:39:58
sorry peeps, i'm still struggling for time

it looks ugly because all those random vertices are all from the Type-2 vertex chunks. I have not been able to find a way to cleanly get the correct vertices out of those chunks, because there are usually more fake vertices than real ones. Those fake vertices might be padding or maybe something else. But there just isn't a clear-cut way to tell which is which.

I didn't want to have to do this, but have a look at some of the characters as they look now, demonstrating what I mean:

(http://img516.imageshack.us/img516/4938/mooglesan2.th.jpg) (http://img516.imageshack.us/my.php?image=mooglesan2.jpg)

Zack:
(http://img516.imageshack.us/img516/4334/zackugly.th.jpg) (http://img516.imageshack.us/my.php?image=zackugly.jpg)

Reno:
(http://img18.imageshack.us/img18/8785/rugly.th.jpg) (http://img18.imageshack.us/my.php?image=rugly.jpg)


there were hunderds of fake-vertices, and I have spent the past couple of weeks trying to decypher the "rules", to be able tell which vertices are fake and which aren't.
Some of those fake-vertices are still there, and in other cases, real vertices get mistaken for fake ones, causing the polygon indexing to mess up.

I have made a number of minor discoveries, but I wont spam this topic with useless bits of posts until I make a breakthrough.

I have also been given acess to the Quimm-Wiki, I just need to find the time to get on with it.
Sorry for not getting much done this month :oops:
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: ultima espio on 2009-02-22 17:01:36
It's ok Koral, at least your getting somewhere with it. They dont look that bad right there, just missing a few bits.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Satoh on 2009-02-22 18:19:56
Koral, you haven't found any bone data yet have you? It's starting to look to me, that some of those fake verts may be bone joins and endpoints... namely the ones at the ends of Reno's hands there, they are in a conceivable place to be the ends of the wrist bones...

Another thing to note... Is it just me, or does that large cluster of crap under Zack not have a similar shape to the buster sword?

On a seconds inspection, the rod Reno uses is beneath him, so that must be the sword. Perhaps some of the fake verts are related to secondary textures required by the engine... It looks like the buster sword is using Zack's texture, rather than a new one.

I know you may have already noted much of what I've said, but in case you somehow missed it, I hope my ideas have helped somehow... On the other hand I could be completely wrong.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: BlitzNCS on 2009-02-22 19:10:19
Wow, you really seem to know what you're talking about, X-Dina! it all seems to make quite a lot of sense.
Koral, that's a magnificent job you're doing here, just look at what you've accomplished! A few weeks ago, i would have thought the textures would just be a dream, let alone the actual models! you're so close, so keep going!
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Satoh on 2009-02-23 12:14:26
Wow, you really seem to know what you're talking about, X-Dina! it all seems to make quite a lot of sense.
Koral, that's a magnificent job you're doing here, just look at what you've accomplished! A few weeks ago, i would have thought the textures would just be a dream, let alone the actual models! you're so close, so keep going!

I don't know as much as you think... I'm just making a lot of observations....
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: MrAdults on 2009-02-24 09:54:38
This thread just randomly caught my eye - nice work, koral. I thought I'd mention that the data you are looking at is bone-relative, meaning the vertex positions need to be cumulatively multiplied by bone matrices to end up with the correct base pose position. There should also be some bone index data and weight values in there with the X-Y-Z's, as per common PS2/PSP vertex blending block arrays. Though that isn't necessarily the case, if there are bone indexes in there they should be pretty easy to spot.

Hope that helps, good luck.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: spidergaby on 2009-02-26 12:48:24
Anyway, nice work koral ;) continu !
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: ignitz on 2009-02-28 01:41:46
if I have knowleged of grafics, but I don't, then I don't have to help, but Koral make a excelent work and I hope to mek progress XD.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-02-28 10:17:00
I really should check here more often  :-P

ignitz, don't worry about it, I think I have almost solved this  :-D
your help with the scripts was useful, so if you want you could try looking into other things, like Monster data maybe?

X-Dina and MrAdults thanks for your observations, but I doubt that those vertices are bone-positions, or in pre-bone-heirarchy-multiplied positions because all the vertices are actually present and parsable.
But I do agree that some kind of skinning-data might also be mixed into those Type-2 vertices.

This was how Tifa looked when only Type-1 vertices are parsed:
(http://img13.imageshack.us/img13/9804/coltifa.th.jpg) (http://img13.imageshack.us/my.php?image=coltifa.jpg)

And like this when I attemtpted to parse Type-1 vertices out of the Type-2 chunks:
(http://img3.imageshack.us/img3/2160/tifnow.th.jpg) (http://img3.imageshack.us/my.php?image=tifnow.jpg)

As you can see, all the vertices are there in correct positions, and it's only the polygon indexing which is messed up. The vertices are actually listed as a huge Triangle list within these [!]-files, listed one after the other in approximatly 24-byte chunks.

I have discovered that there seems to be a sort of 0x1C-byte "header" before the beginning of Type-1 or Type-2 vertex groupings, with a short at 0x14 giving a count of how many Type-1 vertices follow. This doesn't seem to work with Type-2 vertex grouping "headers" though, and I am guessing that those are probably multiple "headers" listed one after another until a Type-1 vertex comes along. Or maybe skin data with bone weights.

Or something which isn't imporant at any rate. :lol:
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Satoh on 2009-03-01 03:12:05
Well, I'm not good at reading headers, and as such I won't even attempt it just now...

But I do have an idea about them...

IF the header is structured like this...

Code: [Select]
File sections
       vertex pool size
              type one vert amount
                            actual verts


then it could be that the type one verts are counted with the type one vert amount, and everything in the vertex pool AFTER the end of the type one's end offset is a type 2.

EX:
Code: [Select]
Vertex type one amount = offset 0x01A


Vertex pool starts at 0x010 and ends at 0x01F



0x010 This is a Type 1
0x011 Type 1
0x012 Type 1
0x013 Type 1
0x014 Type 1
0x015 Type 1
0x016 Type 1
0x017 Type 1
0x018 Type 1
0x019 Type 1
0x01A Type 1
0x01B This is after the specified size... everything after this is Type 2, even though Type 2's size isn't listed
0x01C Type 2
0x01D Type 2
0x01E Type 2
0x01F Type 2
0x020 This is after the end of the vertex pool... Everything from here on is something else...


This is all speculation...
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: MrAdults on 2009-03-01 14:17:04
Pre-multiplied positions would actually be parsable and appear just as your mog screenshot depicted. In the last model you show, I believe the case might be that it happens to have relatively few bone weights, so most vertices are already in the correct position. However, I still notice some verts in that shot appear out of place, like the one next to the left leg.

It seems likely that type-2 vertices are just vertices with an implicit bone weight reference. The problem is that even though the 1-weight vertices happen to be in triangle list order, that is not the "proper" way to draw them, as there is probably an index key which reference both 1-weight and multi-weight vertices according to face. Again though, totally speculating, as I have no actual data to reference.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-03-03 21:13:09
I guess you guys have a point :oops:
What I should do is take a couple of these files and upload them for you guys to view at your leisure, I am sure I may be missing the obvious.
"1-weight vertices" does sound like a more accurate description of TYPE-1 vertices, but I can't seem to decipher any possible indices anywhere obvious.

Right now, my spare time is going elsewhere.
I have started to Wikify all my findings into Quimms wiki: http://wiki.qhimm.com/FF7:CC
It is still far from complete, but I should manage to get it all filled up in the next few days.

Huge thanks to The SaiNt for giving me access and getting me started!
 :-D
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-03-04 21:22:42
Sorry for the double-post, but whilst I was updating the Wiki I noticed something I should have noticed a long long time ago:

click http://wiki.qhimm.com/FF7:CC (http://wiki.qhimm.com/FF7:CC#Miscelenous Findings)
then goto the "Miscelenous Findings" section


Quote
<TEXT>
 Materia has leveled up!
<END>

<TEXT>
 Nothing more to do, kupo!
<END>

in all ATEL files this script is present, and because the Moogle text is related so strongly with the Debug-Menu data in file 00000.RAW, the first line about the Materia-Levelling-Up can be assumed to also be linked to that.

In other words: Maybe there is a "hidden" Materia to use the Debug-Menu!

All the Materia-Data is present in files 08628 - 08690, so all we have to do now is put two-and-two together!
 :-D
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: MrAdults on 2009-03-05 15:07:54
Cool, I'd be happy to have a look, if you get the chance to put some binary up somewhere. My Crisis Core disc is currently at a friend's house, and I don't have an early-firmware PSP to assist in ripping the contents myself. :)
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-03-05 17:42:49
Here are a couple of the !-Model files that you or anyone else could look at: http://www.mediafire.com/?yiyjj22nm0d

02373.RAW is Moogle-san (representing a simple character sample) and 02190.RAW is Reno (a more complex character).

The actual vertex data begins after a line containing nothing but "wwwwwwww" (or "777777777" hex)
The offset of the Texture-data is located at 0x1C from the beginning of the file, so the size of the entire block can be determined pretty easily.

Best of luck!  :-D
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-03-05 19:12:15
Apologies for the double-post (yet again  :oops:) but I would like to share some new interesting findings.

I quickly ran through the [MAGIC] files and they all have ASCII names (or some other kind of visual tag) at the end of the files, describing what they are.

Here is a list of all the names exactly as I read them (starting from file 08628):
Code: [Select]
FIRE
FIRA
FIGA
BRIZD
BRIZRA
BRIZGA
THD
GRABIDE
ANKOKU
ULTIMA
?EMPTY?
?EMPTY?
?EMPTY?
REJNE
DREIN
GRABIDE
POISON
QUAKE
STOP
BARRIER
MABARRIER
WALL
?EMPTY?
LEAVES_WUTAI SMOKE_FALL, SMOKE_BOSS_FALL
LEAVES_SPIN, SPIN_SPIN, SPIN_KISEKI
LEAVES_HIT_TEST, HIT_FLASH, SWORD_KISEKI
?EMPTY?
?EMPTY?
HIT
HIT_01
?EMPTY?
LEAVES_GOD_HAND, STEAL_HAND, STEAL_ARM
RECOVERY, SWORD_KISEKI
JUMP_JUMP, LAND_HIT, KISEKI_SHINRA , KISEKI_BASTER, KISEKI_PRASOL
RAKURAI
FIRE_HIT, SWORD_FIRE
BRI_HIT, SWORD_BRI
THD_HIT, SWORD_THD
SHOOT_THROUGH_HIT
SWORD_KISEKI
ZENI_COIN, ZENI_HIT, ZENI_PITCH
LEAVES_STEAL, STEAL_HAND
SP_CHARGE
LEAVES_SAILESS, SAIRENT2, SAIRENT4
HIT01_DEATH
ASPIL, ASPIL_HIT, ASPIL_SIPPO
QUAKE
?EMPTY?
HIT_RECOVERY
HIT01, SEORD_KISEKI
EXPLOSION_HIT, KISEKI_SHINRA
LEAVES_TEKKEN_PUNCH
LEAVES_FLARE, FLARE_FILTER, FLARE_HIT
?EMPTY?
THDGA
LEAVES_GOBURIN_PUNCH
?EMPTY?
?EMPTY?
?EMPTY?
DESP

The "?EMPTY?" ones are just my own tags, because those are 1KB files which do not contain any ASCII text.
The comma seperated ones represent multiple ASCII texts following one another in the same file.

But everything else is exactly from the files, spelling mistakes included.

My observataions:


These can probably be called "Zack's Battle Attacks and Actions" to be more precise, encompassing Materia as well but by no means stopping there.
Can anyone spot anything else?
 :-D
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: MrAdults on 2009-03-06 04:48:16
Thanks for putting those files up! I got up and running with them tonight and have textures working and have found pointers to some vertex data (with some extra vertex info, will document that once I have things working a bit more solid). I'm now working on trying to find some kind of index or skeletal data, and will let you know if I have any success.

BTW, you accidentally put up the ShinRa storage box model instead of Mog. :) However, in working with that model, I noticed the texture is 128x128, but it seems I need to treat it as though it has 2 interlace rows (just like the Reno texture which is 256x256) to decode it properly. Is this a specific thing to this one texture in this chunk, or is the "width/128" rule not reliable/correct?
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: ignitz on 2009-03-06 07:57:56
Apologies for the double-post (yet again  :oops:) but I would like to share some new interesting findings.

I quickly ran through the [MAGIC] files and they all have ASCII names (or some other kind of visual tag) at the end of the files, describing what they are.

Here is a list of all the names exactly as I read them (starting from file 08628):
Code: [Select]
FIRE
FIRA
FIGA
BRIZD
BRIZRA
BRIZGA
THD
GRABIDE
ANKOKU
ULTIMA
?EMPTY?
?EMPTY?
?EMPTY?
REJNE
DREIN
GRABIDE
POISON
QUAKE
STOP
BARRIER
MABARRIER
WALL
?EMPTY?
LEAVES_WUTAI SMOKE_FALL, SMOKE_BOSS_FALL
LEAVES_SPIN, SPIN_SPIN, SPIN_KISEKI
LEAVES_HIT_TEST, HIT_FLASH, SWORD_KISEKI
?EMPTY?
?EMPTY?
HIT
HIT_01
?EMPTY?
LEAVES_GOD_HAND, STEAL_HAND, STEAL_ARM
RECOVERY, SWORD_KISEKI
JUMP_JUMP, LAND_HIT, KISEKI_SHINRA , KISEKI_BASTER, KISEKI_PRASOL
RAKURAI
FIRE_HIT, SWORD_FIRE
BRI_HIT, SWORD_BRI
THD_HIT, SWORD_THD
SHOOT_THROUGH_HIT
SWORD_KISEKI
ZENI_COIN, ZENI_HIT, ZENI_PITCH
LEAVES_STEAL, STEAL_HAND
SP_CHARGE
LEAVES_SAILESS, SAIRENT2, SAIRENT4
HIT01_DEATH
ASPIL, ASPIL_HIT, ASPIL_SIPPO
QUAKE
?EMPTY?
HIT_RECOVERY
HIT01, SEORD_KISEKI
EXPLOSION_HIT, KISEKI_SHINRA
LEAVES_TEKKEN_PUNCH
LEAVES_FLARE, FLARE_FILTER, FLARE_HIT
?EMPTY?
THDGA
LEAVES_GOBURIN_PUNCH
?EMPTY?
?EMPTY?
?EMPTY?
DESP

The "?EMPTY?" ones are just my own tags, because those are 1KB files which do not contain any ASCII text.
The comma seperated ones represent multiple ASCII texts following one another in the same file.

But everything else is exactly from the files, spelling mistakes included.

My observataions:

  • "R"'s are used instead of "L"s, as per Japanese fashion
  • Japanese words like "KISEKI" are mixed in too
  • References to "SHINRA" and "BASTER" and "PRASOL" refer to the 3 swords which Zack has during the game, namely the Shinra standard, Buster Sword and the parasol (used when on vacation at Costa De Sol)
  • The spelling mistakes suggest that these ASCII characters were typed in very quickly without much care, so they probably were nothing more than aides to the developers
  • Most of them seem to be obvious Materia magic, but others (like the "HIT" ones) are standard attacks used by Zack
  • "DESP" is probably DMW attack activation
  • "ASPIL" is probably items being used
  • There are TWO "GRABIDE" files, kindof strange, no?
  • "RAKURAI" and "ANKOKU" seem just as mysterious
  • The one called "LEAVES_WUTAI" is quite interesting too
  • "LEAVES_TEKKEN_PUNCH" :-P

These can probably be called "Zack's Battle Attacks and Actions" to be more precise, encompassing Materia as well but by no means stopping there.
Can anyone spot anything else?
 :-D

This is a Debug DATA, it's normal this unique "lenguage" that some the programmer only "understand"   :evil:
The wrong translate leave to this
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: MrAdults on 2009-03-06 18:37:29
I'm still perplaxed by the lack of face indices - I've tried all kinds of bruteforce searches through the file for every type of index I could think of. The most promising thing I found between both files you put up were from a DWORD pointer at 0x48, it leads to an array of data that looks very much like triangle indices. However, the size and stride does not appear correct, so if it is index data, there is probably another key somewhere else into the file pointing into it. I have not tried getting offsets to the specific "faces" and scanning the file for them yet.

I have to get back to a contract project for a couple days here, so I won't have time to look further until I'm done. In the mean time, good luck! :)
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-03-07 09:22:55
Woops, sorry for the missing Mog!  :oops:
I have uploaded it here now, along with the other two as well: http://www.mediafire.com/?yzzmoiandee

But thankyou so much for trying to help!  :-D
The file-data is really quite unusual, and as inefficient as un-indexed triangle-lists are, it seems to be the method they chose to store the data, possibly just directly dumping the contents into RAM for the GPU (although I am not so sure about the internal functioning of PSP's myself).

I haven't fully understood the texture-interlacing, but the rule of dividing it by 128 seems to work majority of the time. Zande had pointed out before that there might be different interlacing modes determined by a value in the header, so it is worth keeping in mind that my interlacing rules are temporary.

The skeletal data probably does follow on after the vertices, in [unknown-chunk-3] and [unknown-chunk-4], because the ShinRa box and the barrel dont contain any data at those points.
If I had to choose, I would say that any face-indexing is in [Unknown-chunk-1], before all the vertices.
But the values are mostly zeros, so maybe they are just initial bone rotation values?

Just do what you have to do, we are not on any strict kind of deadlines here!  :-P
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: ultima espio on 2009-03-07 11:54:42
Just do what you have to do, we are not on any strict kind of deadlines here!  :-P

Yeah you are :-P :-P Only joking lol
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Satoh on 2009-03-07 16:14:58
Apologies for the double-post (yet again  :oops:) but I would like to share some new interesting findings.

I quickly ran through the [MAGIC] files and they all have ASCII names (or some other kind of visual tag) at the end of the files, describing what they are.

Here is a list of all the names exactly as I read them (starting from file 08628):
Code: [Select]
FIRE
FIRA
FIGA
BRIZD
BRIZRA
BRIZGA
THD
GRABIDE
ANKOKU
ULTIMA
?EMPTY?
?EMPTY?
?EMPTY?
REJNE
DREIN
GRABIDE
POISON
QUAKE
STOP
BARRIER
MABARRIER
WALL
?EMPTY?
LEAVES_WUTAI SMOKE_FALL, SMOKE_BOSS_FALL
LEAVES_SPIN, SPIN_SPIN, SPIN_KISEKI
LEAVES_HIT_TEST, HIT_FLASH, SWORD_KISEKI
?EMPTY?
?EMPTY?
HIT
HIT_01
?EMPTY?
LEAVES_GOD_HAND, STEAL_HAND, STEAL_ARM
RECOVERY, SWORD_KISEKI
JUMP_JUMP, LAND_HIT, KISEKI_SHINRA , KISEKI_BASTER, KISEKI_PRASOL
RAKURAI
FIRE_HIT, SWORD_FIRE
BRI_HIT, SWORD_BRI
THD_HIT, SWORD_THD
SHOOT_THROUGH_HIT
SWORD_KISEKI
ZENI_COIN, ZENI_HIT, ZENI_PITCH
LEAVES_STEAL, STEAL_HAND
SP_CHARGE
LEAVES_SAILESS, SAIRENT2, SAIRENT4
HIT01_DEATH
ASPIL, ASPIL_HIT, ASPIL_SIPPO
QUAKE
?EMPTY?
HIT_RECOVERY
HIT01, SEORD_KISEKI
EXPLOSION_HIT, KISEKI_SHINRA
LEAVES_TEKKEN_PUNCH
LEAVES_FLARE, FLARE_FILTER, FLARE_HIT
?EMPTY?
THDGA
LEAVES_GOBURIN_PUNCH
?EMPTY?
?EMPTY?
?EMPTY?
DESP

The "?EMPTY?" ones are just my own tags, because those are 1KB files which do not contain any ASCII text.
The comma seperated ones represent multiple ASCII texts following one another in the same file.

But everything else is exactly from the files, spelling mistakes included.

My observataions:

  • "R"'s are used instead of "L"s, as per Japanese fashion
  • Japanese words like "KISEKI" are mixed in too
  • References to "SHINRA" and "BASTER" and "PRASOL" refer to the 3 swords which Zack has during the game, namely the Shinra standard, Buster Sword and the parasol (used when on vacation at Costa De Sol)
  • The spelling mistakes suggest that these ASCII characters were typed in very quickly without much care, so they probably were nothing more than aides to the developers
  • Most of them seem to be obvious Materia magic, but others (like the "HIT" ones) are standard attacks used by Zack
  • "DESP" is probably DMW attack activation
  • "ASPIL" is probably items being used
  • There are TWO "GRABIDE" files, kindof strange, no?
  • "RAKURAI" and "ANKOKU" seem just as mysterious
  • The one called "LEAVES_WUTAI" is quite interesting too
  • "LEAVES_TEKKEN_PUNCH" :-P

These can probably be called "Zack's Battle Attacks and Actions" to be more precise, encompassing Materia as well but by no means stopping there.
Can anyone spot anything else?
 :-D

Rather than treating this as materia with other attacks thrown in, I would suggest that this is all of the battle commands... just some of them happen to be item based like materia and item usage. While this doesn't change anything, it gives a little more insight into the programmers' mindset.

an interesting note about the game that may be useful... or not, is that no matter when you do it, the fight with Minerva ALWAYS uses the Buster Sword. (This may point in some way to how scripted events work in the game at a later point.)

I finally got my custom firmware now, so at any point if you all figure out where the 'DEBUG materia' is if it exists, and you can give me an ISO offset for the memory address materia is in... I can test it out with CWCheat.

I haven't looked into the model data just yet, but I'll post anything that piques my interest once I do.

AH! and about the type 1 and type 2's MrAdults mentioned they might be 1 weight verts. This would mean that the other verts, type 2, are weighted to multiple bones. If that is the case, then we would need to figure out how the weight is displayed...

Or rather, another way to look at it... the type 1 and type 2 verts are exactly the same but the bone data for the type 2's continues after the 1st bone index.

This means, we simply need to figure out how to determine the difference between vertex data and bone relations. I imagine the bone indeces wouldn't extend past 4... and more likely would only hold 2 bones.

Now, there are 2-3 things bone indeces would need...

Vertex in question's XYZ orientation + texture UV + :
1. how many bones are weighted...
2. bone's name (probably just a number)...
3. Bone's actual amount of influence...

here is how I would set it up if I were making the format.

Code: [Select]

x,y,z,u,v,#bones,bone1,weight1,bone2,weight2

20 32 12 20-42 02 10 32
11 32

This means vertex 32, 50, 18
has a UV of 32, 66
and has 2 bones
Bone 1 is bone number 16, and has a 50% weight.
Bone 2 is bone number 17, and has a 50% weight.

(I used a 100% scale on the bone weight, because a decimal number would be harder to program. the scale goes from 00(0) to 64(100) )


This is only my thoughts on how it might be worded for efficiency. So, looking at that, the type 2 verts might be as long as 10 offsets if they only have 2 bones, and up to 14 if they had as many as 4 bones weighted.(this is unlikely, but possible...)

Type 1 verts mightn't need the bone amount mini header, and would be implicitly converted.

Still, this is all speculation- Educated speculation, but speculation nonetheless.


In short... We need to decipher the bone data.

EDIT:
Ok, Upon examining I find that the verts are separated by a series of 00 chunks... between 00 00 00 and 00 00 00 00 00 00 long... what are these for? I could speculate that the length of this padded 0 data is used in determining the type of vertex it is. I notice the very first vert begins with a 4 byte long 00 00 00 00.
(In Reno)

So, my guess is that the verts begin with the padded 0's to determine their type, and then use actual coords to determine their location. I could be wrong, but that is my first thought.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: MrAdults on 2009-03-07 19:10:11
One other thing that occurs to me is, even though the models seem to have only a single texture map, the models would need to be drawn in sorted/batching order for the purpose of alpha blending. For example, Reno's hair has a very soft-edged alpha, and as I recall playing the game, those surface did use real alpha blending instead of hard-edged alpha tests.

So, there would have to be a way to batch those verts into separate "surfaces", for depth-tested non-alpha (or hard-edged alpha) surfaces (like the whole body), and post-sorted alpha blended surfaces (like the hair). That dictates that there must be some kind of container with references to groups of verts, at least, if not an index list. Maybe if we can find these containers, it will shed more light on the vertex drawing order, or at least provide some clean absolute offsets into the vertex list.

Edit: Although, on second thought, it's possible that it was all hard-edged and I just could not tell because of the small screen and size of characters on the screen. :) My copy of the game is still at my friend's house, but if anyone has it handy, it would be worth looking to see if the character models do indeed use alpha blending.

Second edit: No, they are definitely blended. :) Take a look at Genesis's hair:
http://www.product-reviews.net/wp-content/userimages/2008/03/crisis-core-final-fantasy-vii-montage-for-the-sony-psp.jpg
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-03-07 23:09:03
I really like you two, speculative thinkers with the game-programmers mindset :-D
Although I have divided my reply, I hope both parts will be read and considered!

X-Dina:
Yes, understanding those "padding" values would be the key to solving this enigma. I have discovered some interesting patterns to help determine the vertex-data from the "extra" data:



These three are the most "concrete" rules, because they seem to work through all the files I have examined. The other tags which i had posted about a while back are unreliable.

Also, I noticed an interesting value at offset 0x14 (from the first byte after the "77777777777777" chunk seperator).
If you read it as a SHORT and minus one from it, it gives you the count of how many 24-byte (TYPE-1) vertices are listed starting from offset 0x1C.

So for the Moogle file (02373), the (absolute) file offset is 0x714 (local chunk offset 0x14), which gives a value of 0x42 (66). Then starting from (absolute) file offset 0x718 (local chunk offset 0x1C), you will find the first TYPE-1 vertex (tagged by "0000803f"), followed by 64 more such TYPE-1 vertices.

But the pattern breaks down after that, reaching TYPE-2 vertex data. I am sure there must be some kind of offset here too to determine where the next batch of TYPE-1 vertices begin, but it isn't easy to spot.

The Reno file (02190) has a value of 0x1 at the 0x14-offset (absolute:0xcb4), and contains zeros at 0x1c (ie, no TYPE-1 vertices), leading upto the first occurance of the "0000803f" tag (in this case it is an "embedded" TYPE-1 Vertex within the TYPE-2 data chunk).


MrAdults:
Interesting observation :-)
I know that z-sorting of transparent polys has to be done when developing for PC's, but dont games-console PPUs handle that sort of thing automatically? I am not an expert on console-hardware, so I will assume that the vertices would need to be stored in batches for optimal rendering of transparent objects.

Could we not conclude then, that we would expect to find similar transparancy-parts (eg, hair clumps) grouped together, not with any other vertices belonging to "solid" parts?
For example, for Reno, the vertices making up his hair would be distinct from the vertices from everything else (indexed or not).

The Moogle texture file is interesting.
(http://img9.imageshack.us/img9/7271/02373rawchunk2.png) (http://img9.imageshack.us/my.php?image=02373rawchunk2.png)
The only alpha it contains is within its wing-segment, but I can recall that the vertices belonging to the wings were a seperate batch of TYPE-1 vertices from its head. Likewise, Moogle-san's ears and nose were separate batches of TYPE-1 vertices from each other too.

Each of those seperate parts (batches of TYPE-1 vertices) has its own UV-space on the texture-map.
Maybe they just modelled (and later stored) these model parts in such a way that the transparant-parts would be differentiated on their own just as a "side-effect" of seperating the parts to better fit the UV map?

In other words, no discreet form of "transparent-part-tagging" or indexing would have been neccessary (or present)?
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Satoh on 2009-03-08 03:11:21
I think I know why they were separated into different groups.

When I import a 3DS file from 3DS Max, all of the edges that correspond to separate texture chunks, are always cut open and I have to re-sew them together again.

I believe the engine that saved the files had to separate the verts so there wouldn't be any texture warping on the model when rendered.

This would mean that when read by the PSP itself, it looks solid, only because the verts are touching each other, but in actuality there are sliced open edges all over.

Can you confirm this?

As for transparency, I believe the alpha is simply drawn as solid on the first pass over the entire model, then the engine does a secondary render that multiplies the alpha to create the transparency(the transparent rendering is kept in memory and is written to the screen from there rather than directly from the disc).

Again... speculation... based on past experience.

I believe, in a shorter phrase, the entire models are treated as alpha transparent enabled unless the texture image has NO alpha at all.


As for the 0x14 data.... that's very interesting...I had a feeling there was a header that noted the size in there somewhere...


and !!! the unknown data you mentioned that is 'will either have a zero UV or messed up XYZ value' could be the bone weighting info we've been guessing about!
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: MrAdults on 2009-03-10 03:43:53
Yeah, it is possible that they just leave both blending and depth writing enabled, and have the verts that reference part of the uv that is actually transparent ordered after the non-transparent verts. However, they would still have to software-sort their models by distance to camera in this case, or you would have cases where you would not see another character model behind a transparent surface, as that kind of sorting can't be done on the hardware level (due to the fact that it would need to defer rasterization to support it). It also assumes the PSP GPU has blend optimizations to save fillrate on blend operations if the written depth fragment is previously clear, which it may, but I'm unsure.

In either case, it would make a lot more sense to me to, at the very least, have some kind of marker between vertices to signify a change in blend mode, unless they were very desperate to cut down on primitive batches, without regard to the cost of software distance sorting or potentially greater fill expense. However, it doesn't make a lot of sense to do a post-depth alpha blend pass if you have rendered your alpha surfaces to the depth buffer, and the information in the framebuffer is stomped on whenever a depth fragment passes, which is why the hardware itself can't support an alpha post-multiply.

I'll have a look around some of those verts that reference transparent areas of the texture map, and see if I can find anything that would be indicative of a change in blending between batches. It seems more and more like the verts are more like a "draw command list" of some sort, which would explain some of the seemingly arbitrary tagging values thrown in there. However, I don't know enough about programming for the PSP, particularly with commercial GPU interface libs, to know if there are any official display list formats this data could resemble.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: halkun on 2009-03-10 04:51:18
I awake..... from my slumber....

When I helped reverse the FF8/CC models, it was no unheard of for the programmers to put GPU opcodes in the model data.

Sadly, I only know how the GPU works in the PSX/PS2. I have not have any hands on experience with the PSP.

Keep in mind the models probably were made from object trees (waist, torso, head, left_upper_arm, left_frearm, left_hand, etc). You probably need to work on parsing one body part at a time as the vertex pools are usually grouped together by body part. The Mog head was a good start.

Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Aurenasek116 on 2009-03-10 15:14:07
All I want to say is - this is one great piece of work. I'm looking forward to this project being finished and able to fully render the models.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Satoh on 2009-03-12 20:01:19
I awake..... from my slumber....

When I helped reverse the FF8/CC models, it was no unheard of for the programmers to put GPU opcodes in the model data.

Sadly, I only know how the GPU works in the PSX/PS2. I have not have any hands on experience with the PSP.

Keep in mind the models probably were made from object trees (waist, torso, head, left_upper_arm, left_frearm, left_hand, etc). You probably need to work on parsing one body part at a time as the vertex pools are usually grouped together by body part. The Mog head was a good start.



Halkun do you have any FF8 info that isn't posted anywhere? It would certainly help me out if you could add it to my Model Project thread, where I'm compiling model format info links.

All I want to say is - this is one great piece of work. I'm looking forward to this project being finished and able to fully render the models.

I think everyone that's posted here is eagerly awaiting the completion.

Yeah, it is possible that they just leave both blending and depth writing enabled, and have the verts that reference part of the uv that is actually transparent ordered after the non-transparent verts. However, they would still have to software-sort their models by distance to camera in this case, or you would have cases where you would not see another character model behind a transparent surface, as that kind of sorting can't be done on the hardware level (due to the fact that it would need to defer rasterization to support it). It also assumes the PSP GPU has blend optimizations to save fillrate on blend operations if the written depth fragment is previously clear, which it may, but I'm unsure.

In either case, it would make a lot more sense to me to, at the very least, have some kind of marker between vertices to signify a change in blend mode, unless they were very desperate to cut down on primitive batches, without regard to the cost of software distance sorting or potentially greater fill expense. However, it doesn't make a lot of sense to do a post-depth alpha blend pass if you have rendered your alpha surfaces to the depth buffer, and the information in the framebuffer is stomped on whenever a depth fragment passes, which is why the hardware itself can't support an alpha post-multiply.

I'll have a look around some of those verts that reference transparent areas of the texture map, and see if I can find anything that would be indicative of a change in blending between batches. It seems more and more like the verts are more like a "draw command list" of some sort, which would explain some of the seemingly arbitrary tagging values thrown in there. However, I don't know enough about programming for the PSP, particularly with commercial GPU interface libs, to know if there are any official display list formats this data could resemble.

I suppose you know a lot more about how alpha is rendered than I... I'll just let you handle that area XD;
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: halkun on 2009-03-12 22:17:29
I don't, I just remember seeing GPU packets mixed in with the UV data. The Packed discribed if the pool of vertexes were a triangle or a quad...

Hold on....

Hey, did you know that the GPU for the PSX could take a quads as primitive, along with triangles? The PS2 could take Tirangles, Quads, triangle fans, and triangle strips. Maybe that might help. You might be processing everything as if they were just triangles. (I'm blindly guessing here). Like I said, I don't know much about the PSP's GPU, but it could stand to reason that it may be similar to the PS2's...

Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-03-13 00:09:35
The more (people interested) the merrier!  :wink:

One of the first things I tried was rendering those vertices as quads, tri-lists and tri-strips (similar mechanism to rendering FF8 MCH files), but that didn't work because not all the vertices in these !-Files are vertices at all!
There are padding bytes, and distinct sub-headers of sorts, all garbled together.

The model "parts" are definitly categorised depending on the number of bones which influence them (TYPE1 being single weighted, TYPE2 being multi-weighted), but the ordering of these parts must be in-sync with how the PSP would render them, taking transparency into account. So heavy-transparent parts (like hair strands) we would expect to find towards the end of the vertex-data chunks. I haven't been able to confirm this yet.

Those "tags" are most definitly some kind of GPU opcodes (NeoCloudStrife has suggested this to me before).
For some reason (most likely because of improved efficiency) it looks like the mesh data is read directly from the UMD (or RAM) to the GPU for rendering, which is unlike how meshes usually get rendered on other consoles.
Maybe its a PSP thing, or maybe its just Crises Core?

What about Dissidia? Would that be worth looking into?


We really need to start by understanding how the PSP works I suppose!  :lol:
Are there any good documents available? I haven't been able to find anything detailed enough.

Its funny how FF7's file-format investigation had fueled better understanding of the PSX all those years ago, and now it seems like Crises-Core will do the same for the PSP  :-P
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: MrAdults on 2009-03-13 03:21:17
Yeah, I agree, having some PSP hardware/GPU documentation would make this a lot easier. Unfortunately, Googling is not turning up too many resources, but I don't think I'm searching for the right things. I keep getting this thread thrown back near the top of my search results. :)

I know that homebrew libraries have been able to utilize 3D features for some time. Unfortunately, I know those libraries work by loading official Sony kernel modules, so just looking at the homebrew devkit probably will not reveal low-level register info or GPU formats. I guess I'll keep Googling. Please anyone else who has some helpful links in this area, feel free to chime in. :) I'll report back if I find any applicable documentation.

Edit: I think I have just found almost everything we need to know in the source code for "Potemkin", a PSP emulator. It has vertex decoding/pipeline modules, and lists for all of the types and flags for vertex types, and it even shows how to handle the vertex weights within the vertex stream! :) The PSP appears to support up to 8 bones per command list, so meshes/command lists will no doubt be broken up for characters that use more than 8 bones. Now we just have to figure out the bone data. Each bone should be a 3x4 (or 4x3) matrix. From the emulator source, I believe it has to be floating point, but I may be missing a part where it does conversion.

Edit 2: I've verified that these vertex blocks are indeed a match to the PSP GPU formats and are decoded in the same way hardware decodes them. For "type 1" vertex blocks, there is a DWORD of value "0x00000723" usually 1 DWORD before the actual number of vertices. This value means:

Code: [Select]
int tc = 3; //2 floats
int col = 0; //no color
int nrm = 1; //x, y, z in bytes
int pos = 2; //3 shorts
int weighttype = 3; //float
int idx = 0; //none
int morphcount = 0;
int nweights = 0;

DWORD fmt = 0;
fmt |= tc;
fmt |= (col<<2);
fmt |= (nrm<<5);
fmt |= (pos<<7);
fmt |= (weighttype<<9);
fmt |= (idx<<11);
fmt |= (morphcount<<18);
fmt |= (nweights<<14);
//(fmt == 0x00000723)

When you get the sizes and offsets of the various components with that vertex type DWORD, padding and aligning along the way according to hardware spec, you get 24 byte vertex blocks with correct offsets to uv, position, and normal. Now that we have this knowledge, no matter what "type" a vertex is, we can know how it should be indexed (if at all), and where to get the essentials to render, such as position and UV. Now the only thing left to do is to figure out a reliable way to get those GPU control codes out between vertex batches. There is no reliable offset to the next control code/vertex number, from what I can find, and I'm not sure what the other bytes in between chunks mean yet.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: MrAdults on 2009-03-13 07:49:00
Success! :)

(http://richwhitehouse.com/pubimg/mogmodel.jpg)

You guys can probably piece things together from what I've said so far, but I will nonetheless be writing up the spec details (to include the PSP-related info) once everything is a bit more automated. Still haven't located a place to read the exact number of segments, or bothered to see if there is a command-terminator (there probably is).
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Aurenasek116 on 2009-03-13 07:56:49
Holy damn. That's really great about findind how the GPU works.
However I'd like to ask if it's normal that some of the textures that I view with the 0.30 version appear to have noise or scanlines on them, making them impossible to tell what are they. I was looking for Sephiroth's and Hojo's texture, I swept through all the model files and couldn't find them in those that were viewable, so my guess is their must be in those that have noise/scanlines or them, any way to fix that??
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: MrAdults on 2009-03-13 08:39:24
Alright, got this working perfectly with all of the models I have.
(http://www.richwhitehouse.com/pubimg/reno.jpg)
There is a single WORD (2 bytes) value after every segment, which is conveniently just "1" all the way until the last chunk, where it is 0. :) Another WORD right after that gives you an offset to look for the beginning of the next vertex data. Can anyone send me some more models to try out? Ideally some more characters (Genesis, Tifa, Zack, etc). I would like to start looking for skeletal data soon in order to actually make use of the bone weights.

Edit: Here is a full blog write-up of my findings, skip to the bottom for the meat:
http://www.richwhitehouse.com/index.php?postid=28
Please feel free to directly or indirectly take any information or code snippets and assimilate it into this thread or the wiki.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: yoshi314 on 2009-03-13 09:26:06
i must say i am impressed with how fast this was figured out.

either the consoles are getting more intuitive, or the hackers are getting smarter ;-)
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: BlitzNCS on 2009-03-13 16:04:47
HOORAY!!!!
WOOO!!!
haha! you guys did it! majorly awesome :D
really, thank you, all of you :D
i'm just sorry i couldnt help more!
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-03-13 16:06:16
WOOHOOO!!!
Fantastic stuff Mr Adults!

I tried it out, and it worked absolutely brilliantly!

witness the Crises-Core Character Collage (sample):

(http://img15.imageshack.us/img15/917/ccccn.th.jpg) (http://img15.imageshack.us/my.php?image=ccccn.jpg)

Alpha-blending has been disabled temporarily because Irrlicht needs a little work to correctly sort out the polys. I was just being lazy  :-P

The two Clouds (infantry and SOLDIER costumes) and Sephi have some strange texture-map issues, but there is no doubt the geometry can be parsed and rendered fully using this method.
I now just need to find a way to correct the texture problem, and it should be good to go for another public release!

To be honest, bones and animations were never on my TODO list (unless they had been required to correctly render the meshes) so I may not go any deeper into these Model files.
If Mr Adults tools would want to incorporate those things, then it would make it easier for me to just move onto something else.

THANKYOU EVERYONE for helping me get so much done so quick!
I suppose all the right people just happened to be at the right place at the right time (doing the right thing) :lol:
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Aurenasek116 on 2009-03-13 16:48:33
Now if only someone could compile a viewer for those and release it since I have totally no experiance with coding whatsoever, only 3Ds Max, Maya and Photoshop.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: spidergaby on 2009-03-13 17:48:20
Wow !!!  :-o
Awesome work !!  :wink:

Please continue  :-)
Now if only someone could compile a viewer for those and release it since I have totally no experiance with coding whatsoever, only 3Ds Max, Maya and Photoshop.

I'm agree with you Aurenasek116  ;)
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: BlitzNCS on 2009-03-13 18:44:55
Just a question, is it possible to add a sort of "Convertor" type thing for the models like what biturn has? for instance to just turn these models into .OBJ or .3ds files, because i have vista, and 3DRipperDX seems to have trouble with vista as far as i can tell. it crashes the instant i try to dump with shader model 3.0, and when i use shader model 2.0, it doesn't do it right at all...
But you guys rock! apart from animations and bones, what else would there be to look into for this game?
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: ultima espio on 2009-03-13 18:50:03
Neocloudstrife, 3d printscreen would work well with it i think. If we try with it first, it's less work for Koral/MrAdults.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: NameSpoofer on 2009-03-13 20:06:45
Once again I would like to say thank you to koral and those others ( ignitz, mradults, x-dina ) for this awesome project.

The models from CC put be great for the mods Im working on.

Muchas gracias

Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: BlitzNCS on 2009-03-13 20:56:06
Neocloudstrife, 3d printscreen would work well with it i think. If we try with it first, it's less work for Koral/MrAdults.
That doesn't seem to want to work for me either (crashes on loading anything 3D)
STUPID VISTA  :cry:
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-03-13 22:15:43
Don't worry, I will add in a .OBJ exporter as standard with the next release :wink:
It is such a simple format to work with, it wont be a problem for me to do and for you guys to get into Max, Blender or whatever

I am converting the project into a Win32 MFC form, so it should be easier to work with.
Or maybe if mirex could add support for these files into Biturn then I might not have to?


apart from animations and bones, what else would there be to look into for this game?

There are maps, missions, materia, and the Debug-Menu somewhere in there too!
I also believe this game may be quite easy to Mod because of how the files are layed out, although I am not sure how else this game could be improved  :-P

But I am trying to solve the Sephi and Cloud texture issue right now, because a lot of other characters suffer from this problem too (such as Angeal).
It wouldn't do to release a buggy version, would it?
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: BlitzNCS on 2009-03-13 22:27:06
you're right, and as someone mentioned earlier, i think they're the ones with the weird scanline thingies. the textures are probably in a different format for some reason. they look like how the other textures started out, before you got them working.
lol, maybe it's just a joke from square or something. is it just a coincidence it happened to both cloud AND sephiroth? :P
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Aurenasek116 on 2009-03-13 22:51:57
I've worked with many console games before, and they all used .dds format for textures. It's common format used in games these days, it's very similar to .tga but smaller than it. It simply preserves high quality and alpha layer while maintaining smaller disk space than bloody .tga. I don't know if that helps but I'm trying to be helpful.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-03-13 23:33:03
I always use PNGs for texture-maps, lossless compressed textures with alphas. Not as good as DDS I will admit, but definitly easier to for windows (open with Paint etc to view them). TGAs are useless  :-P

Anyway, I just discovered that there are multiple Textures in some of these !-Model files!

Example, the Cloud [file: 02183.raw] contains three textures:

(http://img23.imageshack.us/img23/9938/02183rawchunk2.png)  (http://img15.imageshack.us/img15/8572/02183rawchunk4.png)  (http://img18.imageshack.us/img18/6577/02183rawchunk3.png)

The way to access these other textures is to jump to the Texture-Offset (from the file header at 0x1c) then to read in 16-bytes until the first 0x77 byte.
Each of those 16-byte is a texture-map header which contains the image sizes and file offsets of where the pixel/pallete data are contained (ie, one 16-byte header per texture).

Ofcourse, this now suggests that the vertex data must also contain within it some kind of Texture-Index so the correct texture-map can be applied.
Seems like we are not quite done yet  :|
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: MrAdults on 2009-03-13 23:38:49
I was just coming here to tell you this, koral. :)
(http://www.richwhitehouse.com/pubimg/seph.jpg)
It seems you have already figured out how to get at the multiple textures. I'm certain there are also multiple models outside of the main segment buffer, that can probably be accessed as well at the end of the main segment list. Not sure how that is exactly set up yet.

But! We got very lucky, and are given a WORD that is a direct index into our texture list for which texture to use when rendering a given segment. :) Going along with the example code on my web site, to get at that word, you would do this:
Code: [Select]
WORD texIdx = ((WORD *)vinfd)[2];And then texIdx is a simple 0-based index into the texture list, in the order it was parsed. I have tested this across quite a few models, and it seems reliable.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Aurenasek116 on 2009-03-14 00:37:45
Could you add option to extract the textures in .dds or .tga format though? PNG alpha layer isn't as good as dds/tga in jedi academy engine, which is modified quake 3 (For which I mod :P)
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: MrAdults on 2009-03-14 00:56:16
Haha, that is a funny coincidence, Aurenasek116. I was responsible for the multiplayer portion of Jedi Academy. :) And Jedi Knight II, actually.

Not sure what you mean by the alpha layer isn't as good, though. DXT5 compression is pretty far from lossless, and you should be able to export a png to a dds and retain the alpha channel. I personally use PNG for everything as well, and distribute both PNG (for lossless no-compression rendering) and DDS for all of my games. It should work great, unless I'm totally unaware of some png alpha limitation.

BTW, I forgot to mention. Once I'm done integrating this support into my toolset, it will be able to export the models as well, to my own model format as well as .smd (Valve's pre-processed model format, for which every modeling suite in existing has an importer). Which will be useful for retaining skeletal/weighting data, if we ever get that fully working. However, it's not in my immediately scheduled timeline to get that release out. It will probably happen sometime after I look more into the skeletal data, to make sure it isn't just an obvious/simple thing to get working. koral's obj export functionality should be quite suitable for exporting static meshes, so I'm not worried about getting that functionality out there.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Aurenasek116 on 2009-03-14 04:30:42
In what way responsible for the multiplayer portion? Do/Did you work at Raven?

As for PNG I've noticed that atleast in jedi academy when using alphafunc with it on detailed half-transparent textures it seems to ignore the small radius around the alpha resulting in black/white outline while rendering. TGA doesn't have that problem.

Retaining weighting would be great since it could allow exporting models in poses using original animations instead of that "Jesus-stance".
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: apz freak on 2009-03-14 06:15:45
Fabulous work guys! I look forward to exploiting this to further my low poly knowledge!
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-03-14 12:32:43
Yaay!

(http://img13.imageshack.us/img13/1609/ccccb.jpg) (http://img13.imageshack.us/my.php?image=ccccb.jpg)

But it seems to crash strangly when I try to load Genesis [file: 02170.raw], also with the Barrel model [file: 02450.raw]. Maybe it is something I am doing wrong somewhere with texture loading, because they were working fine before i started playing the multi-textures.
Apparantly, those files don't contain any 0x77s to seperate the texture-data headers from the textures, so it is probably just a case of mistaken identity.
Perhaps there is a texture count somewhere?


And you may notice something peculiar with Tifa here [file: 02192]:
(http://img3.imageshack.us/img3/219/cctf1.jpg) (http://img3.imageshack.us/my.php?image=cctf1.jpg)

No hat!
Remember my earlier attempts showed her hat, and everyones weapons too.

MrAdults, you suggested the possibility of multiple models right? I will have to look for them before making the release, but if the multi-textures are any guide, then they should follow on immedietly after the primary model data.


And NO, I will not be adding in any more export formats for textures, because it should be simple enough to convert them to other formats.
I may have to switch to X files later if we discover precise skeletal and animation data, but OBJ should suffice until then.

Maybe we are getting a little ahead of ourselves here  :-P
Thanks for the support everyone!  :-D


EDIT:
I figured out the Texture-Count thing, there is a SHORT at 0xe of the file-header giving a count of the number of texture-headers present:

(http://img13.imageshack.us/img13/1615/ccg1a.jpg) (http://img13.imageshack.us/my.php?image=ccg1a.jpg)

now to look for the count and location of other models within the same file!  :lol:


EDIT2:

Figured that out too!

(http://img7.imageshack.us/img7/9463/cccd.jpg) (http://img7.imageshack.us/my.php?image=cccd.jpg)

The file header contains a SHORT at 0xa of the number of models, and jumping to the vertex-data offset simply read in the offset to the start of the vertices, located every 8-bytes. There is an unknown DWORD before each offset, a count for something in each vertex group but no idea what.

I ran through ALL the character !-Files [files: 02163 - 02457] and they all load up correctly, although some have screwed up texture-maps (because of differing interlacing or something).

So I can now happily annouce that the next release is on its way!
ETA: sometime later today
 :-D
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: ultima espio on 2009-03-14 15:57:52
 :-o :-o :-o Neocloudstrife is gonna die when he sees this! (http://i85.photobucket.com/albums/k61/ultima_espio/1aatIgMD1C68W2CqAIP65EXMC8.gif)
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Aurenasek116 on 2009-03-14 17:47:42
If this gets released you will see swarm of crisis core models ported to Jedi Academy.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-03-14 18:08:26
Here is the link: [link removed, baaaad file]
I have updated the first post too, get it now and extract to your heart's desire!  :lol:

Wouldn't it be cool if Marlie-chan (and her friends) could find their way into a new FF7 mod?
(http://img5.imageshack.us/img5/6353/ccmarlie.jpg) (http://img5.imageshack.us/my.php?image=ccmarlie.jpg)
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Aurenasek116 on 2009-03-14 18:23:45
Uhm, I get error when trying to run the exe. It says the application has wrong configuration and can't be run...
I'm running Win XP Professional 32bit if that's needed.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: ultima espio on 2009-03-14 18:28:58
Koral this is amazing, thank you sooo much for your hard work!
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: ignitz on 2009-03-14 18:29:46
HUUUUUUU, KORALL GREAAAATT WOOOOOOORKK, you made it, yes, now you unleashed, XD

-------------------------------
Same problem with font and crash with 3D model, can you be fix these things?
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-03-14 19:40:57
iiya, sorry about that  :oops:
I was messing with MFC and forms and stuff, probably some rubbish got left over, I will need to sort it ASAP!
Sorry

EDIT:
um, try this (v0.4b, GL): http://www.mediafire.com/?mtz0nyl2d3h
please tell me if this doesn't work, it must be something else then...

EDIT2:
this is v0.4c, which uses DX9 and is perfectly compatible with 3DRipperDX: http://www.mediafire.com/?yewwz3nnlwy

The program is extremelybuggy and likes to crash sometimes when you select a file.
There is nothing else I can do unless I completely revert away from irrlicht.

Sorry I couldn't make it easier :cry:
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-03-14 19:54:09
[ignore]
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: MrAdults on 2009-03-14 20:43:14
Awesome work again, koral. :) I was looking for skeletal data again last night, it's definitely not there in any obvious form. It seems there are groupings of 3 ints and 4 floats in one of the files. There is a short that specifies the number of those groups (seems to be "3" a lot), followed by that many groups, and some other unknown data in between. Those 4 floats could be a quaternion, but the 3 ints don't make sense as coordinates in any kind of interpretation. Plus it would be very strange to hold bone orientations in these kinds of groupings. So, I'm not confident that these models actually have the bone data/skeleton in them, though I'm not discounting it yet. But it would not be uncommon to have the skeleton in another file with the character's animations.

It seems very possible that the skeleton is "flattened" with no hierarchy, and because matrices are base-relative, the animation data could just be a flattened series of orientations that were computed based on the base pose matrix, which is no longer needed at run-time. The benefit of flattening is that no hiererchical transform is needed for joint matrices, although there is a downside of requiring more storage space for orientations. However, for Crisis Core, this means they could get away with absolutely no CPU cost for bone transforms. I am guessing they could not resist that even in the face of some extra memory overhead. ;)

All of that said, I did find where the bone index values are stored for each drawn segment (I think). So now I know which bones each existing weight is referencing. With that knowledge, I can actually re-create a skeleton and approximate the original joint hierarchy. I'm going to go ahead with this approach, because it will allow the characters to be exported and posed with rough bone positions, keeping the original vertex weighting intact. Since weighting is the hard part, this is suitable to my personal needs, at least. If the skeleton has problems, that's easy to tweak and change hierarchy in a modeling program, while retaining weighting info.

Quote
In what way responsible for the multiplayer portion? Do/Did you work at Raven?
Yeah, I worked there for a few years. As you probably know, we had separate executables for singleplayer and multiplayer. I was the guy responsible for the multiplayer codebase (which meant writing all of the MP-specific engine/game code, porting over all of the SP code to be multiplayer-friendly, etc).
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Aurenasek116 on 2009-03-14 22:20:25
Ah I see. Well then you know it's sad the game never got patched further than 1.01 due to running out of license for Star Wars universe (Or atleast that's what I heard when I was still a staff member at jk3files.com). I know I probably shouldn't ask for that but many people in JK3 community always wanted SP source code, would it be possible to obtain it? That's all for offtopic.

I've noticed that when the model exports to obj, double faces are facing the same way, which means one has to copy mesh and flip normals on it to appear without holes, but then when merged together the mesh will have quadruple faces which sadly in game engine like Quake 3 when shaded looks like chessboard except with triangles instead of squares. Is it possible for model viewer for export the double faces correctly?

PS. If you can't understand what I mean here's the picture:

(http://i373.photobucket.com/albums/oo176/xxSchrodingerxx/1-1.jpg)
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: MrAdults on 2009-03-14 22:34:35
Quote
Well then you know it's sad the game never got patched further than 1.01 due to running out of license for Star Wars universe (Or atleast that's what I heard when I was still a staff member at jk3files.com). I know I probably shouldn't ask for that but many people in JK3 community always wanted SP source code, would it be possible to obtain it? That's all for offtopic.
Nope, unfortunately, it can't happen. I have been advocating releasing that code, along with multiplayer engine code, for years (well after leaving Raven/Activision). Unfortunately, LucasArts will not respond to my inquiries. It's all in their hands - Raven is happy to pack up the code if Lucas gives the go-ahead. I've even asked for John Romero's help (since he has played a part in many Quake-engine-related open source releases), and LucasArts would not reply to him either. Additionally, no one that was working at Lucas and supervising/producer JK2/JA is even there anymore, from what I can tell. They seem to completely cycle out employees quite often - apparently most of the Force Unleashed team is no longer there either. As you can tell, I'm no more pleased by the situation than the rest of the community, but my repeated efforts over the years have all failed.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: BlitzNCS on 2009-03-15 01:40:06
OMGOMGOMG!!!
you guys are amazing! thanks so much koral!
i'm actually on a mobile phone so i can't try it, but i WILL :D
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: MrAdults on 2009-03-15 03:05:40
Good news! Even though I still can't find anything that could be a bone orientation in the file, that chunk we've known about for a while now at 0x80 contains hierarchy data. All of those 0xFFFF's (-1 short) initially indicated to me that the skeleton was flattened. However, I decided to ignore that number, and parse through those chunks up to numBones using the first value as the parent, and that's what it was. Additionally, the first WORD right before the number of models in the header appears to always equal "numBones-7". Why -7, I'm unsure, but even "static" models like the crate that have only 1 bone to reference with all 1.0 weights have a value of 8 here, making it consistent with the fully boned/skeletal models.

The bone orientations can be approximated with knowledge of the associated weights and the hierarchy, now, so we get a full skeleton for the vertex weights to interact with.

The only problem is, those values I thought indicated which bones a segment referenced don't appear to be right. They are 100% right for 1-weight segments, so stuff like the machinery and choppers has a nice perfectly-weighted skeleton. However, soft weights aren't using the values I thought for indices. I have no idea how to get the right value for that, so I'll be continuing to look for that.

Edit: Alright, I figured it out. For reference, bone index values are located starting the WORD before the number of vertices/primitives for the segment, and then skip 1 WORD between each index. Not sure what those WORD values are in between.
(http://www.richwhitehouse.com/pubimg/renoallboned.jpg)
I'll post back here once my tools are all ready for distribution.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Aurenasek116 on 2009-03-15 04:57:36
Wow, that's a great progess there. With ability to import models with their weighs it will so much easier to pose them.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Silver Wings on 2009-03-15 10:21:37
You guys are just amazing, thank you so much  :-D
I've been exploring through out files containg the models (2012-2457) and make a list for you (at least, that's all I can do, hope it would help somehow ^^)
Just a note:
- I cant view files from 02012 to 02160, every time I choose the files, the viewer crashes immediately
- In the others left (2161-2457), there are some files didnt show up any models so I didnt list them
Code: [Select]
2162 Zack - Soldier 1st Class
2163 Zack - after Angeal's death
2164 Zack - Soldier 2nd Class
2165 Zack at the beach
2166 Angeal
2167 Angeal's clone (Hound form)
2168 Angeal w/ wing
2169 Sephiroth
2170 Genesis
2171 Genesis w/ wing
2172 Genesis being degenerated
2173 Genesis's clone w/o mask
2174 Lazard
2175 Lazard in Angeal form
2176 Tseng
2177 Cissnei
2178 Cissnei at the beach
2179 Hojo
2180 Hollander
2181 Hollander at the last stage of degeneration
2182 Aerith
2183 Cloud - the infantry man
2184 Cloud in Soldier 1st class uniform
2185 Infantry man - the captain
2186 Infantry man
2187 Soldier 2nd Class
2188 Soldier 3rd Class
2189 Gillian
2190 Reno
2191 Rude
2192 Tifa
2193-2194 Angeal and Zack with weird shadow on faces (actually, the 2 Soldiers in the Epilogue)
2195 Researcher
2196 Man in suite
2197 Woman in suite
2198 Man
2199 Woman in suite
2200 Boy
2201 Girl
2202 Man
2203 Woman
2204 Man
2205 Wutai warrior
2207 Spriggan
2208 Thunderbird
2209 One eye
2210 Angeal's clone (One eye)
2212 Angeal's clone (Sahagin form)
2214 Angeal's clone (Hound form)
2215 worm
2216 Stinger
2218 Flying machine
2219 Bomb
2220 Griffon
2221 Angeal's clone (Griffon form)
2222 Demon
2223-2224 Genesis's clone w/o wing
2225 Genesis's clone w/ wing
2227 Genesis's clone
2228 Genesis’s clone being degenerated
2229 Sweeper (machine)
2230 Tarantula (machine)
2231-2232 Tank
2234-2235 Wutai's giant monster
2236 Ifrit
2237 Bahamuth
2238 Fury Bahamuth
2239 Angeal Penance
2240 Genesis Avatar
2246 Chair
2250 Truck
2252 Aerith
2253 Sephiroth
2254 Zack - Soldier 2nd Class
2255 Angeal
2256 Genesis
2257 Mask of Genesis's clone
2259 Needle machine
2260 Chain machine
2261 Missile machine
2262 Sweeper (machine)
2263 Wutai's giant monster
2264 Thunderbird
2265 Hound
2267 Dual horns
2368-2370 Saucer (machine)
2371-2372 Alert Head (small machine w/ armour)
2373-2274 Drill machine
2275 Tarantula (machine)
2276 Support machine
2277 Bug
2280 Sephiroth
2281 Genesis's clone (Predator)
2282 Tseng
2283 Sephiroth
2284 Zack - Soldier 1st Class
2285 Zack - after Angeal's death
2288 Man
2289 Woman
2290 Boy
2291 Girl
2292 Angeal w/ wing
2293 Angeal
2294 Zack - Soldier 2nd Class
2295 Genesis's clone
2296 Genesis’s clone being degenerated
2297-2298 Genesis's clone (Predator)
2300 Files
2304 Lazard
2305 Lazard in Angeal form
2306 Cissnei
2307 Genesis w/ wing
2308 Helicopter
2309 Genesis being degenerated
2310 Angeal clone (Sahagin form)
2311 Cloud - the infantry man
2316 Heel
2317-2318 Wutai warrior
2320 Hornets (bees)
2321 Death Claw
2322 Mover
2324 Dorky face
2325 Hungry
2327-2328 Grangalan
2329 Chocobo
2330 Magic pot
2331-2333 Tonberry
2334 CaithSith
2335 Hollander
2336 Motorcycle
2337 Angeal w/ wing
2341 Infantry man (the captain)
2343 Sweeper (machine)
2345 Genesis’s clone w/o wing
2346 Genesis’s clone w/ wing
2347 Man
2348 Woman (the shopkeeper)
2349 Boy (the little thief)
2352 Woman
2353 Boy
2356 Woman
2357-2358 Boy
2359 Juice-can
2361 Book (cyan)
2362 Dumbapple
2363 Book (magenta)
2366 Wutai warrior
2367 Genesis's clone (Infantry man form)
2368 Shinra rocket
2369 Cissnei at the beach
2370-2371 Genesis's clone (Infantry man form)
2372 Genesis’s clone (Predator)
2373 Moogle
2374 Zack at the beach
2375 Rooms
2376 Normal Behemoth
2377 King Behemoth (golden)
2378 Cloud in Soldier 1st class uniform
2379 Genesis’s clone (Dominator)
2380 Hollander at the last stage of degeneration
2382 Worm
2383 Dual horns
2384 Item box
2394 Jenova
2398 The monster in tube
2399-2401 Genesis’s clone (Shadow scythe/knight/mage)
2402 Camera
2405 Top floor
2406 Path?
2407 some kind of machine
2408 Hollander at the last stage of degeneration
2410-2411 Genesis at the last stage of degeneration
2412 Wutai warrior
2414 Tarantula
2415 Sahagin
2416 Stinger
2418 Spriggan
2419 Alert Head (small machine w/ armour)
2420 Hound
2421 Demon
2422 Dorky face
2424 Death Claw
2425 One eye
2426 Bug
2427 Bowl?
2428 Water pot
2429 Strong box
2430 Golden Coin?
2431 Minerva (The Goddess)
2432 Malboro
2434 Black woman in suite
2435 Black man in suite
2436 Yuffie
2438 Documentation?
2439 Statue
2440 Hollander's bag
2441 Plain?
2444 Genesis’s clone (Dominator)
2443 The monster in tube like
2444 Genesis’s clone w/ wing
2445-2447 Flower Wagon
2448 Sniper
2449 Truck
2450 Barrel
2451 Helicopter
2453 Potion
2454 Helicopter
2455 Worm
2456-2457 Zack's last moment
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: MrAdults on 2009-03-15 11:02:10
Alrighty, a download link to the tool and all of the information you need to get going is here:

http://www.richwhitehouse.com/index.php?postid=30

Unless there are terrible bugs with the tool, I'm stick-a-fork-in-it-done with this format. :) I don't intend to go find the bone orientations/animations, although if anyone else ever does, I'll probably go ahead and add support for them to my tool. Currently the exported bones are generated based on the weighting data, and since they do have the original hierarchy, they seem to be quite sufficient for posing the models.

Also, my tool does preview, but it's very barebones functionality. It's mainly intended to be a powerful exporter. If you just want to look at the game content, you should probably stick to koral's tool.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: DiNeRo on 2009-03-15 13:09:36
this is epic

im still a noob at weighing, but youll see lots of crisis core models in JK3 soon  :roll:
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: ultima espio on 2009-03-15 14:06:54
The Zack and Angeal with weird heads, they are infact weiss and nero :wink:
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Silver Wings on 2009-03-15 14:16:07
Oh, I totally forgot the 2 strangers in the epilogue :D thank you for pointing it out ^^
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-03-15 14:50:45
Awesome findings MrAdults!
I haven't had a chance to look into the skeleton stuff yet, but it seems to make perfect sense.
I tried your viewer too, works with no problems. I suppose that was one way of making the viewer easier to use!  :-P

The character/monster/event-model files are the only ones which parse correctly at the moment (as SilverWings has just confirmed  :-D).
I am sure that all the other files before that (02012-02160) must be Locations/Map-models which are similar to the viewable ones, minus animation data.

Sorry about the problems with my viewer Aurenasek116, but that is happening because of how the data was read from the file, and I have been too lazy to resort the polys in a more efficient way. Maybe I will look into it in the future, or you could just use MrAdults's tools (as and when they appear).

Since Rich (MrAdults) is doing such a fabulous job on the skeletal and animation front, I may as well delve deeper into those other files (02012-02160) and see what I can dig-up.

Thanks everyone!
 :-D :-D
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Aurenasek116 on 2009-03-15 16:54:42
I'm using MrAdult's extractor already and it does a perfect job with extracting models so no problem there koral, great work nonetheless since I prefer to use your viewer as it has a GUI to view models, and MrAdult's for extracting.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-03-16 00:09:47
I agree, MrAdults's tool is a much better choice to extract what you find!

Just shouting out that I have updated the first post and the wiki a little.

I haven't got around to documenting these !-models fully yet, but it will be quite a large task and I don't want to butcher it.
I owe much thanks to NeoCloudStrife, X-Dina and MrAdults for making it possible, and for their sake it will have to be a job well done.

Materia information is still hidden, which would be the first step to discovering if the Debug-Menu truly is called through a Materia, or if it is just a special kind of Event.
If it's the latter case (an Event), then there is a very high probability that a simple memory-hack cheat could be used to trigger it somehow.

The game itself does not seem to have anything more "mysterious" or unused that I have come across, which is what you would expect from games these days I suppose.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: ff7rules on 2009-03-16 01:30:33
Alrighty, a download link to the tool and all of the information you need to get going is here:

http://www.richwhitehouse.com/index.php?postid=30

Unless there are terrible bugs with the tool, I'm stick-a-fork-in-it-done with this format. :) I don't intend to go find the bone orientations/animations, although if anyone else ever does, I'll probably go ahead and add support for them to my tool. Currently the exported bones are generated based on the weighting data, and since they do have the original hierarchy, they seem to be quite sufficient for posing the models.

Also, my tool does preview, but it's very barebones functionality. It's mainly intended to be a powerful exporter. If you just want to look at the game content, you should probably stick to koral's tool.

Wow thanks for this Mradults! im currently intersted in the psx version, model changing and stuff (see the replacing and adding new models thread)
I do have a question though would it be possible for you to make it so we can extract the PSX models so we can finally change them? :D Like all there data and will be just like they were when we compress them back to lzs? Also i noticed the weapons are in the middle of the model any chance of us ever been able to edit them? Thanks for this.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: MrAdults on 2009-03-16 05:02:53
Wow thanks for this Mradults! im currently intersted in the psx version, model changing and stuff (see the replacing and adding new models thread)
I do have a question though would it be possible for you to make it so we can extract the PSX models so we can finally change them? :D Like all there data and will be just like they were when we compress them back to lzs? Also i noticed the weapons are in the middle of the model any chance of us ever been able to edit them? Thanks for this.
That FF7 LZS support has been in the tool for quite a long time, I did it a year or two ago, I believe it was. You could use the tool to convert those models to .smd, but they would lose the vertex coloring. I don't know much about what's out there currently to extract FF7 PSX models, I figured there was already quite a myriad of tools to view and convert the models. I believe .ase export would support everything FF7 models need, including vertex colors and node (mesh) orientations, but my tool doesn't currently export to that format.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Aurenasek116 on 2009-03-16 07:16:09
Just wanted to show you some progress on converting to JK3:

(http://i373.photobucket.com/albums/oo176/xxSchrodingerxx/ISITOVERNINETHOUSAND.jpg)

It still needs work, mainly weighs on hips and textures to be added on hands, since in JK3 it's simplier to use basic hands from already default base model rather than the ones from crisis core models. Wing can be turned off in .skin file allowing for easier management ingame. (That was pretty much directed only at those who know about JK3 modding).

Cheers and thanks to Koral and MrAdults aswell as others that helped making this project possible.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: ff7rules on 2009-03-16 13:31:30
There's no Tools with the ability to make so we can create and add to old ones yet. I did some testing and the models seem happy to replace with other stuff, Like i replaced Ruby weapons model with Sephiroth worked, same with Cloud and Tifa. I could also replace Nib Sephiroth with anyone i wanted so the models seem quite flexable as far as moving them goes. Any Chance of been able to export to .ase format in the future? so we can edit them and add to them, Also are the weapon's just meshes from bones or are they separate from the model?
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: DiNeRo on 2009-03-16 16:24:17
to add to Aurenasek's post (we are working together, me as his weigher)

currently we have weighed for jka genesis and sephiroth(although both have major weighing issues, with clipping mainly - i am still working to fix that)

(http://i135.photobucket.com/albums/q140/DiNeRo_/Untitled-1-8.png)
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: MrAdults on 2009-03-17 00:01:00
There's no Tools with the ability to make so we can create and add to old ones yet. I did some testing and the models seem happy to replace with other stuff, Like i replaced Ruby weapons model with Sephiroth worked, same with Cloud and Tifa. I could also replace Nib Sephiroth with anyone i wanted so the models seem quite flexable as far as moving them goes. Any Chance of been able to export to .ase format in the future? so we can edit them and add to them, Also are the weapon's just meshes from bones or are they separate from the model?
Ah, it sounds like you want the ability to convert back to the FF7 PSX model format, too. I'm not sure how well that would work out, given that you'd probably have size and address restrictions to maintain the alignment. In any case, it's not something I plan to do, as it would be a lot of additional from-scratch functionality. I've seen PC version replacement models, though, so on that side of things there must already be a tool-chain for what you want to accomplish.

BTW, cool to see the models making their way into Jedi Academy.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Satoh on 2009-03-17 04:48:04
I picked the wrong week to not check up for a few days O_O;

That's absolutely amazing, Koral.

Now I just have to rip my disc and start digging through the models for myself!! *starts cackling like Hojo*

EDIT: Ok after much floundering about with my PSP... I successfully...

gave up and downloaded a copy... but I do have the game legally, hell, I bought a PSP JUST for crisis core.

anyway, I have now ripped a series of models XD; I'm ecstatic!

Anyway I noticed the viewer tends to crash a lot after trying to switch from 2D image into 3D renders... not the other way around... Not a big issue...

The one thing that does rather annoy me is the files starting at 02012 and up (on the front page this is the start of the model section) seem to auto-crash me... however working from the bottom up, I can view a lot of models... Is there a possibility of this being just me, or is it the files themselves, or something else entirely?

Btw, I should say again that I am dumbfoundedly impressed with this as is. I especially like the OBJ export function!

I hope that when you're satisfied with this format you'd be willing to investigate some others. (time and relaxation permitting that is. After all... you've done a lot of work already, you deserve a vacation... and a cookie.)

EDIT2: 02012 - 02161 seem to be offending files on my side... I haven't checked every one of them... but the first and last 10 with several sample tests in between all crash me as soon as I even click on the number.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: ff7rules on 2009-03-17 12:49:11
Quote
Ah, it sounds like you want the ability to convert back to the FF7 PSX model format, too. I'm not sure how well that would work out, given that you'd probably have size and address restrictions to maintain the alignment. In any case, it's not something I plan to do, as it would be a lot of additional from-scratch functionality. I've seen PC version replacement models, though, so on that side of things there must already be a tool-chain for what you want to accomplish.

BTW, cool to see the models making their way into Jedi Academy.

Yeah, I'm thinking of turning my full attention to the PC version  since PSX can't be modded to the fullest like PC can. The formats are allot different PSX uses LZS compression as you know, and PCs version's files are in pieces. I know one way i could get the Model though 3D printscreen, but its pointless as a loss  of data for animations and such is inevitable. As far as i know, there is no way to keep the data. Anyway thanks for the reply.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-03-18 19:09:07
I have been wondering why those files (02012-02160) crash and after looking into them I think I may be onto something.

Those files have the exact same file-headers as the viewable character ones, and the offsets also lead to exactly the correct parts of the files.
There are multiple textures present in there too which i can confirm are identically organised as the viewable files (albeit with different interlacing with most of them).

The reason why they crash though is because their Vertex-data is completely different.
In the viewable-model files when we jump to the Vertex-data start (dictated by the offset at the beginning of the file) we always skipped 4-bytes that were always zeros.
In these other files, those four bytes have a different value: they represent a DWORD and hold a value of 1.
These files also contain many model-parts (50+ usually).

Looking at the data which followed the initial DWORD, there was nothing but a huge list of SHORTS, and those could be plotted as XYZ-vertices to produce almost recognisable shapes: Location Map models.

And these vertices represented QUAD-Lists, not Triangle-lists.

But here is the twist:
Only the first 8-vertices (8 times 3-SHORTS for XYX) at the beginning of every model-part seemed to be "real" vertices, the rest were "fakes" (potentially UV data).

And I can't seem to determine the group-strides, but I am extremely sure that these vertices will be organised in a far more logical manner.

I will continue looking into it, but if anyone would like to help then they are more than welcome  :wink:

Stay tuned, the characters were only the beginning  :-D
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: MrAdults on 2009-03-18 22:54:55
I haven't actually looked at those files at all, but in pure speculation, I believe they should be embedded with some kind of scene graphing data structure. A quad-tree would seem quite suitable for those map layouts, but it could be anything.

It seems a good bet that the positional data you're looking at is actually part of that encapsulation tree, which is why it might happen to look right rendered as verts with quads, as the data nodes are probably in a specific order with a rather uniform alignment, especially if it is a quadtree.

If I'm right, there is probably an index or index list associated with most of those nodes into an actual surface bundle. I'm betting the actual vertex lists are still in the same PSP-GPU vertex format. They may be more versatile in storage than the character models, though, so you might have to use those GPU specs to support other types of coordinates and primitive indexing.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Satoh on 2009-03-19 02:22:13
maybe they're the walkmesh and wall collision models for the area maps...

That was my first though from your post anyway...

EDIT: Also I noticed a few models that seem to have broken textures...
02336 has a broken texture on the wheel of the bike, can you confirm this to make sure it isn't just a bad dump?
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-03-19 11:31:23
Those "broken textures" are textures which have different interlacing modes, and it looks like most of those models use them. The viewer doesn't support that interlacing mode yet.  :-)

MrAdults is onto something there, because those quads are all aligned perfectly in space so most likely represent Bounding-Boxes of some kind per model-part.
I haven't seen any familiar TYPE-1 or TYPE-2 vertex structures though, but maybe the data will still parse through correctly.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-03-19 20:58:33
They contain nothing but TYPE-1 vertices afterall! All in a nice long triangle-list.

(http://img10.imageshack.us/img10/4797/ccls1.th.jpg) (http://img10.imageshack.us/my.php?image=ccls1.jpg)(http://img12.imageshack.us/img12/3559/ccls2.th.jpg) (http://img12.imageshack.us/my.php?image=ccls2.jpg)
(http://img14.imageshack.us/img14/1214/ccls3.th.jpg) (http://img14.imageshack.us/my.php?image=ccls3.jpg)(http://img8.imageshack.us/img8/2931/ccls4.th.jpg) (http://img8.imageshack.us/my.php?image=ccls4.jpg)

These places dont seem recognisable, but I probably missed a chunk of model-parts somewhere. And once the textures are in place they should become obvious.
All i need to do now is find the index to textures and we should have these location models in the bag too!  :-D
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: BlitzNCS on 2009-03-19 23:12:32
Wow, this is amazing koral :-o
I've never actually used a map in a 3d modelling program before O.o
BTW, number 2 looks like the training room from my guess....
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: MrAdults on 2009-03-19 23:19:19
Sweet. :) I wonder if there is visibility data in there too, specificying visibility of each of the quad (or whatever they are) nodes to each other. If we preserved that, we could automatically convert it over to a BSP'd, vis'd Quake 3 map or something. That would be pretty novel. :) Although they may just have another map format, where they placed all of those map bits together with portals or something to specify visibility regions.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-03-20 01:17:18
From what I have seen of the data so far, there is only a single BBox before the list of vertices, no kind of other visibility data at all. They probably used vanilla camera-frustrum checks for data strorage efficiency, or something.

I figured out the texture indices, and here are the results so far:

(http://img338.imageshack.us/img338/4136/ccls1b.th.jpg) (http://img338.imageshack.us/my.php?image=ccls1b.jpg) (http://img4.imageshack.us/img4/6780/ccls2b.th.jpg) (http://img4.imageshack.us/my.php?image=ccls2b.jpg) (http://img4.imageshack.us/img4/2585/ccls3b.th.jpg) (http://img4.imageshack.us/my.php?image=ccls3b.jpg)

Most texture-maps are still corrupt because of different interlacing modes, but the correct maps are now applied to the correct vertices.

There are still lots of missing polygons though.  :|
I still haven't discovered where the precise count of vertices are located, because the vertex-group "header" seems to be different from the characters so the PSP-GPU method doesn't work quite as expected.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: MrAdults on 2009-03-20 01:38:45
Ah, oh well. I was thinking of some of the bigger missions - particularly the Wutai complex inflitration one. In total that map had far more parts overlapping than they could have dealt with using exclusively frustum culling. But, I bet that level was made out of a bunch of these map geometry files, that is held together by another structure (which perhaps uses portals or something to isolate visibility sections).

About the verts: Since they have index values in there, the 32-bit vertex encoding value should be changed to reflect that. Have you found the location of that encoding value in these new files yet? It could be the key to decyphering the other vertices. If those verts are really type 1 except with the inclusion of index values, the exact 32-bit encoding value you should be looking for is 0x00000723 + (indexType<<11). indexType could be 1-3 depending on the index type.

As always, good luck. :)
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-03-20 15:49:59
Quote
indexType could be 1-3 depending on the index type.
This statement had me scratching my head for a few minutes  :lol:
I will look into the encoding value thing when i get home.

And there most likely will some sort of visibility information somewhere, there is a lot we dont know yet about these !-model files.
We will bump into them sooner or later  :-D
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Satoh on 2009-03-20 18:10:31
You two are made of epic... I can't describe it any better...
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: BlitzNCS on 2009-03-20 18:19:10
*seconds*
ya, they do win. you're pretty cool too though, X-dina!  :-P

by the way, i may have missed something here, but what exactly ARE the [&&] files?
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Satoh on 2009-03-20 19:14:28
I was wondering that myself...

It says type [&&] Model,but are they really models at all?
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-03-21 00:17:49
WOOHOO!!!
(http://img5.imageshack.us/img5/4943/cclc.th.jpg) (http://img5.imageshack.us/my.php?image=cclc.jpg)

All the location models now load correctly! (geometry anyway)

Although these models had a different encode value (0x000011f) and an extra 0x30 bytes to skip (the bounding-box data) the clincher was the "next-vertex-index" which was 6 and not 5 like the character models used.

Each Map (as loaded in the game) is a seperate file, in most cases humongous models with several textures.

Texture interpolation is still a problem though. The only reason why i have repetedly only shows these four locations is because they have almost a "perfect" set of decoded textures.
As soon as I figure out how to correctly decode the other textures, expect to see a new release!  :-D
(MrAdults might need to update his tool too, these are the same !-files after-all :wink:)

BTW
&&-files also contain model data and are very similar to these !-files. As to what models they would contain... beats me  :lol:
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Satoh on 2009-03-21 00:41:54
So the new version is up and can view field areas now? If so... awesome. (I'll find out either way soon enough)

EDIT: OK guessing you didn't upload the newest version.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Aurenasek116 on 2009-03-21 01:18:35
That's really interesting find, being able to view locations. Now if it was possible to extract them as models, props could be made and so on for other games. (I'd benefit from those so they could be used in JK3 maps :P)
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: ignitz on 2009-03-21 02:05:02
WOOOOO, it's not finished.... My GOD the people of Qhimm don't have limits.
 :evil:
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: MrAdults on 2009-03-21 02:54:23
Great work again, koral! :) I guess I might as well put proper support in for those files, too. I'll wait until you finish documenting your findings on the wiki, though, so I'm not re-doing the work you've done in finding the new offsets and data structures in these files.

I've actually been working on Silent Hill: Homecoming formats over the past few days in my spare time - I have their storage formats and eveything else figured out, and just got the models rendering properly the other day (though there are a few issues with the texture coordinates). So when that's done I'll jump back over to CC, and maybe we'll even have some Disssidia models to play with by then. ;)
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Satoh on 2009-03-21 04:38:15
speaking of using stuff from CC for other games, this is all perfect for a Turks game I was planning in my head... it would be an action RPG with stealth and platforming elements... (I guess you could say similar to devil may cry... but that's not my intention at all)

the player creates a character, name, weapon proficiency, and appearance, does a multi-part tutorial level (involving elements of every weapon, skill, and play style, with multiple possible ways to finish) to help them decide on a 'class' so to speak.

and then the game would be carried out in mission format similar to Crisis Core's but with more plot behind each mission...
[/stuff no one cares about]

I'm really amazed at how fast this has progressed... I remember the Chrono Cross people had what seemed to be several months and in all likelihood over a year invested in just getting the character models figured out... and there were a few people working on it. I helped out with that too actually... nothing more than educated guesses like I do now... I never did quite figure out how to read all that stuff in the hex... I could only get as far as the basic header...

anyway, I'm glad I came back to Qhimm when I did, or I'd have never found this mighty project.

EDIT: also, I'd say your viewer is plenty well user friendly.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Aurenasek116 on 2009-03-21 14:05:01
That's nice information about Dissidia there MrAdults, I wanted to suggest it after this project was finished because I thought it would be kind of not nice to ask for new "toys" when the current ones are still fresh. Would be nice to have support for it.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Satoh on 2009-03-22 00:50:51
That's nice information about Dissidia there MrAdults, I wanted to suggest it after this project was finished because I thought it would be kind of not nice to ask for new "toys" when the current ones are still fresh. Would be nice to have support for it.

I should point you in the direction of a topic koral started after a suggestion I made... some work with dissidia has been started already... I'll be right back with a link..

http://forums.qhimm.com/index.php?topic=8290.0
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: BlitzNCS on 2009-03-22 02:46:39
Hey Guys, got a treat for ya :D
remember in the dissidia thread i mentioned my friend?
Well,  he managed to get into the debug room for crisis core somehow :D
Here's Some screenshots:
(http://i182.photobucket.com/albums/x74/beefpwnage/CostadelSol.jpg)
(http://i182.photobucket.com/albums/x74/beefpwnage/DebugMenu.jpg)
(http://i182.photobucket.com/albums/x74/beefpwnage/JenovaRoom.jpg)
(http://i182.photobucket.com/albums/x74/beefpwnage/StrangeRoad.jpg)
(http://i182.photobucket.com/albums/x74/beefpwnage/TheLifestream.jpg)

Koral, perhaps You can get up those maps with your viewer?

And wow. Costa del sol. *uuber-nostalgia from original FF7 kicks in*
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Aurenasek116 on 2009-03-22 03:25:08
Thanks Dina but textures I've been able to extract quite a while ago, like 2 weeks after Dissidia release or so. It's the models that I'm interested in :P

That debug room is very interesting, is there any information on how to enter it?
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-03-22 16:51:25
Wow  :-o
Your friend must be really good!  :wink:

Those screens have got me interested in looking for it, and I believe all the answers seem to be in the first couple of .raw files.

File: 0001.raw is made up of 5 sections, 2 GT files and 3 MBD Files, and contains universal "events", menus and dialogues:

GT file 1 = "Now Loading" image
GT file 2 = "Pause" image

MBD 1:
Code: [Select]
<TEXT>Materia has leveled up!<END>
<TEXT>Nothing more to do, kupo!<END>
<TEXT>Load failed.<END>
<TEXT>Load canceled.<END>
<TEXT><CHOICE1>: Open<END>
<TEXT><CHOICE1>: Push<END>
<TEXT><CHOICE1>: Go up<END>
<TEXT><TEXT><CHOICE1>: Go down<END>
<TEXT><CHOICE1>: Use elevator<END>
<TEXT><CHOICE1>: Use gondola<END>
<TEXT><CHOICE1>: Rotate<END>
<TEXT><CHOICE1>: Read<END>
<TEXT><CHOICE1>: Move<END>
<TEXT><TEXT><CHOICE1>: Examine<END>

MBD 2:
Code: [Select]
<TEXT><VAR> obtained!<END>
<TEXT>Obtained <VAR> x <VAR>!<END>
<TEXT>Stole <VAR>!<END>
<TEXT>Stole <VAR> X <VAR>!<END>
<TEXT>Nothing to take!<END>
<TEXT>Couldn't steal!
<END><TEXT>Escaped!<END>
<TEXT>Can't escape!<END>

MBD 3: (the most interesting)
Code: [Select]
<TEXT>Use an item.<END>
<TEXT>Equip materia and accessories.<END>
<TEXT>Use materia.<END>
<TEXT>Fuse materia.<END>
<TEXT>Confirm learned DMW.<END>
<TEXT>Go on a mission.<END>
<TEXT>Abort mission.<END>
<TEXT>Check mail.<END>
<TEXT>Buy and sell items.<END>
<TEXT>Unused<END>
<TEXT>Equip materia and accessories.<END>
<TEXT>Sort materia and accessories.<END>
<TEXT>Unequip materia and accessories.<END>
<TEXT>Select strongest combinations.<END>
<TEXT>Equip for strongest physical attacks.<END>
<TEXT>Equip for strongest attack spells.<END>
<TEXT>Equip for best defense.<END>
<TEXT><TEXT>Use items.<END>
<TEXT>Select sorting method.<END>
<TEXT>Sort items automatically.<END>
<TEXT>Sort items manually.<END>
<TEXT>Use materia.<END>
<TEXT>Select sorting method.<END>
<TEXT>Sort materia automatically.<END>
<TEXT>Sort materia manually.<END>
<TEXT>Select materia for fusion.<END>
<TEXT>FUSE<END>
<TEXT>Yes<END>
<TEXT>No<END>
<TEXT>Select item for fusion.<END>
<TEXT>Unused<END>
<TEXT>Set quantity to use for fusion.<END>
<TEXT>Unused<END>
<TEXT>Proceed with fusion?<END>
<TEXT>Do you want to proceed with fusion?<END>
<TEXT>Fusion successful!<END>
<TEXT>Do you accept this mission?<END>
<TEXT>Do you want to abort this mission?<END>
<TEXT>Moved mail to <VAR> folder.<END>
<TEXT>Moved mail to <VAR> folder.<END>
<TEXT>Buy items and materia.<END>
<TEXT>Sell items.<END>
<TEXT>Buy<END>
<TEXT>Cancel<END>
<TEXT>Sell<END>
<TEXT>Sell items.<END>
<TEXT>Unused<END>
<TEXT>Convert materia to SP.<END>
<TEXT>Select shop.<END>
<TEXT>Shinra Building Shop<END>
<TEXT>Sector 8 - Materia Shop<END>
<TEXT>Sector 5 - Materia Shop<END>
<TEXT>Sector 6 - Accessory Shop<END>
<TEXT>Sector 7 - Shop<END>
<TEXT>Research Dept. QMC<END>
<TEXT>Research Dept. QMC @<END>
<TEXT>The Happy Turtle<END>
<TEXT>Wutai Secret Shop<END>
<TEXT>Gongaga Trading<END>
<TEXT>Nibel Accessories<END>
<TEXT>Junon Souvenirs<END>
<TEXT>Bone Village Commerce<END>
<TEXT>Mythril Mine Traders<END>
<TEXT>Network Shop Duo<END>
<TEXT>Network Shop Shade<END>
<TEXT>Network Shop Shade<END>
<TEXT>Open to all Shinra employees.<END>
<TEXT>Top sales in the world; plenty of useful selections.<END>
<TEXT>A well-established shop; traditional selections in stock.<END>
<TEXT>Wildly popular with all generations of shoppers.<END>
<TEXT>An underground outlet favored by anti-Shinra groups.<END>
<TEXT>For all members of the Turks to procure their equipment.<END>
<TEXT>Only for selected members of the Turks.<END>
<TEXT>A countrywide chain with its head office in Wutai.<END>
<TEXT>An illegal operation favored by Wutai remnants.<END>
<TEXT>Pioneered online shopping to compensate for its rural location.<END>
<TEXT>Made possible by the abundance of mako.<END>
<TEXT>A shop by the sea, popular with tourists.<END>
<TEXT>A shop in Bone Village where many rarities are dug up.<END>
<TEXT>A shop in the famous Mythril Mines.<END>
<TEXT>Lord Duo's online shop based in Gold Saucer.<END>
<TEXT>An online shop in an unknown location.<END>
<TEXT>An online shop in an unknown location.<END>
<TEXT>????<END>
<TEXT>Currently equipped!<END>
<TEXT>Not enough money!<END>
<TEXT>You can't carry any more!<END>
<TEXT>You have collected all DMW panels!You obtain a <VAR>!<END>
<TEXT>You have viewed all DMW scenes!You obtain a <VAR>!<END>
<TEXT>You have completed all missions!You obtain a <VAR>!<END>
<TEXT>Save<END>
<TEXT>Delete<END>
<TEXT>Return to title screen<END>


File: 0002.raw contained a GT file with the "FF7 10th Aniversary" logo.
Files: 0003-0005 (before the first video file) are unknown.
Files: 0006 onwards are video (PSMF) fies.

But if files 0001 and 0002 are anything to go by, then we can assume that:



So that leads me to two conclusions:
1. The Debug-Menu can be activated anytime from within the game (because it is a universal entity)
2. It only initiates during the game's boot-up process, otherwise it is skipped and becomes "disabled" altogether.

Any opinions, observations, speculations, treats, etc, etc anyone would like to share? :-)
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Satoh on 2009-03-22 19:20:30
One thing That has me stumped is... I didn't see the phrase "Where do you want to go?" in there.

I suggest looking for that specifically if you can... (This may mean writing a text searcher into the viewer *hinnnnt~* )

I've been thinking about what I worked on with the Chrono Cross group...

I wonder if it would be possible to recompile the game, with mods, into a CSO and have it play... (I've also been toying with the idea of trying to rip metal gear portable ops plus models... but I don't think anyone here would be interested in helping... at least no one with any skills...)


Still, find some way to do a search for that phrase, and those names. That may hold the key to accessing it.

EDIT: Found the Jenova room yet? I always wanted a closer look at her CC model...
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-03-23 17:55:35
Um, we all know where the debug menu is: File 0000.raw
I have already mentioned this on the Wiki too (http://wiki.qhimm.com/FF7:CC#Miscelenous_Findings), you must have missed it  :wink:

I was thinking, in the ISO we could maybe "swap-out" the MBD data from file 00002.raw (10th aniversary logo) and put in its place the debug-menu instead. But that would only work when running the game from a memory-stick.

The file data seems to be stored at 0x393200 in PSP RAM.
It may be possible to simply assign a different pointer address and force it to execute the data from debug-MBD instead?


I am getting a little distracted here, I need to finish the location-viewer stuff first
:lol:
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-03-29 00:09:31
An update here at long last!  :lol:

I have finally figured out how to load some of those location texture-maps. They were 4-bit (16-colour) palleted, with two pixels per byte (one per nibble).
I think Zande mentioned this a very very long time ago, my bad for not investigating sooner.

Most of those textures were 128x128 sized and loaded correctly, but many smaller ones (64x64 for example) dont, so I wont be releasing a new version yet.

But here are a bunch of screenies to tease you little:
(warning: huge images!)

(http://img24.imageshack.us/img24/2858/ccl3z.th.jpg) (http://img24.imageshack.us/my.php?image=ccl3z.jpg) (http://img27.imageshack.us/img27/4070/ccl4ban.th.jpg) (http://img27.imageshack.us/my.php?image=ccl4ban.jpg) (http://img9.imageshack.us/img9/3858/ccl6cos.th.jpg) (http://img9.imageshack.us/my.php?image=ccl6cos.jpg)
(http://img12.imageshack.us/img12/5944/ccl10slums.th.jpg) (http://img12.imageshack.us/my.php?image=ccl10slums.jpg) (http://img14.imageshack.us/img14/6286/ccl12mid.th.jpg) (http://img14.imageshack.us/my.php?image=ccl12mid.jpg) (http://img16.imageshack.us/img16/3970/ccl15jun.th.jpg) (http://img16.imageshack.us/my.php?image=ccl15jun.jpg)
(http://img19.imageshack.us/img19/8740/ccl17.th.jpg) (http://img19.imageshack.us/my.php?image=ccl17.jpg) (http://img22.imageshack.us/img22/4602/ccl19midout.th.jpg) (http://img22.imageshack.us/my.php?image=ccl19midout.jpg) (http://img24.imageshack.us/img24/3115/ccl21jen.th.jpg) (http://img24.imageshack.us/my.php?image=ccl21jen.jpg)
(http://img19.imageshack.us/img19/3259/ccl22lib.th.jpg) (http://img19.imageshack.us/my.php?image=ccl22lib.jpg) (http://img22.imageshack.us/img22/7976/ccljun.th.jpg) (http://img22.imageshack.us/my.php?image=ccljun.jpg) (http://img24.imageshack.us/img24/413/ccs18nib.th.jpg) (http://img24.imageshack.us/my.php?image=ccs18nib.jpg)

They look uglier than the actual game because alpha-blending is disabled, and as I had mentioned earlier many smaller textures are still not correctly loaded.

But talk about nostalgia  :|

These models have incredible amount of detail, and I cant help but wonder if these models could be used to update the backgrounds for FF7 in some way.
Aali's new FF7-graphics driver is making rapid progress too
 :wink:
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: MrAdults on 2009-03-29 10:39:41
Awesome! I expected the models to be more broken up into chunks than they are. I wonder what they did in-game for lighting. I'm having a hard time recalling the look of the maps, but maybe there are vertex colors embedded in the verts? Or maybe they used lightmaps, but they'd have to have another set of texture coordinates somewhere.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Aurenasek116 on 2009-03-29 12:35:15
Whatever they used for lighting and special effects in Crisis Core, they used it again in Dissidia. Just look at models of levels from Dissidia and then at any gameplay with that level, ingame it's looking a lot better.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: sorayahya on 2009-04-06 14:58:17
Hi how did you get the maps plz tell me
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Prince Lex on 2009-04-06 15:04:34
Hi how did you get the maps plz tell me

Read through the rest of the posts.

And Koral... that's amazing. Seriously.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: sorayahya on 2009-04-06 15:14:12
I read all but I still don't get it :?
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-04-06 15:34:37
sorayahya, (and everyone else who might be looking for map-models)

THE VIEWER [v0.40] DOES NOT VIEW OR EXPORT MAP MODELS!!!

It is still something I am sorting out right now, because the program crashes everytime I try to change from viewing one Location-map model to something else.


But I have fixed the ugliness issue:
(this will be in v0.50, not released yet)

(http://img7.imageshack.us/img7/218/ccnib.jpg) (http://img7.imageshack.us/my.php?image=ccnib.jpg) (http://img8.imageshack.us/img8/4020/cccosta.jpg) (http://img8.imageshack.us/my.php?image=cccosta.jpg)

They have used vertex-colours extensively for shading, it could even be reffered to as an ambient-occlusion effect.
Top-notch stuff from SE!  :-D
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: sorayahya on 2009-04-06 15:37:09
but you still didn't answer my question How can I at lest viewer them
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-04-06 15:37:51
I realised i wasn't clear in my post and edited it.

Simple answer:
YOU CANT! ONLY I CAN (because I made it)  :lol:
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: sorayahya on 2009-04-06 15:49:56
My apologys
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-04-06 16:10:35
It's cool, dont worry about it  :-D

I will release it as soon as I can stop the program crashing, it is still quite buggy.

And you can also see from the screenshots, some texture-maps are still messed up.
Annoying interlacing quirks  :|
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: MrAdults on 2009-04-06 16:49:34
Awesome, looks great with the vertex colors. In case it's any help with the interlacing issues, it seems like all GIM files that use the same 16x8 tile ordering use the same number of columns regardless of image dimensions, and there is nothing in the header but a 1 or a 0 to indicate if the image is tiled, and the only exception is the 4-bit images (which have 32x8 tiles, probably because the 16x8 is a byte read order optimization thing or something).

Although, I have never actually come across a GIM that is non-paletted RGBA and in "fast" pixel order mode, even though that is a valid format for a GIM. In theory, those could actually have 4x8 tiles.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Cyberman on 2009-04-06 23:01:51
sorayahya, (and everyone else who might be looking for map-models)

THE VIEWER [v0.40] DOES NOT VIEW OR EXPORT MAP MODELS!!!

It is still something I am sorting out right now, because the program crashes everytime I try to change from viewing one Location-map model to something else.


But I have fixed the ugliness issue:
(this will be in v0.50, not released yet)


They have used vertex-colours extensively for shading, it could even be reffered to as an ambient-occlusion effect.
Top-notch stuff from SE!  :-D
For location changes you might have several problems. Are you getting Exception dialogs? If not you may want to add several exception makers and look for bad states within the objects you are using.  IE trying to use NULL pointers. Not deallocating something. 

It sounds to me that you are not properly deconstructing the first scene you loaded. IE you need to clear out all your data so that you can load the new scene. That appears to not be happening correctly.  Be sure you are handling object disposal properly.

Cyb
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-04-07 09:54:01
Cyb, that is precisely the problem: I can see that something isn't being cleaned up correctly (or in the correct order) because this crash is a classic example of what happens when trying to dereference a pointer to a variable which has had its memory cleared.
For rapid-development reasons, I have used Irrlicht predominantly for everything in the viewer, from texture-creation to mesh-rendering. Something is going wrong with the Irrlicht side of things, I am sure of it.

The strange thing is, the viewer has absolutely no problems switching between the Character models rapidly. Which suggests maybe that there is an issue with bulk-memory allocation/deallocation going on with the map-models specifically.

I may just have to switch to static pre-allocated arrays, or do an extra initial pass to determine memory requirements and deallocate/allocate prior to parsing the data. :-)


MrAdults, the textures I am specifically having problems with are 64xN sized ones (where N is any value other than 64). And these are all 4-bit images, presumably with 32x8 tiles because using that value, the interlacing works fine with 64x64 textures.


And I forgot to mention this before:

Notice the big tree blocking the entrance to Nibelheim? (strange, no  :lol:)
Those must be independant models which get positioned by the game at sometime, but are initially planted at the origin (0,0,0).
It is also the same in the Church map, where the door (leading out) is also initially loaded at the origin (on top of the flower-bed actually, standing upright).

Perhaps their precise position values are what is contained in [unknown-chunk-1] of these [!]-model files?
We never did find out what that chunk contained  :wink:
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: SOLDIERBOY on 2009-04-07 18:52:51
Hi Everyone i'm new  if you can't tell lol anyways I got these models and started  working on Zack but I ran into a problem.. If anyone can please help me I would be very thankful since he is a project  i'm still new to 3D max  and I got alot of it down. I know I am a Noob. if you can help be great The problem is his legs  when i went to test him out his legs for some reason are stuck together and I have tried to readjust my mesh but nothing is working I tried to look for something that could help me but there was nothing out there been stuck on him for like 2 days now I've seen people that were able to do it so what am I doing wrong? :|
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: MrAdults on 2009-04-08 17:22:56
Ahh, w != h images. I don't know if I ever ran into any of those. I can see why they would be problematic. :) There must be something special about them, it's probably the same for GIM files too.

SOLDIERBOY: Sounds like the model you're using hasn't been weighted. You could use the weight/bone export of mesh2rdm and import to Max as .smd using a third party importer, or weight the model yourself. That would be a helpful thing to know anyway if you are setting out to become a modeler. Here are some links:

http://www.google.com/search?source=ig&hl=en&rlz=1G1GGLQ_ENUS316&=&q=max+vertex+skinning+tutorial&btnG=Google+Search&aq=f&oq=

Take your pick. :)
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: SOLDIERBOY on 2009-04-08 17:38:46
Thank You Very  I'm new to this whole thing but I've been waiting for these models. to come out you guys are awsome for getting this I'll try my best! :lol:
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: duoblade on 2009-04-08 18:43:57
(http://i51.photobucket.com/albums/f373/final16fantasy/SoldierFirstClassHumeMale.jpg)

Thanks for making such an awesome tool.  Don't know if there is any FFXI players on here.... but I used your tool to extract textures to make this outfit for Hume Male on FFXI. Keep up the great work ^^
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: SOLDIERBOY on 2009-04-08 18:49:44
How do you use the mesh2rdm? I'm alittle airheaded some times lol I bet  it's there i'm just not seeing how it works I can look at the models but i can't do anything else xD i'm so lame I know.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-04-08 21:59:19
duoblade, that looks cool  :lol:

SOLDIERBOY
You aren't the first person to ask about using mesh2rdm, so I have added a little compendium of useful pointers to using the tool in my first post here: http://forums.qhimm.com/index.php?topic=8290 (scroll down)

And please dont post in both this topic and the Dissidia one about the same problem, it makes things a little confusing.
Just pick one and post there.  :wink:
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: SOLDIERBOY on 2009-04-09 02:33:40
Sorry about that I kinda got mix up when I was posting. maybe i should wake up alittle more befor getting on the computer.  :-)but thank you very much. we did get to work now i am just trying to figure out how to get it to work in my max  he imports fine and everything but cant get anything to  stay on the mesh.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Karlislie on 2009-04-09 03:09:38
Id like to post some ideas and information.

Has anyone noticed that in some files that the phrase "leaf" is always mentioned?
You can see it in my search.
Download here: http://www.mediafire.com/download.php?j4wdmmqnzox

Also I assume it isnt possible to play animations from the files?
From what I recall from yaz0r he said that he compared the files that have animation from the ones that didn't or something and was able to animate them, how ever I'm not 100% sure.
Maybe the same could be done for CC?

On another note;
How exactly are the MBD files scripted? Whats the proper syntax to script new events for a possible extended play?
Maybe a save/edit function could be added.

And how can the files be pac ked back to pkg?
Or maybe they can be read from a data folder?
That way new changes dont require repacking.

Will the feature for auto assigning the textures upon export be added?

And will it be possible to view the high quality heads and other parts?
You've come  so far why not go the extra mile?

In the ATEL files there is an order to making events?
Like Play BGM first, move char, dialogue???

And lastly has anyone looked into the elf binary? (boot.bin)
At 001a2bc0h you can see 41 65 72 69 74 68 Which translates to Aerith
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-04-09 15:07:17
Welcome to Quimm's!
A lot of questions you have there, the answer to most of them is a solid "maybe", possibly a "no".  :wink:
But I will try to be a little more specific, clumping your questions for easier reading/answering.

Also I assume it isnt possible to play animations from the files?
From what I recall from yaz0r he said that he compared the files that have animation from the ones that didn't or something and was able to animate them, how ever I'm not 100% sure.
Maybe the same could be done for CC?
I cant care less what Mr. yaz0r had to say (unless he posts here to tell me about it  :-P)
Animations are there in the files, but I am not interested in them right now. Maybe someday, but not anytime soon  :-D

Quote
Has anyone noticed that in some files that the phrase "leaf" is always mentioned?
How exactly are the MBD files scripted? Whats the proper syntax to script new events for a possible extended play?
Maybe a save/edit function could be added.
And how can the files be pac ked back to pkg?
Or maybe they can be read from a data folder? That way new changes dont require repacking.
In the ATEL files there is an order to making events? Like Play BGM first, move char, dialogue???
I noticed those "Leafs" a long time ago too, most likely they represent some manual heirarchy for the Event models, perhaps for culling purposes or event sequencing.

By the sound of your other questions, it seems you want to Mod the game (and want to know more about how Event scripting works). I have no problem with that, maybe other people would be interested in it too.  :-)
But I wont be looking into event-stuff much though (now that I can extract characters, and soon maps), and my "viewer" tool definitly wont ever have the options to allow re-writing stored data.

So you will either have to write your own tool, or wait for someone else to do it, but it wont be me :lol:

Quote
Will the feature for auto assigning the textures upon export be added?
That is a quirk with the export option, and I need to correct it soon.  :oops:
It makes it impossible to export location-map models  :-P

Quote
And will it be possible to view the high quality heads and other parts?
um, you can already view them :-)
Try exporting them out, you wont notice the difference in the viewer, but you will when you look at the wire-mesh in any 3D proggy

Quote
You've come  so far why not go the extra mile?
This really bugged me.
Not all of us are as gifted programmers/hackers as MrAdults or yaz0r, so to "go that extra mile" for those guys may not be of much consequence for them, but generally means a lot of extra hours or days of coding for me.
I am definitly not in the same league as them, I have zero qualifications in programming, and everything I do, I do because I want to.
I like 3d-modelling more than programming actually, so the "extra mile" would be spent there :-P

Quote
And lastly has anyone looked into the elf binary? (boot.bin)
At 001a2bc0h you can see 41 65 72 69 74 68 Which translates to Aerith
I haven't looked myself, I dont know about other people.


If you have any more "information" then dont fret to post here.
They might not interest me much, but other Quimm-ers may be extremely interested to listen to what you have to say
:wink:
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: MrAdults on 2009-04-09 17:20:00
Also I assume it isnt possible to play animations from the files?
From what I recall from yaz0r he said that he compared the files that have animation from the ones that didn't or something and was able to animate them, how ever I'm not 100% sure.
Maybe the same could be done for CC?
.ffb and .ffa are not at all related to Crisis Core models. Additionally, at least in the cases I observed, the animations aren't in the same raw files as the model data. It's certainly find-able (maybe koral already knows which files it's in?) but it isn't really a trivial matter to put all the pieces together, particularly pertaining to automatically attaching mesh files to animation/skeleton files without some manner of hardcoding (bone counts and hierarchy matches would be rather haphazard).

And will it be possible to view the high quality heads and other parts?
As koral said, this should already be possible in both of our viewers. I noticed higher-quality versions of a few characters in there (like Genesis). You just have to find them.

You've come  so far why not go the extra mile?
I'm not sure if this was just in reference to high-quality parts (which you can already view) or the sum of all of your questions/requests here. But if it was in reference to the sum, that isn't really an extra mile, it's more like doubling the length of the journey. :)
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Karlislie on 2009-04-09 18:15:51
Hi there,

Thanks for replying koral and MrAdults.

I wasn't referencing to the .ffa/.ffb itself, I meant the method on which he was able to animate them.
But yeah the fact that the models are viewable is a major accomplishment, the animations are not really important at the moment.

Adding extended parts to the game is my intention.

The reason I brought up the hi quality data is because koral saaid here:

About those "split-textures", NeoCloudstrife is right they are just higher-quality textures (and models) of the head and torso split like this to make it more efficient. No point loading a super high-quality model if only the head will be shown, right?

I didn't know that they were actually viewable.

I'll post some stuff able the elf executables later.

And you can actually repack the files, its a tool called Xpert2,
Here is the plugin for Crisis Core.

Also in the file is the LBA.

Download here: http://www.mediafire.com/download.php?j5tnduxoemd

G's extractor seems to be a better quality from what I saw, I'll send him a PM.

Edit: Sorry if I jump from item to item, I'm just trying to get what I need to say out.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: aljnx on 2009-04-10 15:37:03
I just wanted to add that the background music (SSCF audio files) can be converted to wav or mp3 without using the psp. :-) These are the SSCF audio files right? You can convert them using this program here:
http://www.marcnetsystem.co.uk/himdrenderer054.zip
I know it's a Hi-md renderer, but it converts the background music perfectly. Interesting huh?
The voices ( RIFF I think)can be converted by simply changing the extension to .wav and installing the ATRAC3 codec. The wiki says the only way to decode the SSCF audio is using the PSP hardware and AT3 decoder, though :-D. 
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-04-11 09:47:00
Thankyou Karlislie and aljnx for those bits of (informative) information!
Time to update the Wiki again!  :-D
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: sorayahya on 2009-04-11 11:33:08
How much left till you will release v0.50 I rely need to get the maps is there any way I can help you to release it?
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Karlislie on 2009-04-11 18:50:12
sorayahya, Don't rush koral, its worth the wait.
I know when she releases the new version all of us will love it.

aljnx, thanks for the tip, I'v been wondering how to open them.
I wonder if there is a codec for it...

Here is a listing of all SSCF files.
Download it here: http://www.mediafire.com/download.php?jime4mmzydn

The syntax follows:

0   SSCF     Ã–  É                       @   @   D¬d    ÿ_Ê              @   @   D¬d    ÿ_Ê              €+      Å“n(    ÿ_Ê               0  p   Ã°U-    ÿ_Ê              C  0)  À]d    ÿ_Ê              Àl   :   NP (  ÿ_Ê              `§  à   "Vd    ÿ_Ê   
 01351.raw  -  72.176 bytes  -  Mon, 06.04.09 at 17:15  -  C:\Documents and Settings\*\**

The first 0 is the line number SSCF appeared on (this matters because if its on a line number like 1.160.784 it probably isn't one)
01351.raw is obviously the file name

Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: aljnx on 2009-04-12 06:02:14
  errrmm.... Are my files named differently or something? I'm confused. Mine are named file000000, file000001, ect. :? I extracted them using Xpert 2. Could someone clear this up?
  Also, I opened up the background files in a hex editor and it begins with RIFF.  :oops: I just assumed they were SSCF since they didn't open like the other sound files.  Sorry for any confusion. Like I said before, only the Hi-MD renderer opens the background music.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Karlislie on 2009-04-12 18:26:14
You used Xpert 2, that should only be used for repacking the files.
In my option it doesn't extract all of the files properly, not to mention all of the extra Zero byte files.

Also not to mention it names all of the files differently, the easy thing for everyone to be on the same page is to extract the files by using G's extractor.

All of the files that we are referencing to were extracted with G's extractor get it here: http://superg.org.ua/ff7cc_extractor.tar.bz2

I haven't opened the SSCF files yet, I'm looking for a way to open them, not convert them.
Anyone have some ideas on that?

Karlislie


Edit:

Has anyone noticed that the monster data hasent been seen yet?

I did some comparing and I noticed that Magnitude 8 appeared in seven files.
In file

1474
1708
1775
1776
1777
1805
1909

So I will be using 1474;

I was using koral's viewer and splitting it apart, in to a total of 52 some chunks, so I did another look for Magnitude 8 and It is in 1474 Chunk #12 (Remember zero is counted)
Its at offset: 166784
So what does chunk twelve contain....?

Apparently it can be viewed in mesh2rdm.exe but its only a black screen, but thats not all something else cought my eye.
When its viewed in koral's
Theres four files

? 1
? 2
image (type -T-) for some reason it cant be viewed
? 3

But that's not all.
Open it in a Hex editor ( I use UltraEdit v14 )
At the top you can see its a && File

Go to 000005b0h and theres an ATEL header. (41 74 65 6C)
Apparently there are Atel files within Atel files.
And apparently this file contains monster information on Vajradhara Rakshasa (go to 00006be0h the last line)
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-04-13 12:22:28
I am caught up in something right now and I might not get much free time with this project for the next couple of weeks, so I have uploaded the new version of the viewer so you check out the map-models in the mean time.

The crash was due to something with the texture-loading and/or creation routines, so to get around it I had to disable textures altogether for this release.

The program now has a name and an identity: The RINOA File Viewer :wink:

and you can download the latest version of it here (v0.5 pre-release alpha):
http://www.mediafire.com/?lhdwztmh2xn


Notes:


I will continue working on this when I get time, but it wont be for a while.
But I wanted you to all know I have not given up yet :wink:

Karlislie, I havent read your post fully, but it sounds like some interesting stuff you have discovered.
I will look into it sometime later.

 :-D
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: BlitzNCS on 2009-04-13 14:39:23
Thanks a lot, Koral, and nice new name :D
However, the viewer doesn't seem to be able to view map models. it is supposed to, right?
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-04-13 16:14:08
It should work, have you tried selecting the samples?

(http://img26.imageshack.us/img26/8353/rscreen.jpg) (http://img26.imageshack.us/my.php?image=rscreen.jpg)

It looks ugly without textures, but it's the Banora Outskirts map and you can see the vertex-shading they have used.

RINOA uses DX9 permanently now, so it should work fine on vista and everything.
Could anyone confirm for me if the viewer works properly for them? I know it works for me, but I may have done something weird again  :oops:
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: ultima espio on 2009-04-13 16:23:21
It works fine for me.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: BlitzNCS on 2009-04-13 16:49:32
ah, i remember now. I used to have aproblem where only your samples would work, and my files wouldn't, even though they were the same.... i'll have to try and remember how we sorted that out...
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: sorayahya on 2009-04-13 16:59:40
nice now I can continue my work
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: MrAdults on 2009-04-13 19:31:57
Great work again, koral. :) I'm looking forward to browsing through some of the locations.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Lucleonhart on 2009-04-15 11:00:58
I'm gonna check this out, too... Downloaded it two times... Please update first post with new version. :)
Thanks for your awesome work! :)
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: squallff8 on 2009-04-16 20:09:15
2 Koral
BTW is it possible to rip Bike model from exebition floor in Shin-ra building? And use it in FF7 bike level? :?
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Satoh on 2009-04-18 01:55:54
If you'll pardon me for answering...

Surely it can be ripped, as for using it in the bike minigame, I can't say with any certainty... I've never seen anyone specifically replace that model. (By this I mean I myself haven't seen it. I do not imply anything about whether or not it can be done.)


*Exhibition.

Ahh... every time I come through Tech-Related I get a bit sad about my inability to rip things like some of the rest of you... I have so many games I want investigated, but no time, or skill, necessary.

Koral and MrAdults specifically, you two are my model decoding heroes. I've never seen anyone else get so much done so fast. Is there a secret you have? Like being aliens?
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: squallff8 on 2009-04-18 22:41:09
If you'll pardon me for answering...

Surely it can be ripped, as for using it in the bike minigame, I can't say with any certainty... I've never seen anyone specifically replace that model. (By this I mean I myself haven't seen it. I do not imply anything about whether or not it can be done.)


I saw bike model by my self(when tried Kimera program) in minigame LGP archive.I'll try to find it someday later.

Actually i wanted to replace it with THIS model from PSX FFVII International Bonus Disc:
(http://i300.photobucket.com/albums/nn15/squallff8_2008/snap2.jpg)
(http://i300.photobucket.com/albums/nn15/squallff8_2008/snap4.jpg)
 but looks like there no PSX model ripper/viewer :cry:
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-04-18 23:43:15
Sorry about the half-hearted releases, but my life is really touch-and-go at the moment so I can only work on it at random spurts any chance I get  :|

I have fixed the issue with the textures now, so here is the new version of the RINOA viewer (v0.5 pre-release beta): http://www.mediafire.com/?dolini4ljyg

So now it is possible to flick through all the map-models and character models, complete with textures.
But there are still some major issues with the interlacing, and random "noise" also seems to have crept in, resulting in little black (or odd coloured) pixels littering the scene.

Notes:
  • No export options (it uses DX9 if you really want to rip them)
  • Press SPACEBAR to toggle between 1st-person camera and overview camera
  • G and H keys toggle file-select-window visibility
.
I had also forgotten to mention that [GT], [ATEL] and [MBD] files are not currently viewable in this new viewer tool (hence the beta status), I will get around to it later.

And there are some maps which never load, and simply crash the viewer.
Those Blacklisted files are:
.
Code: [Select]
2016
2024
2027
2037
2045
2062
2063
2078
2105
2138



squallff8
the vehicles are part of the Shinra-museum(ground floor) map, so probably can be ripped if you extract the entire map.

I dont have the FF7 Bonus disc so I cant look into those nice models you have shown us, but if you (or anyone) could upload some of the data files then I might check it out.


x-dina
dont forget the legendary Quimm and his band of merry FF7 (de)coders!
Without those awesome guys, we wouldn't be here discussing this stuff!  :-D
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Karlislie on 2009-04-18 23:48:18
In 00004 there is a pattern,

t the beginning at column 9 there is an increase when scrolling down
at 00006320g to 00006350g there is a sequence from efgh to u

I use Ultraedit...

By the way, in the wiki there is no information on the ATEL and MBD docs, could anyone post them?

Edit:

Also in the 8703 there is this pattern;

      0 @      4@      @@
    1@       5@      A@
  2@       6@      B@
3@      7@      C@
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-04-19 00:20:04
Karlislie check the first post, that is where I had initially dumped all my findings before I had received access to the Wiki.
Everything there isn't in the wiki, and vice-versa.
(in other words, all the known ATEL and MBD findings should still be there)


Sorry for bumping, but incase my previous post got ignored  :lol:

I have released a new version of RINOA (v0.5beta) which can show fully textured map-models.
For more info, check link: http://forums.qhimm.com/index.php?topic=8163.msg101036#msg101036
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: BlitzNCS on 2009-04-19 13:49:22
Sweet, but I can't load my own maps, but oddly i can load my own characters. only map i can load is 02109, banora, the one that came with the RAR...
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Karlislie on 2009-04-19 16:40:54
As mentioned in my post earlier toy can see in 00004 there is a pattern(http://i141.photobucket.com/albums/r56/Jaimion/00004.png)

And in this screenshot you can see another alphanumeric pattern(http://i141.photobucket.com/albums/r56/Jaimion/th_08703.png) (http://s141.photobucket.com/albums/r56/Jaimion/?action=view&current=08703.png)

By the way koral, would you mind sharing Rinoa's sourcecode? I know I sure would love to develop it further.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-04-20 00:06:41
Sorry Karlislie, but RINOA will remain closed source as my own personal project  :wink:
But all the information (as we have discovered) is out in the open, so you shouldn't have difficultly writing your own tool(s) for specific purposes.

Those screens look like [MBD] scripts, ignitz posted the codes for them here: http://forums.qhimm.com/index.php?topic=8163.msg98499#msg98499

But they could just as easily be numbered indices or offsets or something else. :-P
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Karlislie on 2009-04-20 00:12:43
hello koral  :lol:

Thanks for the link and for answering, never hurts to ask   :-)

With that post almost of the files are documented except for [ATEL] itself....

Could you post how to open [ATEL]? or how those bytes, dwords on [ATEL]  :?
Since some of the games interesting content seems to inside them :-P
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: MrAdults on 2009-04-20 22:31:54
*Exhibition.

Ahh... every time I come through Tech-Related I get a bit sad about my inability to rip things like some of the rest of you... I have so many games I want investigated, but no time, or skill, necessary.

Koral and MrAdults specifically, you two are my model decoding heroes. I've never seen anyone else get so much done so fast. Is there a secret you have? Like being aliens?
I used to wonder how people did it, too, back in my early programming days. :) There's no real trick, it's just a matter of having a wide base of knowledge to narrow down the possibilities. If you know what you're looking for before you start looking, you'll find it faster. On the other hand, you can't be too focussed on a single possibility, or you'll waste a lot of time looking for the wrong thing.

Some practical examples on what to look for in terms of vertex or triangle data (assuming your data set is not compressed/encrypted to start out with, which is another different set of issues):
Usually you can find triangle lists by searching a file in 1-byte-offsets for shorts or ints backed up against each other, all within a specific sanity range (where the maximum value would be the highest vertex index). If that fails (triangle indices may not be back-to-back) you can can try searching for specific geometry bits in different pattern orders, like groups of 3/4 indices. The list goes on into all kinds of different and weird/more obscure indexing modes.

Vertices are usually trickier. If UV's or positions are in floating point, it's pretty easy. Just search for valid floating point value ranges in a specific stride, or back-to-back in the case that each vertex component is in its own array. Bytes, shorts, and ints, on the other hand, can blend in with all kinds of data when utilizing their full ranges. In that case, you can try some patterns based on known index values, or if you don't have known index values, you can check value ranges and failures to comply with those value ranges outside of a given stride. If all of that fails, you usually have to start manually looking through the hex and interpreting values, and just trying to find patterns and/or consistently recognizable values in the interpreted data. The biggest thing to know, regarding the whole picture, is that the more pieces of a file you document, the easier it becomes to narrow down the rest. Since if you already know what some piece of data is, it can't be the other piece of data you're looking for. :)

Then outside of scanning for raw values, you have cases where there is some obvious header data with offsets to specific pieces of data (like the Crisis Core models), or cases where the file is broken up into chunks with a common chunk header, like the Dissidia/GMO models. Those things are pretty easily visible by interpreting the data as raw shorts/ints/etc. and just taking a look at the kind of data that's at the offsets (finding offsets is, I guess, mostly a matter of intuition and trial-and-error - if it's in a storage type that can hold up to the filesize, and the value is less than the number of bytes in the file, then maybe it's an offset :)). If you're lucky enough to have that kind of data at your disposal, it's much easier to interpret each specific chunk/offset of the file for a specific type of data, rather than having to bruteforce through the whole thing. On the other hand, there are also much more complicated methods of breaking files up into sections, and games will often have complete sub-filesystems within files for given types of files, or bits of files that are compressed, or all kinds of other things. Dealing with all of the scenarios just requires a comprehensive base of knowledge, so it's one of those things that you just get better at doing with the more you learn about storage techniques and other areas of development (for mesh data specifically, knowledge of rendering, more-so of rendering on the model's target platform(s), is most useful).

I guess I'm getting a bit off-topic there. I just wanted to take the opportunity to de-mystify the process a little bit for anyone interested. :)
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-04-20 22:49:21
Mr Adults
Your reply is surely worth wikifying somewhere!  :lol:
I could never explain such things as clearly and precisely as you do, I tend to just speak (type) my mind, never becoming too formal when posting stuff.

My mantra is more along the lines of "trial and error", logically filtering through all possibitilies of the data (as I understand it) until either I reach a conclusion asto its purpose, or I ignore it altogether.
So far I have been lucky, but with Ehrgeiz I realised my methodology isn't failsafe  :-D


Karlislie
I dont know anything about the [ATEL] files really  :-P
All I found out was that if I scanned the first 8-bytes every 16-bytes, I would come across known file-types eventually (such as MBDs and SSCFs).

The MBDs are the most interesting so far, because they contain the game scripts and are organised in such a way that we know what kind of data might be near it.
For example, during a cut-scene event, an NPC might say something and perform a specific animation, so we can find the precise MBD which shows us the file and the location where that dialogue was said, and we can look around to find the other information related to that event.

There would be camera information somewhere too!

Come to think of it, maybe those [&&] files were actually Animation data?!
There were hundreds of them mixed in with the ATEL files!!
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Karlislie on 2009-04-21 22:48:48
Well I meant the code about ATEl, because RINOA must have some sort of code that allows it to understand and see what the ATEL contains, I was hoping that you could post that snippet...
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-04-21 23:28:22
I was talking about [ATEL] files, but I dont know anything about it so I ended-up talking about MBDs and animations :lol:

I dont know how much you know about c++, but a minimalistic program to scan an ATEL file for MBD chunks would be something like this:

Code: [Select]
#include <stdlib.h>
#include <stdio.h>
#include <fstream>
#include <iostream>

void main()
{
// Open the file
std::ifstream *File = new std::ifstream( "ATEL_File.raw", std::ios::binary);


// Read bytes until End of File
while( !File->eof() )
{
// read in 16 bytes of this line
unsigned char c[16] = {0};
File->read( (char *)(c), sizeof( unsigned char ) * 16 );

// check for [MBD] file start by comparing first three bytes
if( (c[0] == 0x4d) && (c[1] == 0x42) && (c[2] == 0x44) )
{
printf( "found [MBD] at offset: %ld", unsigned int(File->tellg()) - 16 );
}
}

// Close and Delete
File->close();
delete File;
}

// Job done!!  ^_^

I just wrote this quickly right now from the top of my head, so it may not compile or work as expected.  :oops:

But all the essential functions are there, from reading in a file, recursing through its data, then closing it again.
What you do with that data (and how you show it in a meaningful way to the person running it) is something I cant really help you with.  :-)
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Karlislie on 2009-04-21 23:39:36
Thanks koral! :-D

By the way I think the && files are models+animations, they can be viewed in RINOA.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-04-21 23:51:16
I thought so!  :-o
Thankyou for the news, those && files are now number one on my hitlist.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Karlislie on 2009-04-22 00:33:23
hey koral,

A good && to try is 1482 chunk# 10 (remember 0 is counted)
What 1482 is, a treasure chest.

Since its only 37KB it might be easy, plus it has to have an animation since Zack opens them.

Edit: if you dont want to try that one use 1995, its a Cactar, the cactus. :-P
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: MrAdults on 2009-04-23 02:38:20
Oh, the && files are basically !! with animations, then? That's quite convenient! :) Let me know if you'd like me to have a look at any particular file/piece of data, koral. I'll otherwise probably just continue doing what I'm doing. ;) I do still plan to go back and add location model support, and now perhaps "proper" skeleton support, to mesh2rdm, though, once all the findings are wikified.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-04-23 23:27:29
I seem to have gotten distracted again  :lol:

this time with FF8 models!

So far I have only got as far as getting mch (field) models to render correctly with RINOA. The skeletal heirarchy is intact, but the joint positions are still pretty much AWOL.
Worst-comes-to-worst, I will just guess the positions for some of the "high-poly" field model joints and use text-files to parse them in. That would also make it easy for anyone to fix (or even manually pose) those models before exporting them out.

Battle-models are still iffy, nobody has yet gone the full distance to analyse and document their structures.
I hope I will be able to do it someday! The wiki could do with a new FF8 section too  :-D



Karlislie: I had a look at the files you mentioned, and there seems to be nothing "out of the ordinary".
I have added ATEL file parsing support back into RINOA, and I realised that I did know more about ATEL files which I forgot to post about:

The complete list of ATEL sub-chunks (embedded-files) which I currently know of are:
Code: [Select]
TEX
GT
!
SSCF
MBD
FEP
MDL
VTL
ANM
&&

TEX, GT, !, SSCF and MBD are perfectly viewable and understood, but the other 5 still open to investigation.

And if the names are anything to go by, then it is highly probably that the ANM chunks would contain some sort of Animation data, whether it be for simple Treasure-chest opening, or for skeletal NPC animations.

Which puts && files onto the back-seat for now  :wink:



Mr Adults
You dont have to get involved just yet, we need to find those animations first  :lol:
Which reminds me, I have yet to complete the wiki entry for the exp-models... yet another thing stuck somewhere in my todo list  :-P
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Karlislie on 2009-04-24 01:29:53
I thought the && files were different since they are used in battles and use animations, how much different are they from the ! models?
By the way, if you were to experiment and try to get the animations, wouldn't it be easier to use a very small ! or &&  file? and compare 1 or 2 other smaller files from there?

And have you noticed that in the ATEL files there is a GT image at the top? Perhaps that is the map file for that specific place where the event is happening? (You have to chunk it, then restart RINOA to see it, it looks garbled abit)
And if you were to take a look at an event, (have you noticed that the actual event data is ALWAYS the last MBD, not the first 1st or 2nd?)

Anyway, not to get sidetracked, lets take 01473 as an example,

<TEXT>
Enemy attack! It's SOLDIER!
Don't let him get through!
<END>

What happened to the part that goes here? What calls the battles? What calls the camera position?
What is defining that? Is there some part of the MBD that isn't being properly displayed, or is missing?

<TEXT>
Were you able to get inside the fortress?
<END>
<TEXT>
Piece of cake! I could have done it blindfolded!
<END>
<TEXT>
Obtained <VAR> gil!
<END>

But yes there are lots of unanswered questions, I have a number on my list.
Perhaps FEP could be the camera data?

Or MDL is for MODEL?
And ANM is ANIMATION?

Ah maybe MDL + ANM are used at the same time???

Could there be something, to match and compare patters in these files, like agaist all other files? Not something like TRID.

By the way, I was toying with Xpert 2 under the relinker option, has anyone noticed the columns?
DecStartLba, DecFileSize, LBAPos, FileID HexStartLBA, HexFileSize, HashString,

Maybe there is a way to search and find a file from one of the above options and see how it is being used a good place to start would be in a Memory Dump of CC running.
I just made a memory dump, here: http://www.mediafire.com/download.php?ny1kmmzywnd

Its Zack in the church, right after meeting Aerith.

Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Lucleonhart on 2009-04-24 07:36:37
this time with FF8 models!
Jeha! FF8 FTW!!! *ggg*
Kepp on your grrrrrrrrrrrrreat work! :)
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: squallff8 on 2009-05-05 11:45:56
Sorry for offtop

2 koral
If you still interested,i will upload FF7 Bonus Disc files for you :-)
here is image of original Bike model. File name is bwfd in field/char.lgp
(http://i300.photobucket.com/albums/nn15/squallff8_2008/bikebwfd.jpg)
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: deadlyxvalentine on 2009-05-06 21:40:40
Well, my first post! I read early on about how someone wanted these for JK3 and I'm wondering if anyone knows how to rig these to Valve's .smd format and to rig it for Half Life, more specifically, The Specialists 3.0 which is a Half Life 1 mod. If anyone would know and could help me I'd very much appreciate it.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Landarma on 2009-05-06 23:35:53
Did anyone test it with data from Japanese version?  I wonder.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: MrAdults on 2009-05-08 20:12:35
Well, my first post! I read early on about how someone wanted these for JK3 and I'm wondering if anyone knows how to rig these to Valve's .smd format and to rig it for Half Life, more specifically, The Specialists 3.0 which is a Half Life 1 mod. If anyone would know and could help me I'd very much appreciate it.
Well, mesh2rdm.exe exports the models straight to SMD, but it sounds like you want to use them with a different skeleton. I can tell you that you will need to import your source skeleton into the modeling program of choice, then import the FF7:CC model (in either smd or obj format, doesn't matter if it has weighting data) and manually re-weight it to the Half-Life skeleton. If you don't know how to do that, you'll have to find out in the documentation for your modeling app of choice. But be warned that re-distributing the models with Half-Life mods is a good way to draw unwanted attention to yourself and get a C&D, and it is illegal in most regions (depending on how you pull it off).

Did anyone test it with data from Japanese version?  I wonder.
Should in theory work fine, the model data sets would be no different, only offsets to the files (and presumably G's extractor handles this fine).
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: deadlyxvalentine on 2009-05-08 20:35:13
Well when most people release models they have ripped and re-rigged, they usually give credits with it. Like, one ripped and rigged DMC4 Nero for TS and gave creds to DMC4 Modeling team and you know, anything else they did they said they did and they rightfully own. They said the work they did aswell, and if the anims weren't theirs, then they'd say would was the original animator as well. So, I'm guessing this is how people get around the whole getting into trouble thing?
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: MrAdults on 2009-05-08 21:21:58
You're distributing copyrighted content, so giving credit to the authors of the work (who may or may not even own the copyright(s)) makes no difference in terms of legalities. Whether you are in violation of actual copyright law depends on where you are located. If you were to construct a likeness yourself, that's another, different set of legal issues that can be sidestepped with some twists and turns. But the bottom line is, if you draw attention to your project using content/characters from a popular IP, you risk getting a C&D regardless of whether you're within your legal rights. Just something to consider. If only a group of 20 people are going to play your mod, practicality dictates you have nothing to worry about. But before going ahead with anything like that, you should be well aware of the legal constraints.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: deadlyxvalentine on 2009-05-08 22:52:31
Woah woah woah. This isn't MY mod. This mod was made years ago buddy. Here's a link: http://www.specialistsmod.net/
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Landarma on 2009-05-09 01:12:56
In fact, I happened to get Japanese CC data, and most of extracted files were not recognized by CC viewer nor mesh2rdm.  I don't think it's fault of G's extractor, for I could recognize some files, like 00000.raw which has [ATEL] header, and movies and sound data(identified by header).  Maybe offset matters, but it's somewhat confusing, for 00001.raw(which would be GT image), [GT] is at 80h, while 00002.raw(also can be GT image), I can see [GT] header at 40h.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: MrAdults on 2009-05-09 04:08:18
In fact, I happened to get Japanese CC data, and most of extracted files were not recognized by CC viewer nor mesh2rdm.  I don't think it's fault of G's extractor, for I could recognize some files, like 00000.raw which has [ATEL] header, and movies and sound data(identified by header).  Maybe offset matters, but it's somewhat confusing, for 00001.raw(which would be GT image), [GT] is at 80h, while 00002.raw(also can be GT image), I can see [GT] header at 40h.
It would be pretty unusual for the model spec to change in any way whatsoever across territory releases, but it's possible. But without seeing a few files, I couldn't tell you if it's extracted incorrectly or legitimately different.
Edit: Oh, I didn't really read what you said. Yes, it sounds like they are not being ripped properly. Of course offset matters, otherwise you aren't really getting a clean cut of the file, which means it's not extracted right. :) (and it's probably missing something off the end too based on your description, but who knows)

Woah woah woah. This isn't MY mod. This mod was made years ago buddy. Here's a link: http://www.specialistsmod.net/
I was not implying that you made The Specialists. I used to be quite an Action Half-Life fan/player myself, and have heard of the mod before. But the matter of legalities stands, please do educate yourself before you distribute anything.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: deadlyxvalentine on 2009-05-09 04:14:36
Alright, I've been educated. But thing is I'm probably not gonna release this at all. Not saying I'm angry, but you keep freaking me out saying I need to learn of these legalities and such, and I HAVE........ Only problem is right now is that I'm stuck! I have Reno in .smd format and I can't seem to get him rigged onto the TS skeleton cuz well...I'm a newb. Also, the textures are messed up.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Landarma on 2009-05-09 04:34:39
Yes, it sounds like they are not being ripped properly. Of course offset matters, otherwise you aren't really getting a clean cut of the file, which means it's not extracted right. :) (and it's probably missing something off the end too based on your description, but who knows)
Perhaps yes, although audio and video files seemed to be ripped properly. 
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-05-10 21:05:35
mmm, seems like G's extractor has its limitations.
But he provided the source, so it may be something trivial like offset values which simply need adjusting or whatever.
Although I don't suppose many people will bother with the Japanese version when they have the US or PAL versions to rip models from, right?

I really must get back to this project, you guys are all probably awaiting the map-models pretty eagerly, huh  :-P
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Landarma on 2009-05-10 23:21:18
Although I don't suppose many people will bother with the Japanese version when they have the US or PAL versions to rip models from, right?
I don't think we Asians will ever buy US or PAL version.  ;)  Anyway, I need to confirm if it's extractor's fault, though I don't think it will be done any time soon....
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: MrAdults on 2009-05-11 05:14:55
Although I don't suppose many people will bother with the Japanese version when they have the US or PAL versions to rip models from, right?
I don't think we Asians will ever buy US or PAL version.  ;)  Anyway, I need to confirm if it's extractor's fault, though I don't think it will be done any time soon....
You could just send koral or I a few of your raw files to verify that. See koral's list on the first page for which files should be models. But I can guess with a high degree of probability what the answer will be, and you will not be any happier after hearing it. ;)
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Landarma on 2009-05-13 07:16:38
One more: Extracting data files results in 8787 .raw files.  As wiki states, 0000-0005 can be some kind of miscellaneous files, and 0006~1494 are media files(movie, sound effect/dialogue, and BGM).  [ATEL] and some more files which have recognizable headers can be read by CC viewer, so I do not need to worry about them so much.  However, I need to look through more.

**I asked people at somewhere for confirmation, but I doubt if they're interested on this.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-05-13 23:41:44
I will try to get hold of the Japanese version of the game and see what I can do.
But really, there shouldn't be much difference at all, except script changes and perhaps different audio.
Although I believe if there are any unused or hidden items etc in CC, we are more likely to find them in the Japanese version of the game than in any other release.  :wink:
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Aurenasek116 on 2009-05-14 05:54:57
Japanese one has two diffs. First of all, japanese version has more audio files than english one (It was confirmed while making an undub of english one ;P) and has changed Minerva statue model. In japanese version it looked too much like cathlic Maria so in US release they made completly new model for it.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Landarma on 2009-05-14 06:49:36
For some files from Japanese version:
00000~00005: Some miscellaneous files.  00000.raw is started with [Atel], while 00001 and 00002 have [GT] inside.
00006~00062, 08769~08787: [PSMF0014] header.  Video clips.
00063~01426: [RIFF] header.  Dialogues or sound effects.  Even with ATRAC3 codec, I couldn't play it just by changing its extension.  You may need Sonicstage and Hi-MD renderer to decode & listen it, but I'm not sure.
01427~01494: [SSCF] header.  BGM or something.

Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Aurenasek116 on 2009-05-14 06:57:04
From what I know both Crisis Core and Dissidia use Atrac3+, so you won't be able to play them without ac3+ codec and sadly such doesn't exist yet for PC.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Landarma on 2009-05-14 07:01:28
From what I know both Crisis Core and Dissidia use Atrac3+, so you won't be able to play them without ac3+ codec and sadly such doesn't exist yet for PC.
However, AT3 files in Dissidia:FF can be decoded with Hi-MD Renderer.  It may be able to decode Crisis Core files, however, bitrate matters(if SonicStage or Hi-MD renderer does not support certain bitrate, it cannot be converted).
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-05-18 22:49:26
Japanese one has two diffs. First of all, japanese version has more audio files than english one (It was confirmed while making an undub of english one ;P) and has changed Minerva statue model. In japanese version it looked too much like cathlic Maria so in US release they made completly new model for it.

Oh, OK. I didn't know that. I will wikify this little trivia for future reference.

But I wonder why there is more Japanese audio?
Usually it takes longer to say something in English than in Japanese, so the only explanation I can think of is that the Japanese CC dialogues were more in-depth and may have suggested better character development or something.
I will dig into the game scripts to see what else I can find  :-D


But anyway, I wished to update you all on the progress on RINOA: progress is slow (slower than a snail walking :lol:) but the tool is still alive and well.

I have finished adding FF8 [MCH] Field Models now (albeit with rough guesstimates of bone positions) and unlike mirex's biturn, I intend to allow exporting these models out with the bone and weights intact, probably as directx models although I am looking into alternatives.

The idea is, these simple MCH files have allowed me to create the foundation for animated characters within RINOA, so in the future it should be able to handle the more complicated bone-weighted characters from CC and Dissidia (or other games RINOA might fancy).

And karlisle reminded me that the OBJ format needs the broken material-file fixed, so I will also be fixing that sharpish.

I cannot give an exact date for the next release, but I can promise it will be before the end of this month.

In the words of a small, wise green thing: Patient, you must be  :-)
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: Aurenasek116 on 2009-05-19 06:16:33
Quote from: koral
Patient, you must be  :-)

Fast road, to dark side leads.
Title: Re: FF7 Crisis Core - File Format Investigation (and my CC File-Viewer Tool)
Post by: koral on 2009-05-20 20:29:58
Boring, the slow road becomes  :lol:

I decided to get cracking on RINOA and when I had the OBJ exporter fixed, I decided to just get it out for you all and explain the future of the project a bit.
I created a new Topic just for RINOA here: http://forums.qhimm.com/index.php?topic=8451.0

Please try to keep this topic focused entirely on the CC Data side of things, for anything regarding viewing or extracting the said data, use the other RINOA topic instead.
Sorry to split it up and complicate matters a bit, but it was the only logical thing to do  :-D

My next steps will be to look for the character animations, and karlisle has provided a number of leads already.
But I will probably look into that ZACK files (08623 - 08627) next, because they should contain everything about the field model that the player controls and runs around the maps with.
Title: Re: FF7 Crisis Core - File Format and Data Investigation
Post by: obesebear on 2009-05-21 01:54:55
I noticed when extracting the characters that the summons aren't in the same mix as the npc's.   Now I haven't taken the time to go through EVERY file (because it seems to cause Koral's program to randomly crash) but does anyone know where they might be?  A crisis core Bahamut, Tonberry, Magic Pot, Cait Sith, and Ifrit would be quite nice
Title: Re: FF7 Crisis Core - File Format and Data Investigation
Post by: Aurenasek116 on 2009-05-21 07:14:19
I noticed when extracting the characters that the summons aren't in the same mix as the npc's.   Now I haven't taken the time to go through EVERY file (because it seems to cause Koral's program to randomly crash) but does anyone know where they might be?  A crisis core Bahamut, Tonberry, Magic Pot, Cait Sith, and Ifrit would be quite nice

Erm, all models are there o.O I had no problems finding tonberry, both bahamuts, ifrit, cait sith and others.
Title: Re: FF7 Crisis Core - File Format Investigation
Post by: Colly on 2009-05-30 07:50:15
Here
http://i464.photobucket.com/albums/rr10/ignitzhjfk/TABELA.jpg
I found thias in the lost files
XD
I delete the header to make this table font to be visualizble

In files extracted by G's extractor, this table file has number 01418~01420, correct?

How did you make that this file is visible? I know that you delete the header, but how should I arrange the rest of the file to see the font?


I found pointers to the some of MBD files, but to the some, I can't find. I have to seek deeper. ;)
Title: Re: FF7 Crisis Core - File Format and Data Investigation
Post by: killerhack on 2017-06-19 12:55:26
Hello guys  ;)
My name is Daniel, I understand that this topic is very old, but I was very happy to find it.
I am from Brazil and here Crisis Core has not yet been translated because the team that was translating gave up and now there is nothing left.  :-\
I just wanted to know if you still have the project files, if you have not, at least explain to me what method you use to identify the files and texts inside the game files.
What programming language do I need to learn to find myself as you did?  ???
Who knows I can even translate gradually the game, I understand that the process is very time consuming.
Title: Re: FF7 Crisis Core - File Format and Data Investigation
Post by: Colly on 2019-05-28 21:49:30
After almost ten years of abandoning the investigation of Crisis Core, I have finally found the pointers to text.  :-D
They are directly in every Atel file. Offset to every string is counted started from MBD flag, and stored reversed, so if new text is started, for example, 1156 bytes after MBD start, then it's 04 84, not 84 04!
There is also a section which contains sizes of parts inside the Atel files (MBD, SSCF and another files are inside). It looks like below and these sizes are stored not reversed:
(http://www.balamb.pl/qh/ptcc.PNG)

First 8 bytes is number of entries, and then 4 bytes of address starting from BDD0 (in this exact file), and number of bytes. Note that there is a padding to nearest 16 bytes.
Title: Re: FF7 Crisis Core - File Format and Data Investigation
Post by: killerhack on 2020-04-26 22:22:20
After almost ten years of abandoning the investigation of Crisis Core, I have finally found the pointers to text.  :-D
They are directly in every Atel file. Offset to every string is counted started from MBD flag, and stored reversed, so if new text is started, for example, 1156 bytes after MBD start, then it's 04 84, not 84 04!
There is also a section which contains sizes of parts inside the Atel files (MBD, SSCF and another files are inside). It looks like below and these sizes are stored not reversed:
(http://www.balamb.pl/qh/ptcc.PNG)

First 8 bytes is number of entries, and then 4 bytes of address starting from BDD0 (in this exact file), and number of bytes. Note that there is a padding to nearest 16 bytes.

Can you create an "unpacker" for the txt files?