Author Topic: [FF4 PC/Steam] - Is there a resource unpacker and repacker available?  (Read 947 times)

Fraggoso

  • Crazy poster
  • *
  • Posts: 204
  • Karma: 8
    • View Profile
I also asked in the modding subforum for FFIV:

Is there a tool that can unpack the resource file and repack them back?
I want to mess with my graphics pipeline to get rid of the big pixel/3DS look of the textures.

I couldn't find anything so far. :/

Maki

  • Lana Del Rey
  • Insane poster
  • *
  • Posts: 430
  • Karma: 76
  • Final Fantasy VIII reverse team
    • View Profile
Are you sure we are talking about the same game?
Mine Final Fantasy IV for Steam has ALL the files unpacked to bare formats
There's nothing to unpack and repack



Sound/.*AKB files are normal OGG, just play at 0x204
*.NCGR - are normal .PNG files

EFFECT.dat, STAGEMNG_f00.dat and event2d_pack.dat are indeed casual containers, but everything else has .lz extension. I bet it's LZS


EDIT:
Okay, .DAT reversed:


Main:


OffsetSizeDescription
0char[4]MAGIC- "SSAM"
4uintEntries count
8 + (EntryID*40)40 bytesEntry
8 + (EntriesCount*40)VariesFILE_data

Entry:

OffsetSizeDescription
0uintFile Pointer + (EntryCount*40+8)
4uintFile size
8char[32]File name


About LZ:

-Not LZSS algorithm [Tested/ creates rubbish]
-Not LZMA algorithm [Tested/ unknown compression error]
-Not Zlib [Tested/ unknown compression error]

For sure some LZ compression, LZ77?

Okay, I unpacked it succesfully using LZ77 algorithm grabbed from here: https://gist.github.com/Prof9/872e67a08e17081ca00e
« Last Edit: 2017-06-09 17:46:43 by Maki »

Fraggoso

  • Crazy poster
  • *
  • Posts: 204
  • Karma: 8
    • View Profile
Sorry I didn't bought the game yet but I thought it had a container file!
Sorry for the hassle I will take a look into the png files of sort. :)

Lein

  • Cool newbie
  • *
  • Posts: 74
  • Karma: -17
    • View Profile
I played this some time back, it would be nice to see a mod that improves upon the chibi models. The chibi look is really off putting as are the animations. Is it possible to export/import models and animations?

Fraggoso

  • Crazy poster
  • *
  • Posts: 204
  • Karma: 8
    • View Profile
I looked at the ncgr files and it seems there's only map files stored as normal png.
I also found the tutorial pictures and basically everything that's written with graphics but I can't find any character textures.

Do you know where the character textures are stored?
Are they stored in the dat file you mentioned above?

@Lein without a tool that's not possible at all at this time.
« Last Edit: 2017-06-10 09:04:18 by Fraggoso »

Maki

  • Lana Del Rey
  • Insane poster
  • *
  • Posts: 430
  • Karma: 76
  • Final Fantasy VIII reverse team
    • View Profile
Okay. I found some valuable info online. Looks like it's all well-known Nintendo DS formats:

http://www.romhacking.net/documents/%5B469%5Dnds_formats.htm#Generic

There's also info about compressions. :)

EDIT: Yeah... but some formats have different MAGIC than it's written in this documentation, also it lacks 3/4 of formats...

EDIT2: I'm quite busy right now, so I'll be able to reverse everything soon, but not now

UPDATE:

Did you know, that this EXE is a gold mine of debug info? Thanks to not deleted asserts and it gives you almost every possible debug info:

it also gives us info how'd they coded the game- using CodeWarrior for Nintendo DS


Oh, this is just great!
« Last Edit: 2017-06-10 19:36:59 by Maki »

Fraggoso

  • Crazy poster
  • *
  • Posts: 204
  • Karma: 8
    • View Profile
Thanks maki! Very much appreciated.

Fraggoso

  • Crazy poster
  • *
  • Posts: 204
  • Karma: 8
    • View Profile
It's good to see that it's so open. ;)
I really only need the texture files as of now but maybe that can open up the modding to the game as a whole.
The battle UI for instance could really need an overhaul. :)

Maki

  • Lana Del Rey
  • Insane poster
  • *
  • Posts: 430
  • Karma: 76
  • Final Fantasy VIII reverse team
    • View Profile
You see, working as a L2 server support can be sometimes boring, so I coded the unpacker. I didn't have the files, so I can't really test it. Expect LZS decompression+repacking soon:

Code: [Select]
# Final Fantasy IV - .DAT Container unpacker 0.1
# By MaKiPL 21-06-2017
# Python 2.7

import struct
import os.path

mypath = r'FILEPATHHERE' #TYPE FILE PATH

f = open(mypath, 'r+b')
magic = f.read(4)

if magic!= 'SSAM':
    print('This is not FFIV DAT container file!')
    exit

count = struct.unpack('<I', f.read(4))[0]
pointers = [0]
sizes = [0]
names = ['0']

_index = 0
while _index < count:
    pointers[_index] = struct.unpack('<I', f.read(4))[0]
    sizes[_index] = struct.unpack('<I', f.read(4))[0]
    names[_index] = f.read(32)
    print(pointers[_index])
    print(sizes[_index])
    print(names[_index])
    pointers.append(0)
    sizes.append(0)
    names.append('0')
    _index += 1
   
relativeFiles = count*40+8

OutputDir = os.path.dirname(mypath) + '\\' + os.path.basename(mypath)

if not os.path.exists(OutputDir[:-4] + 'dec'):
    os.mkdir(OutputDir[:-4] + 'dec')

_index = 0

while _index < count:
    f.seek(relativeFiles+pointers[_index], 0)
    buff = f.read(sizes[_index])
    temppath = OutputDir[:-4] + 'dec\\' + names[_index]
    if os.path.exists(temppath):
        os.remove(temppath)
    ff = open(temppath, 'w+b')
    ff.write(buff)
    ff.close()
    _index += 1
   
f.close()

Fraggoso

  • Crazy poster
  • *
  • Posts: 204
  • Karma: 8
    • View Profile
Maki that's wonderful!
How can I test it on my end? xD

Thanks and keep me posted on the repacker! :D

Maki

  • Lana Del Rey
  • Insane poster
  • *
  • Posts: 430
  • Karma: 76
  • Final Fantasy VIII reverse team
    • View Profile
You need to install Python 2.7 (I believe it should work on Python 3 too)
Copy this script, paste into new file and name it as you wish, but add .py extensions (e.g. FF4steamUnpacker.py)
Then edit the file putting the container you want to unpack in line 8

This article should help you set up environment paths:
http://effbot.org/pyfaq/how-do-i-run-a-python-program-under-windows.htm

Then you'll be able to just open cmd and type like:
python.exe NameOfScript, like:

python.exe FF4SteamUnpacker.py

:)

Fraggoso

  • Crazy poster
  • *
  • Posts: 204
  • Karma: 8
    • View Profile
Okay I'll give it a try. Thanks again!
Hopefully nothing will blow up on my end. ;P

Maki

  • Lana Del Rey
  • Insane poster
  • *
  • Posts: 430
  • Karma: 76
  • Final Fantasy VIII reverse team
    • View Profile
Okay. I came back home and tested it with original files. I made a mistake. Here's corrected code:

Code: [Select]
# Final Fantasy IV - .DAT Container unpacker 0.2
# By MaKiPL 21-06-2017
# Python 2.7

import struct
import os.path

mypath = r'D:\FINAL FANTASY IV\EXTRACTED_DATA\files\STAGEMNG_f00.dat' #TYPE FILE PATH

f = open(mypath, 'r+b')
magic = f.read(4)

if magic!= 'SSAM':
    print('This is not FFIV DAT container file!')
    exit

count = struct.unpack('<I', f.read(4))[0]
pointers = [0]
sizes = [0]
names = ['0']

_index = 0
while _index < count:
    pointers[_index] = struct.unpack('<I', f.read(4))[0]
    sizes[_index] = struct.unpack('<I', f.read(4))[0]
    names[_index] = f.read(32).rstrip('\0')
    print(pointers[_index])
    print(sizes[_index])
    print(names[_index])
    pointers.append(0)
    sizes.append(0)
    names.append('0')
    _index += 1
   
relativeFiles = count*40+8

OutputDir = os.path.dirname(mypath) + '\\' + os.path.basename(mypath)
if not os.path.exists(OutputDir[:-4] + 'dec'):
    os.mkdir(OutputDir[:-4] + 'dec')

_index = 0

while _index < count:
    f.seek(relativeFiles+pointers[_index], 0)
    buff = f.read(sizes[_index])
    temppath = OutputDir[:-4] + 'dec\\' + names[_index]
    if os.path.exists(temppath):
        os.remove(temppath)
    ff = open(temppath, 'w+b')
    ff.write(buff)
    ff.close()
    _index += 1
   
f.close()

Also run this code on Python 2.x
I'll copy some files on pendrive to test the code on original files tomorrow for LZS decompression and repacking

Fraggoso

  • Crazy poster
  • *
  • Posts: 204
  • Karma: 8
    • View Profile
So I've setup Python and I can see that it works when I use cmd and type python I get the >>>
From there I use python.exe _ffiv.py and I get this error message:

Code: [Select]
>>> python.exe _ffiv.py
  File "<stdin>", line 1
    python.exe _ffiv.py
                   ^
SyntaxError: invalid syntax

I changed your filepath so it matches mine. Am I missing something?

Maki

  • Lana Del Rey
  • Insane poster
  • *
  • Posts: 430
  • Karma: 76
  • Final Fantasy VIII reverse team
    • View Profile
So I've setup Python and I can see that it works when I use cmd and type python I get the >>>
From there I use python.exe _ffiv.py and I get this error message:

Code: [Select]
>>> python.exe _ffiv.py
  File "<stdin>", line 1
    python.exe _ffiv.py
                   ^
SyntaxError: invalid syntax

I changed your filepath so it matches mine. Am I missing something?

You should run it more like this:

But worry not. I'm preparing the C version of it, because no any python LZ77 algorithm works for the files, I don't know why

Fraggoso

  • Crazy poster
  • *
  • Posts: 204
  • Karma: 8
    • View Profile
Okay I'll try it out and see what the results are. :)

Fraggoso

  • Crazy poster
  • *
  • Posts: 204
  • Karma: 8
    • View Profile
It works!
How can I test the LZ77Decompress you posted, Maki. Or should I wait for your C Version?
Sorry, I'm a noob when it comes down to such things. ^^
« Last Edit: 2017-06-27 14:52:55 by Fraggoso »

Maki

  • Lana Del Rey
  • Insane poster
  • *
  • Posts: 430
  • Karma: 76
  • Final Fantasy VIII reverse team
    • View Profile
It works!
How can I test the LZ77Decompress you posted, Maki. Or should I wait for your C Version?
Sorry, I'm a noob when it comes down to such things. ^^

I'm just a total moron- it's not LZ77, but GBA LZ77, which in fact is not LZ77 but LZ10. I almost finished the C tool, it has all- auto extract with LZ decompression and repacking too! But... I'm struggling with this for hours and I still get HEAP corruption no matter what.
« Last Edit: 2017-06-30 19:15:43 by Maki »

Fraggoso

  • Crazy poster
  • *
  • Posts: 204
  • Karma: 8
    • View Profile
Hi Maki, no problem!
I'll gladly wait for it. =)

Maki

  • Lana Del Rey
  • Insane poster
  • *
  • Posts: 430
  • Karma: 76
  • Final Fantasy VIII reverse team
    • View Profile
Okay. I'm done. I finally found the location where I corrupted the memory. Fixed it and everything works fine (yet the software crashes just as it's exits, but that's minor issue, nothing to worry about)

https://github.com/MaKiPL/FFIV_repacker/releases

You can grab it from there ^

There might be a compression issue... After uncompressing GBA LZ file from FFIV and recompressing it again, the file content doesn't match 1:1. The difference are the control bits, they are different, but the whole concept of data structure is the same (therefore that might work)
If anyone have an idea why the software crashes at closing, then feel free to tell

Fraggoso

  • Crazy poster
  • *
  • Posts: 204
  • Karma: 8
    • View Profile
Thanks again Maki!

I tried it and you guess: Epic fail (or almost)...
At least I could decompress the dat's, that worked. The files in the folder can't be decompressed though. It says "Not LZ77 compressed file; Header is wrong!"

If I use -unzl77 on a random *.lz file decompress it as the filename I give it but how do I know it's a texture? Going over aprox 5.100 could take loooong. *_*

Maki

  • Lana Del Rey
  • Insane poster
  • *
  • Posts: 430
  • Karma: 76
  • Final Fantasy VIII reverse team
    • View Profile
I tried it and you guess: Epic fail (or almost)...
At least I could decompress the dat's, that worked. The files in the folder can't be decompressed though. It says "Not LZ77 compressed file; Header is wrong!"

Which file you tried to export? If it's not LZ77 file, then it gets unpacked as is.

About unlz77, we don't know it. We have to reverse the game and find corresponding file. Looks like FFIV uses format like Model+texture- it's impediment. I think easier would be just to hook DirectX renderer and change textures from there.


Fraggoso

  • Crazy poster
  • *
  • Posts: 204
  • Karma: 8
    • View Profile
The lz files that are extracted from the dat like in streaming.
The game operates in OpenGL sadly. :/ I could manage to hook it via umod but it's not running that well and has some graphic error. :/

Fraggoso

  • Crazy poster
  • *
  • Posts: 204
  • Karma: 8
    • View Profile
So right now I can't do anything with your unpacker/repacker or did I miss anything?
What is it good as of now and isn't there really a possibility to unpack the texture files? :(