[FF4 PC/Steam] - Is there a resource unpacker and repacker available?

[Fraggoso]

I also asked in the modding subforum for FFIV:

Is there a tool that can unpack the resource file and repack them back?
I want to mess with my graphics pipeline to get rid of the big pixel/3DS look of the textures.

I couldn't find anything so far. :/

[Maki]

Are you sure we are talking about the same game?
Mine Final Fantasy IV for Steam has ALL the files unpacked to bare formats
There's nothing to unpack and repack



Sound/.*AKB files are normal OGG, just play at 0x204
*.NCGR - are normal .PNG files

EFFECT.dat, STAGEMNG_f00.dat and event2d_pack.dat are indeed casual containers, but everything else has .lz extension. I bet it's LZS


EDIT:
Okay, .DAT reversed:


Main:

Offset	Size	Description
0	char[4]	MAGIC- "SSAM"
4	uint	Entries count
8 + (EntryID*40)	40 bytes	Entry
8 + (EntriesCount*40)	Varies	FILE_data

Entry:

Offset	Size	Description
0	uint	File Pointer + (EntryCount*40+8)
4	uint	File size
8	char[32]	File name

About LZ:

-Not LZSS algorithm [Tested/ creates rubbish]
-Not LZMA algorithm [Tested/ unknown compression error]
-Not Zlib [Tested/ unknown compression error]

For sure some LZ compression, LZ77?

Okay, I unpacked it succesfully using LZ77 algorithm grabbed from here: https://gist.github.com/Prof9/872e67a08e17081ca00e

[Fraggoso]

Sorry I didn't bought the game yet but I thought it had a container file!
Sorry for the hassle I will take a look into the png files of sort.

[Lein]

I played this some time back, it would be nice to see a mod that improves upon the chibi models. The chibi look is really off putting as are the animations. Is it possible to export/import models and animations?

[Fraggoso]

I looked at the ncgr files and it seems there's only map files stored as normal png.
I also found the tutorial pictures and basically everything that's written with graphics but I can't find any character textures.

Do you know where the character textures are stored?
Are they stored in the dat file you mentioned above?

@Lein without a tool that's not possible at all at this time.

[Maki]

Okay. I found some valuable info online. Looks like it's all well-known Nintendo DS formats:

http://www.romhacking.net/documents/%5B469%5Dnds_formats.htm#Generic

There's also info about compressions.

EDIT: Yeah... but some formats have different MAGIC than it's written in this documentation, also it lacks 3/4 of formats...

EDIT2: I'm quite busy right now, so I'll be able to reverse everything soon, but not now

UPDATE:

Did you know, that this EXE is a gold mine of debug info? Thanks to not deleted asserts and it gives you almost every possible debug info:

it also gives us info how'd they coded the game- using CodeWarrior for Nintendo DS


Oh, this is just great!

[Fraggoso]

Thanks maki! Very much appreciated.

[Fraggoso]

It's good to see that it's so open.
I really only need the texture files as of now but maybe that can open up the modding to the game as a whole.
The battle UI for instance could really need an overhaul.

[Maki]

You see, working as a L2 server support can be sometimes boring, so I coded the unpacker. I didn't have the files, so I can't really test it. Expect LZS decompression+repacking soon:

Code: [Select]

# Final Fantasy IV - .DAT Container unpacker 0.1
# By MaKiPL 21-06-2017
# Python 2.7

import struct
import os.path

mypath = r'FILEPATHHERE' #TYPE FILE PATH

f = open(mypath, 'r+b')
magic = f.read(4)

if magic!= 'SSAM':
    print('This is not FFIV DAT container file!')
    exit

count = struct.unpack('<I', f.read(4))[0]
pointers = [0]
sizes = [0]
names = ['0']

_index = 0
while _index < count:
    pointers[_index] = struct.unpack('<I', f.read(4))[0]
    sizes[_index] = struct.unpack('<I', f.read(4))[0]
    names[_index] = f.read(32)
    print(pointers[_index])
    print(sizes[_index])
    print(names[_index])
    pointers.append(0)
    sizes.append(0)
    names.append('0')
    _index += 1
   
relativeFiles = count*40+8

OutputDir = os.path.dirname(mypath) + '\\' + os.path.basename(mypath)

if not os.path.exists(OutputDir[:-4] + 'dec'):
    os.mkdir(OutputDir[:-4] + 'dec')

_index = 0

while _index < count:
    f.seek(relativeFiles+pointers[_index], 0)
    buff = f.read(sizes[_index])
    temppath = OutputDir[:-4] + 'dec\\' + names[_index]
    if os.path.exists(temppath):
        os.remove(temppath)
    ff = open(temppath, 'w+b')
    ff.write(buff)
    ff.close()
    _index += 1
   
f.close()


[Fraggoso]

Maki that's wonderful!
How can I test it on my end? xD

Thanks and keep me posted on the repacker!

[Maki]

You need to install Python 2.7 (I believe it should work on Python 3 too)
Copy this script, paste into new file and name it as you wish, but add .py extensions (e.g. FF4steamUnpacker.py)
Then edit the file putting the container you want to unpack in line 8

This article should help you set up environment paths:
http://effbot.org/pyfaq/how-do-i-run-a-python-program-under-windows.htm

Then you'll be able to just open cmd and type like:
python.exe NameOfScript, like:

python.exe FF4SteamUnpacker.py

[Fraggoso]

Okay I'll give it a try. Thanks again!
Hopefully nothing will blow up on my end. ;P

[Maki]

Okay. I came back home and tested it with original files. I made a mistake. Here's corrected code:

Code: [Select]

# Final Fantasy IV - .DAT Container unpacker 0.2
# By MaKiPL 21-06-2017
# Python 2.7

import struct
import os.path

mypath = r'D:\FINAL FANTASY IV\EXTRACTED_DATA\files\STAGEMNG_f00.dat' #TYPE FILE PATH

f = open(mypath, 'r+b')
magic = f.read(4)

if magic!= 'SSAM':
    print('This is not FFIV DAT container file!')
    exit

count = struct.unpack('<I', f.read(4))[0]
pointers = [0]
sizes = [0]
names = ['0']

_index = 0
while _index < count:
    pointers[_index] = struct.unpack('<I', f.read(4))[0]
    sizes[_index] = struct.unpack('<I', f.read(4))[0]
    names[_index] = f.read(32).rstrip('\0')
    print(pointers[_index])
    print(sizes[_index])
    print(names[_index])
    pointers.append(0)
    sizes.append(0)
    names.append('0')
    _index += 1
   
relativeFiles = count*40+8

OutputDir = os.path.dirname(mypath) + '\\' + os.path.basename(mypath)
if not os.path.exists(OutputDir[:-4] + 'dec'):
    os.mkdir(OutputDir[:-4] + 'dec')

_index = 0

while _index < count:
    f.seek(relativeFiles+pointers[_index], 0)
    buff = f.read(sizes[_index])
    temppath = OutputDir[:-4] + 'dec\\' + names[_index]
    if os.path.exists(temppath):
        os.remove(temppath)
    ff = open(temppath, 'w+b')
    ff.write(buff)
    ff.close()
    _index += 1
   
f.close()

Also run this code on Python 2.x
I'll copy some files on pendrive to test the code on original files tomorrow for LZS decompression and repacking

[Fraggoso]

So I've setup Python and I can see that it works when I use cmd and type python I get the >>>
From there I use python.exe _ffiv.py and I get this error message:

Code: [Select]

>>> python.exe _ffiv.py
  File "<stdin>", line 1
    python.exe _ffiv.py
                   ^
SyntaxError: invalid syntax
I changed your filepath so it matches mine. Am I missing something?

[Maki]

So I've setup Python and I can see that it works when I use cmd and type python I get the >>>
From there I use python.exe _ffiv.py and I get this error message:

Code: [Select]

>>> python.exe _ffiv.py
  File "<stdin>", line 1
    python.exe _ffiv.py
                   ^
SyntaxError: invalid syntax
I changed your filepath so it matches mine. Am I missing something?

You should run it more like this:

But worry not. I'm preparing the C version of it, because no any python LZ77 algorithm works for the files, I don't know why

[Fraggoso]

Okay I'll try it out and see what the results are.

[Fraggoso]

It works!
How can I test the LZ77Decompress you posted, Maki. Or should I wait for your C Version?
Sorry, I'm a noob when it comes down to such things. ^^

[Maki]

It works!
How can I test the LZ77Decompress you posted, Maki. Or should I wait for your C Version?
Sorry, I'm a noob when it comes down to such things. ^^

I'm just a total moron- it's not LZ77, but GBA LZ77, which in fact is not LZ77 but LZ10. I almost finished the C tool, it has all- auto extract with LZ decompression and repacking too! But... I'm struggling with this for hours and I still get HEAP corruption no matter what.

[Fraggoso]

Hi Maki, no problem!
I'll gladly wait for it. =)

[Maki]

Okay. I'm done. I finally found the location where I corrupted the memory. Fixed it and everything works fine (yet the software crashes just as it's exits, but that's minor issue, nothing to worry about)

https://github.com/MaKiPL/FFIV_repacker/releases

You can grab it from there ^

There might be a compression issue... After uncompressing GBA LZ file from FFIV and recompressing it again, the file content doesn't match 1:1. The difference are the control bits, they are different, but the whole concept of data structure is the same (therefore that might work)
If anyone have an idea why the software crashes at closing, then feel free to tell

[Fraggoso]

Thanks again Maki!

I tried it and you guess: Epic fail (or almost)...
At least I could decompress the dat's, that worked. The files in the folder can't be decompressed though. It says "Not LZ77 compressed file; Header is wrong!"

If I use -unzl77 on a random *.lz file decompress it as the filename I give it but how do I know it's a texture? Going over aprox 5.100 could take loooong. *_*

[Maki]

I tried it and you guess: Epic fail (or almost)...
At least I could decompress the dat's, that worked. The files in the folder can't be decompressed though. It says "Not LZ77 compressed file; Header is wrong!"

Which file you tried to export? If it's not LZ77 file, then it gets unpacked as is.

About unlz77, we don't know it. We have to reverse the game and find corresponding file. Looks like FFIV uses format like Model+texture- it's impediment. I think easier would be just to hook DirectX renderer and change textures from there.

[Fraggoso]

The lz files that are extracted from the dat like in streaming.
The game operates in OpenGL sadly. :/ I could manage to hook it via umod but it's not running that well and has some graphic error. :/

[Fraggoso]

So right now I can't do anything with your unpacker/repacker or did I miss anything?
What is it good as of now and isn't there really a possibility to unpack the texture files?

[Fraggoso]

Any news on this?
I really want to enhance the subpar texture quality of it. ^^