Qhimm.com Forums

Miscellaneous Forums => Scripting and Reverse Engineering => Topic started by: Fraggoso on 2017-06-07 07:51:29

Title: [FF4 PC/Steam] - Is there a resource unpacker and repacker available?
Post by: Fraggoso on 2017-06-07 07:51:29
I also asked in the modding subforum for FFIV:

Is there a tool that can unpack the resource file and repack them back?
I want to mess with my graphics pipeline to get rid of the big pixel/3DS look of the textures.

I couldn't find anything so far. :/
Title: Re: [FF4 PC/Steam] - Is there a resource unpacker and repacker available?
Post by: Maki on 2017-06-09 16:45:10
Are you sure we are talking about the same game?
Mine Final Fantasy IV for Steam has ALL the files unpacked to bare formats
There's nothing to unpack and repack

(https://s19.postimg.org/ap59dpydv/Bez_tytu_u.jpg)

Sound/.*AKB files are normal OGG, just play at 0x204
*.NCGR - are normal .PNG files

EFFECT.dat, STAGEMNG_f00.dat and event2d_pack.dat are indeed casual containers, but everything else has .lz extension. I bet it's LZS


EDIT:
Okay, .DAT reversed:


Main:


OffsetSizeDescription
0char[4]MAGIC- "SSAM"
4uintEntries count
8 + (EntryID*40)40 bytesEntry
8 + (EntriesCount*40)VariesFILE_data

Entry:

OffsetSizeDescription
0uintFile Pointer + (EntryCount*40+8)
4uintFile size
8char[32]File name


About LZ:

-Not LZSS algorithm [Tested/ creates rubbish]
-Not LZMA algorithm [Tested/ unknown compression error]
-Not Zlib [Tested/ unknown compression error]

For sure some LZ compression, LZ77?

Okay, I unpacked it succesfully using LZ77 algorithm grabbed from here: https://gist.github.com/Prof9/872e67a08e17081ca00e
Title: Re: [FF4 PC/Steam] - Is there a resource unpacker and repacker available?
Post by: Fraggoso on 2017-06-09 18:39:29
Sorry I didn't bought the game yet but I thought it had a container file!
Sorry for the hassle I will take a look into the png files of sort. :)
Title: Re: [FF4 PC/Steam] - Is there a resource unpacker and repacker available?
Post by: Lein on 2017-06-10 07:57:16
I played this some time back, it would be nice to see a mod that improves upon the chibi models. The chibi look is really off putting as are the animations. Is it possible to export/import models and animations?
Title: Re: [FF4 PC/Steam] - Is there a resource unpacker and repacker available?
Post by: Fraggoso on 2017-06-10 09:02:43
I looked at the ncgr files and it seems there's only map files stored as normal png.
I also found the tutorial pictures and basically everything that's written with graphics but I can't find any character textures.

Do you know where the character textures are stored?
Are they stored in the dat file you mentioned above?

@Lein without a tool that's not possible at all at this time.
Title: Re: [FF4 PC/Steam] - Is there a resource unpacker and repacker available?
Post by: Maki on 2017-06-10 10:03:39
Okay. I found some valuable info online. Looks like it's all well-known Nintendo DS formats:

http://www.romhacking.net/documents/%5B469%5Dnds_formats.htm#Generic

There's also info about compressions. :)

EDIT: Yeah... but some formats have different MAGIC than it's written in this documentation, also it lacks 3/4 of formats...

EDIT2: I'm quite busy right now, so I'll be able to reverse everything soon, but not now

UPDATE:

Did you know, that this EXE is a gold mine of debug info? Thanks to not deleted asserts and it gives you almost every possible debug info:
(https://s19.postimg.org/iy9bmrs5v/assert.jpg)
it also gives us info how'd they coded the game- using CodeWarrior for Nintendo DS


Oh, this is just great!
(https://s19.postimg.org/tmd2lm24z/assert.jpg)
Title: Re: [FF4 PC/Steam] - Is there a resource unpacker and repacker available?
Post by: Fraggoso on 2017-06-10 11:30:53
Thanks maki! Very much appreciated.
Title: Re: [FF4 PC/Steam] - Is there a resource unpacker and repacker available?
Post by: Fraggoso on 2017-06-17 12:51:06
It's good to see that it's so open. ;)
I really only need the texture files as of now but maybe that can open up the modding to the game as a whole.
The battle UI for instance could really need an overhaul. :)
Title: Re: [FF4 PC/Steam] - Is there a resource unpacker and repacker available?
Post by: Maki on 2017-06-21 09:36:54
You see, working as a L2 server support can be sometimes boring, so I coded the unpacker. I didn't have the files, so I can't really test it. Expect LZS decompression+repacking soon:

Code: [Select]
# Final Fantasy IV - .DAT Container unpacker 0.1
# By MaKiPL 21-06-2017
# Python 2.7

import struct
import os.path

mypath = r'FILEPATHHERE' #TYPE FILE PATH

f = open(mypath, 'r+b')
magic = f.read(4)

if magic!= 'SSAM':
    print('This is not FFIV DAT container file!')
    exit

count = struct.unpack('<I', f.read(4))[0]
pointers = [0]
sizes = [0]
names = ['0']

_index = 0
while _index < count:
    pointers[_index] = struct.unpack('<I', f.read(4))[0]
    sizes[_index] = struct.unpack('<I', f.read(4))[0]
    names[_index] = f.read(32)
    print(pointers[_index])
    print(sizes[_index])
    print(names[_index])
    pointers.append(0)
    sizes.append(0)
    names.append('0')
    _index += 1
   
relativeFiles = count*40+8

OutputDir = os.path.dirname(mypath) + '\\' + os.path.basename(mypath)

if not os.path.exists(OutputDir[:-4] + 'dec'):
    os.mkdir(OutputDir[:-4] + 'dec')

_index = 0

while _index < count:
    f.seek(relativeFiles+pointers[_index], 0)
    buff = f.read(sizes[_index])
    temppath = OutputDir[:-4] + 'dec\\' + names[_index]
    if os.path.exists(temppath):
        os.remove(temppath)
    ff = open(temppath, 'w+b')
    ff.write(buff)
    ff.close()
    _index += 1
   
f.close()
Title: Re: [FF4 PC/Steam] - Is there a resource unpacker and repacker available?
Post by: Fraggoso on 2017-06-21 09:39:02
Maki that's wonderful!
How can I test it on my end? xD

Thanks and keep me posted on the repacker! :D
Title: Re: [FF4 PC/Steam] - Is there a resource unpacker and repacker available?
Post by: Maki on 2017-06-21 09:49:16
You need to install Python 2.7 (I believe it should work on Python 3 too)
Copy this script, paste into new file and name it as you wish, but add .py extensions (e.g. FF4steamUnpacker.py)
Then edit the file putting the container you want to unpack in line 8

This article should help you set up environment paths:
http://effbot.org/pyfaq/how-do-i-run-a-python-program-under-windows.htm

Then you'll be able to just open cmd and type like:
python.exe NameOfScript, like:

python.exe FF4SteamUnpacker.py

:)
Title: Re: [FF4 PC/Steam] - Is there a resource unpacker and repacker available?
Post by: Fraggoso on 2017-06-21 09:51:18
Okay I'll give it a try. Thanks again!
Hopefully nothing will blow up on my end. ;P
Title: Re: [FF4 PC/Steam] - Is there a resource unpacker and repacker available?
Post by: Maki on 2017-06-21 16:46:11
Okay. I came back home and tested it with original files. I made a mistake. Here's corrected code:

Code: [Select]
# Final Fantasy IV - .DAT Container unpacker 0.2
# By MaKiPL 21-06-2017
# Python 2.7

import struct
import os.path

mypath = r'D:\FINAL FANTASY IV\EXTRACTED_DATA\files\STAGEMNG_f00.dat' #TYPE FILE PATH

f = open(mypath, 'r+b')
magic = f.read(4)

if magic!= 'SSAM':
    print('This is not FFIV DAT container file!')
    exit

count = struct.unpack('<I', f.read(4))[0]
pointers = [0]
sizes = [0]
names = ['0']

_index = 0
while _index < count:
    pointers[_index] = struct.unpack('<I', f.read(4))[0]
    sizes[_index] = struct.unpack('<I', f.read(4))[0]
    names[_index] = f.read(32).rstrip('\0')
    print(pointers[_index])
    print(sizes[_index])
    print(names[_index])
    pointers.append(0)
    sizes.append(0)
    names.append('0')
    _index += 1
   
relativeFiles = count*40+8

OutputDir = os.path.dirname(mypath) + '\\' + os.path.basename(mypath)
if not os.path.exists(OutputDir[:-4] + 'dec'):
    os.mkdir(OutputDir[:-4] + 'dec')

_index = 0

while _index < count:
    f.seek(relativeFiles+pointers[_index], 0)
    buff = f.read(sizes[_index])
    temppath = OutputDir[:-4] + 'dec\\' + names[_index]
    if os.path.exists(temppath):
        os.remove(temppath)
    ff = open(temppath, 'w+b')
    ff.write(buff)
    ff.close()
    _index += 1
   
f.close()

Also run this code on Python 2.x
I'll copy some files on pendrive to test the code on original files tomorrow for LZS decompression and repacking
Title: Re: [FF4 PC/Steam] - Is there a resource unpacker and repacker available?
Post by: Fraggoso on 2017-06-23 11:53:29
So I've setup Python and I can see that it works when I use cmd and type python I get the >>>
From there I use python.exe _ffiv.py and I get this error message:

Code: [Select]
>>> python.exe _ffiv.py
  File "<stdin>", line 1
    python.exe _ffiv.py
                   ^
SyntaxError: invalid syntax

I changed your filepath so it matches mine. Am I missing something?
Title: Re: [FF4 PC/Steam] - Is there a resource unpacker and repacker available?
Post by: Maki on 2017-06-23 17:19:52
So I've setup Python and I can see that it works when I use cmd and type python I get the >>>
From there I use python.exe _ffiv.py and I get this error message:

Code: [Select]
>>> python.exe _ffiv.py
  File "<stdin>", line 1
    python.exe _ffiv.py
                   ^
SyntaxError: invalid syntax

I changed your filepath so it matches mine. Am I missing something?

You should run it more like this:
(https://s12.postimg.org/xk157d3xp/more.jpg)
But worry not. I'm preparing the C version of it, because no any python LZ77 algorithm works for the files, I don't know why
Title: Re: [FF4 PC/Steam] - Is there a resource unpacker and repacker available?
Post by: Fraggoso on 2017-06-23 20:47:12
Okay I'll try it out and see what the results are. :)
Title: Re: [FF4 PC/Steam] - Is there a resource unpacker and repacker available?
Post by: Fraggoso on 2017-06-27 14:38:36
It works!
How can I test the LZ77Decompress you posted, Maki. Or should I wait for your C Version?
Sorry, I'm a noob when it comes down to such things. ^^
Title: Re: [FF4 PC/Steam] - Is there a resource unpacker and repacker available?
Post by: Maki on 2017-06-30 13:44:20
It works!
How can I test the LZ77Decompress you posted, Maki. Or should I wait for your C Version?
Sorry, I'm a noob when it comes down to such things. ^^

I'm just a total moron- it's not LZ77, but GBA LZ77, which in fact is not LZ77 but LZ10. I almost finished the C tool, it has all- auto extract with LZ decompression and repacking too! But... I'm struggling with this for hours and I still get HEAP corruption no matter what.
Title: Re: [FF4 PC/Steam] - Is there a resource unpacker and repacker available?
Post by: Fraggoso on 2017-06-30 14:20:36
Hi Maki, no problem!
I'll gladly wait for it. =)
Title: Re: [FF4 PC/Steam] - Is there a resource unpacker and repacker available?
Post by: Maki on 2017-07-03 12:23:20
Okay. I'm done. I finally found the location where I corrupted the memory. Fixed it and everything works fine (yet the software crashes just as it's exits, but that's minor issue, nothing to worry about)

https://github.com/MaKiPL/FFIV_repacker/releases

You can grab it from there ^

There might be a compression issue... After uncompressing GBA LZ file from FFIV and recompressing it again, the file content doesn't match 1:1. The difference are the control bits, they are different, but the whole concept of data structure is the same (therefore that might work)
If anyone have an idea why the software crashes at closing, then feel free to tell
Title: Re: [FF4 PC/Steam] - Is there a resource unpacker and repacker available?
Post by: Fraggoso on 2017-07-03 12:41:14
Thanks again Maki!

I tried it and you guess: Epic fail (or almost)...
At least I could decompress the dat's, that worked. The files in the folder can't be decompressed though. It says "Not LZ77 compressed file; Header is wrong!"

If I use -unzl77 on a random *.lz file decompress it as the filename I give it but how do I know it's a texture? Going over aprox 5.100 could take loooong. *_*
Title: Re: [FF4 PC/Steam] - Is there a resource unpacker and repacker available?
Post by: Maki on 2017-07-03 13:42:40
I tried it and you guess: Epic fail (or almost)...
At least I could decompress the dat's, that worked. The files in the folder can't be decompressed though. It says "Not LZ77 compressed file; Header is wrong!"

Which file you tried to export? If it's not LZ77 file, then it gets unpacked as is.

About unlz77, we don't know it. We have to reverse the game and find corresponding file. Looks like FFIV uses format like Model+texture- it's impediment. I think easier would be just to hook DirectX renderer and change textures from there.

Title: Re: [FF4 PC/Steam] - Is there a resource unpacker and repacker available?
Post by: Fraggoso on 2017-07-03 15:19:26
The lz files that are extracted from the dat like in streaming.
The game operates in OpenGL sadly. :/ I could manage to hook it via umod but it's not running that well and has some graphic error. :/
Title: Re: [FF4 PC/Steam] - Is there a resource unpacker and repacker available?
Post by: Fraggoso on 2017-07-11 14:24:06
So right now I can't do anything with your unpacker/repacker or did I miss anything?
What is it good as of now and isn't there really a possibility to unpack the texture files? :(
Title: Re: [FF4 PC/Steam] - Is there a resource unpacker and repacker available?
Post by: Fraggoso on 2017-10-04 10:36:44
Any news on this?
I really want to enhance the subpar texture quality of it. ^^