Author Topic: PyFF7: Python 3 tools to manipulate FF7 files  (Read 430 times)

niemasd

  • Fast newbie
  • *
  • Posts: 12
  • Karma: 3
    • View Profile
PyFF7: Python 3 tools to manipulate FF7 files
« on: 2019-05-06 08:52:36 »
Hey, everybody! I'm relatively new here, but I've been following the FF7 modding scene for some time. I was a bit frustrated with some of the existing tools for manipulating FF7 files (namely packing/unpacking LGP files), so I did some research and implemented my own in Python 3 called PyFF7.

Currently, I have implemented an LGP unpacker, an LGP packer, and a tool to just print information about the contents of an LGP (I tested it on char.lgp and magic.lgp, and it seems to work correctly, even when ulgp didn't). Also, the classes I design can be easily integrated into new tools, and I intend to make it available on PyPI so you can install it via pip. I also intend to expand beyond just LGP files. Note that my LGP packer rebuilds the lookup and conflict tables on-the-fly.

Also, if anybody has any information regarding the "check code"(?) in the LGP file format table of contents, I'd greatly appreciate it. Most files list this value as 14 (but some are 11 if I'm not mistaken), and a lot of old posts I read (e.g. from Aali) suggest that the game doesn't actually check this value so I currently just hardcode the value to 14 for all files, but I really want my tool to be completely correct. I was thinking maybe it's user+group file attributes (14 = 7+7 and 11 = 7+4), but yeah any info would be greatly appreciated.

Anyways, you can find the tools here: https://github.com/niemasd/PyFF7/

They're completely written in Python 3 (no external libraries/packages), so they should work out-of-the-box on any system with a Python 3 interpreter.

picklejar

  • Crazy poster
  • *
  • Posts: 143
  • Karma: 10
    • View Profile
Re: PyFF7: Python 3 tools to manipulate FF7 files
« Reply #1 on: 2019-05-06 16:36:17 »
You should check out this:
https://github.com/cebix/ff7tools

It's a whole suite of Python code for reading tons of FF7 asset files.

Basically, I'd suggest:
Any Python Dev should extend the above Python project.
Any C dev should look at sithlord's ff7 toolkit.
Any JS dev should look at Kujata.


picklejar

  • Crazy poster
  • *
  • Posts: 143
  • Karma: 10
    • View Profile
Re: PyFF7: Python 3 tools to manipulate FF7 files
« Reply #2 on: 2019-05-06 16:43:41 »
BTW, the Python suite I mentioned didn't seem to have lgp utility.

But you could look at Aali's lgp utility source code and maybe find the answer there.

I can't remember where to get Aali's source code, but I made a snapshot copy of it here:
https://github.com/picklejar76/kujata/tree/master/lgp-0.5b/src

Hope this helps!

(Edit: I originally said this was Ficedula's, but it's Aali's.)
« Last Edit: 2019-05-13 12:28:26 by picklejar »

niemasd

  • Fast newbie
  • *
  • Posts: 12
  • Karma: 3
    • View Profile
Re: PyFF7: Python 3 tools to manipulate FF7 files
« Reply #3 on: 2019-05-06 16:48:25 »
My main worry with ff7tools was that it says "The tools are designed for the PlayStation 1 version of the game. They will
not work with the game's PC port." in the README. My primary focus is on the PC and Switch version. For example, they don't have anything regarding handling LGP files as you mentioned

I checked out Ficedula's lgp utility (as well as other utilities in C++, e.g. ff7toolkit like you mentioned), and I'm not 100% certain they're implementing LGP handling properly, namely packing files with duplicate names as well as periods in the first two characters of the filename (e.g. magic.lgp)

picklejar

  • Crazy poster
  • *
  • Posts: 143
  • Karma: 10
    • View Profile
Re: PyFF7: Python 3 tools to manipulate FF7 files
« Reply #4 on: 2019-05-06 16:56:37 »
BTW, you said you tried "ulgp" and it didn't work in some cases? Which specific utility did you try, and which version? The reason I ask is, there are at least 2 or 3 different lgp utilities out there, and different versions, and links can get outdated, etc.

Also... I forgot to say the most important thing: WELCOME! This community can always use more smart developers. (I'm not an official forum moderator or anyone special. Just one of many people who love the game and reverse engineering it. Every time I try to let it go, I keep coming back.)

niemasd

  • Fast newbie
  • *
  • Posts: 12
  • Karma: 3
    • View Profile
Re: PyFF7: Python 3 tools to manipulate FF7 files
« Reply #5 on: 2019-05-06 17:11:00 »
I had found this link (Luksy's ulgp): http://forums.qhimm.com/index.php?topic=12831.0

From there, I tried all 3 versions (1.2, 1.2.1, and 1.3). None of the three worked properly on my magic.lgp (should have 5265 files total, but theirs extracted like 2200 or something)

And thank you for the warm welcome :-) This is an awesome community, so I'm excited to be involved with it all! Also loved the game, so I want to do what I can to help give back!

sithlord48

  • No life
  • *
  • Posts: 1504
  • Karma: 33
  • Dark Lord of the Savegame
    • View Profile
    • Blackchocobo
Re: PyFF7: Python 3 tools to manipulate FF7 files
« Reply #6 on: 2019-05-13 11:47:45 »
I had found this link (Luksy's ulgp): http://forums.qhimm.com/index.php?topic=12831.0

From there, I tried all 3 versions (1.2, 1.2.1, and 1.3). None of the three worked properly on my magic.lgp (should have 5265 files total, but theirs extracted like 2200 or something)

And thank you for the warm welcome :-) This is an awesome community, so I'm excited to be involved with it all! Also loved the game, so I want to do what I can to help give back!

IIRC there are some lgp tools have problems with lgps that store things in subfolders. only one is magic.lgp. (Including the ff7tk one) aali's lgp tool works great for this btw. should be code for it around here somewhere. (http://forums.qhimm.com/index.php?topic=8641.0)

Edit:: The issue with lgp in ff7tk is in how I've written the demo. I guess I should fix my demo.
« Last Edit: 2019-05-13 12:34:42 by sithlord48 »

picklejar

  • Crazy poster
  • *
  • Posts: 143
  • Karma: 10
    • View Profile
Re: PyFF7: Python 3 tools to manipulate FF7 files
« Reply #7 on: 2019-05-13 12:19:54 »
I have a snapshot of a pretty darn recent version of Aali's lgp utility (binary and source) here:

https://github.com/picklejar76/kujata/tree/master/lgp-0.5b

sithlord48

  • No life
  • *
  • Posts: 1504
  • Karma: 33
  • Dark Lord of the Savegame
    • View Profile
    • Blackchocobo
Re: PyFF7: Python 3 tools to manipulate FF7 files
« Reply #8 on: 2019-05-13 15:57:27 »
I've fixed my demo so ff7tk's LGP extraction is working correctly.

picklejar

  • Crazy poster
  • *
  • Posts: 143
  • Karma: 10
    • View Profile
Re: PyFF7: Python 3 tools to manipulate FF7 files
« Reply #9 on: 2019-05-14 03:22:09 »
niemasd: I can confirm that Aali's lgp utility can unpack the magic.lgp file. I just did it this morning and got over 5000 files including subdirectories.

niemasd

  • Fast newbie
  • *
  • Posts: 12
  • Karma: 3
    • View Profile
Re: PyFF7: Python 3 tools to manipulate FF7 files
« Reply #10 on: 2019-05-14 17:46:14 »
The unpacking is fine: it's the repacking that's the issue if I'm not mistaken. I think they all mess up with the Lookup Table. I'm not sure if this affects the actual file usage in FF7, but I remember trying to unpack and then repack without changing anything, and the Lookup Table was incorrect (it was mishandling files with a '.' in the first two characters, e.g. "a.rsd")

EDIT: I originally said "Conflict Table", but I actually meant "Lookup Table". The Conflict Table is handled fine in existing tools I've tried (it's the one that you need working to have files that exist in folders)
« Last Edit: 2019-05-14 20:02:35 by niemasd »

picklejar

  • Crazy poster
  • *
  • Posts: 143
  • Karma: 10
    • View Profile
Re: PyFF7: Python 3 tools to manipulate FF7 files
« Reply #11 on: 2019-05-14 18:23:03 »
Ah, okay, yeah, I don't have much experience re-packing. I wouldn't be surprised if re-packing magic.lgp causes issues.

Does anyone in this forum know if anyone has ever successfully made any mod that involves changes to stuff inside magic.lgp? (Not sure if the existing graphics overhaul mods included any changes to spell effects, for example?) If so, it would be good to know how they were able to re-pack the magic.lgp.

niemasd

  • Fast newbie
  • *
  • Posts: 12
  • Karma: 3
    • View Profile
Re: PyFF7: Python 3 tools to manipulate FF7 files
« Reply #12 on: 2019-05-14 20:01:32 »
According to this post, it seems like upscaled magic mods exist, but they use modpath (or at least this one does): http://forums.qhimm.com/index.php?topic=12836.0

niemasd

  • Fast newbie
  • *
  • Posts: 12
  • Karma: 3
    • View Profile
Re: PyFF7: Python 3 tools to manipulate FF7 files
« Reply #13 on: 2019-05-28 19:27:59 »
I've significantly updated PyFF7 since last time! It can now play with the following file types:
  • Field Files (just basic info for now; will later implement extracting and hopefully changing the image)
  • LGP Files (info, unpacking, and packing)
  • LZSS Files (decompress and compress)
  • NPK Files (info and unpacking)
  • TEX Files (info, converting to different image formats, and changing the image within)