[FF7] Mass field dialogue extractor/importer - FieldTool (0.9.0)

[mav]

Update: Version 0.9.0 released (30.10.2007)

[story of my life]

Hey everyone, long time no see.
I was around here some time ago with 'M4v3R' nickname, but I don't remember my password now and the it looks like that I don't have access to my previous mailbox (don't even know which one it was).

Nevertheless - I wrote few programs in Delphi for FF7 to help translating it to my native language (Polish). Sapphire - main program for translating dialogs was a bit buggy with some levels. Unfortunately (or fortunately ) I don't use Windows anymore so I can't fix it. So I've decided to write a new one - simpler one too so it would be less prone to errors. It's written in Mono C# (uses .Net Framework version 1.1), works in Windows and Linux as well and is a command line tool.

[/story of my life]

FieldTool does two things:
- Export function: it takes flevel.lgp archive, unpacks field files, extracts dialogues from them and puts the dialogues in nicely formatted txt files for you to read/modify.
- Import function: takes a directory with txt files created by export function and inserts the dialogues into field files, then repacks them to the flevel.lgp archive.

Usage is as follows:

Code: [Select]

Usage: fieldtool.exe <MODE> <LGP-FILE> <TXT-DIR> [FILE(s)]

MODE - one of following:
       e - export dialogues from archive
       i - import dialogues to archive (not implemented yet)
LGP-FILE - lgp file to work on
TXT-DIR - output/input directory
FILE(s) (optional) - work only on these field files (separated by space)
For example if you have the program in your FF7 directory you could use it as follows (create the 'txt' directory first!) to extract dialogues:

Code: [Select]

E:\FF7\fieldtool.exe e data\field\flevel.lgp txt
Now you can modify dialogues in txt directory, and if you want, say, import only 'astage_b' file (because you were playing with Event Square dialogues):

Code: [Select]

E:\FF7\fieldtool.exe i data\field\flevel.lgp txt astage_b
The downloads:
- FieldTool v0.9.0 (9.7 kB)
- FieldTool v0.9.0 source files (Monodevelop project) (14.5 kB)

Release notes for v0.8.1 - v0.9.0
Coding late at night has its effects, v0.8 was very buggy when importing files to LGP. Fixed every one that I could find.
Import speed improved (see speed report in posts below).
Special characters handling improved.

Some random notes:
- For decoding text it uses a separate file - 'fieldtool.tbl' - containing table with all possible characters (00 - FF) plus a few special ones (that are after 0xFE character).
- I've tested it with couple of level files, modifying texts (so size was bigger or smaller) and re-importing it. It worked, but I don't guarantee that it'll work correctly for You so please backup your flevel.lgp file before testing it! Knowing my (little) experience in writing programs for Square games there will be files that are buggy; maybe there are other bugs - I finished writing this after ~10 hours of coding in one day!
- Exporting works quite quick, whole archive under 20 secs on my 1.6GHz laptop; single files export is a fraction of second. Importing could use some speed up though (approx. 1 second for each file). I will work on it when I'll be certain that there are no major bugs.
- LZSS routines were written based on Qhimm's Garden sources. Thanks Qhimm! . LZSS encode isn't as efficient as the original alghorithm though, so compressed files are ~10% bigger than original compressed ones.
- For speed purposes, files are put always at the end of LGP archive. It means that it'll grow bigger everytime you import a file to it.
- Quite a few files have an "akao [...]" dialogue at the end. It's a bug of Square it seems, but don't worry about it. Just don't touch this dialogue and it should be all right.

---

Comments are welcome . Please note that this software comes without no guarantee - backup your flevel.lgp file before playing with it!

And to answer question "Why write another tool for FF7 Dialogues" - there is no single tool for this task that is bug-free. Cosmo has some weird bugs, Sapphire has them also. Loveless, latest program that I saw looks promising, but still as reported by my friend it likes to crash a lot, and doesn't support mass exporting/importing. So I try to fill that hole at last .

[dziugo]

Just a thought - it'd be a weird thing to use dotNet just for this kind of portability. If it's meant to be only a command-line utility, just compile 2 versions (one for Windows, and one for Linux) written in a language which doesn't require any framework. A program with a GUI and an option to run in command-line is a different thing, though...

Subforum names could be deceiving, but IMHO, this belong to Game Tweaking.

As for your older account, why not contact the admin?

[mav]

Just a thought - it'd be a weird thing to use dotNet just for this kind of portability. If it's meant to be only a command-line utility, just compile 2 versions (one for Windows, and one for Linux) written in a language which doesn't require any framework. A program with a GUI and an option to run in command-line is a different thing, though...

To be honest I don't want to focus on how to code something, but on what I want to code. I could use C++ and compile it separately for each platform, but it would definately take more time (because I don't know C++ that good and because I don't have direct access to windows mashine). As for GUI - yes - maybe someday I'll add it, and then having existing C# code will be helpful.
And .Net v1.1 is only a 11MB download .

Subforum names could be deceiving, but IMHO, this belong to Game Tweaking.

They are . If it's in the wrong place, then could some mod move it?

As for your older account, why not contact the admin?

Yes, maybe I should, but: "Hello, I lost my password and my email account for username xxx, can You change my password to xxx?" - doesn't sound a little fishy to You?

[dziugo]

You can compile a Windows version from Linux, .NET framework is a part of automatic updates on Windows (so it's not actually a problem) and I think that you and Qhimm know each other to the point where he can think of some nice solution for your account recovery

[mav]

Well, what do You know, the activation email eventually arrived. So I can use my old account now! . The previous one is now unneccessary, but I don't know what'll happen to this thread then [/offtopic]

As for programming language, I think I'll stick with .NET nevertheless. If .NET is a part of automatic updates that's even better.
Right now I'm writing import function, when done I'll post it here.

[mav]

FieldTool v0.8 is out!

Now supports importing txt files back to LGP archive! Links and release notes are in the first post.
Please test the program if you can and post your comments here, thanks!

[Ragna]

- LZSS routines were written based on Qhimm's Garden sources. Thanks Qhimm! . LZSS encode isn't as efficient as the original alghorithm though, so compressed files are ~20% bigger than original compressed ones.
- For speed purposes, files are put always at the end of LGP archive. It means that it'll grow bigger everytime you import a file to it.

Maybe you should fix that in the next update... MAINLY.

[mav]

I don't see growing archive as a major problem. Today, with having 100-300GB hard drives 100 MB is nothing. And this is approx. cost of importing *all* files again into the archive. Maybe I'll add an option to repack the archive from the beginning, but that will be an option if someone wants to wait more while importing.

[Ragna]

If the files are too big, the game will crash, that's it.

[mav]

Well, I didn't knew that. Do you know how big LGP file can be for the game not to crash?

[Ragna]

Nope.

[mav]

I recall that it could be ~150MB and game didn't complain, but I don't know how it'll behave if the file is bigger. Oh well, I'm working on the algorithm right now to make it more efficient, it looks like that Qhimm didn't implement negative offsets and repeated runs in his LZSS code, I'll try to do this myself.

[mav]

FieldTool v0.8.1 is out!

This is mainly a bugfix release. I forgot a few things while writing LGP file structure, also LZSS encode algo was a bit buggy (wrote control bytes at the end of file even though there wasn't any more output). Links are as previous - in first post.

PS. Working on improving LZSS Encode algorithm, now the ratio is ~60% (original compression has ratio at ~55%), so I'm really close .

Edit: There are still a few quirks, please wait for v0.8.2 before using it for importing.

[mav]

FieldTool v0.8.2 (a.k.a. "Finally working") is out!

Again, this is a bugfix release. With it I was able to change dialogues in few locations and play the game with no problems. Maybe some other levels will be problematic, but that's left to find.

Edit: I have successfully edited and played both a couple of blin files (66, 68 floor), and frcyo (Chocobo Farm) files. I also played through the beginning of game 'till meeting Aerith. It looks that everything's working fine .

Edit2: Another bugfix release - v0.8.3 - is out. It seems that previous version wasn't able to import dialogues to field files without AKAO sections, and that's fixed now.

[mav]

FieldTool v0.8.5 is out!

You missed one version because it was internal, with me and my fearless tester - kruci - ironing the bugs in it.
This release brings up two things:
- the usual bug cleaning
- import speed improvement - by about 70%!

I managed to do a small speed report:

Test machine:
- Intel M 1.6GHz
- 768 MB DDR2 RAM
- SATA 60 GB drive

Test results:
- Exporting all files from flevel.lgp: 35 secs
- Importing all files back to flevel: 7:15 min

I think these results are pretty good. Now the room for improvement is in output flevel.lgp size, right now after importing all files it is ~270 MB big, I'm not sure if FF7 will read it correctly (I'll check this tomorrow).

Anyway, links are, as always, in the first post.

[Scorpicus]

Using the export_all.bat/import_all.bat I've been able to alter the text without any issues at all.  However, whenever I tried to open the "fieldtool.exe" the thing came up and then crashed in about a second - it does display some sort of error message but it leaves the screen too fast for me even to press "Screen print" .

[mav]

The fieldtool.exe program itself does nothing, only prints out its usage. If you want to use it in this way (you can for eg. import/export single file, not whole archive), open it in command line window (Start -> Run -> cmd).

Did you test if the game itself works for you?

[Scorpicus]

Ah, I see.  ( at myself).

I used it to export all the text from an flevel.lpg that had the Sephiroth/Cloud name swaps to pack with the small version of the patch.  I tested importing all the txt files back into the game and it worked great 

[mav]

Cool, thanks for testing.

-------------------------------------

Version 0.9.0 is out!

Every known bug has been fixed. Links are in the first post. Enjoy .

-------------------------------------

If the files are too big, the game will crash, that's it.

It has been reported recently (and I've checked it myself) that even with all files reimported and nearly ~300 MB flevel.lgp file, game still runs fine .

[LadyUmiko]

Hey... just to let you know... the links are broken. The files can't be found.

[mav]

Hello,

Thanks for reminding me that. Links are now fixed .