[FF7] command line tools for LGP archives - LGP/UnLGP (v.5)

[Aali]

In light of the recently discovered flaw in LGPTools, I decided to write my own tools to deal with LGP archives.

"lgp" creates an archive from a directory, "unlgp" unpacks an entire archive to the current directory

There's no GUI and they probably don't even compile on windows, but I'm sure it will be useful to someone.

Full source is available here: http://backup.ninjaloot.se/share/lgp-0.5.zip

I will not provide binaries, so don't ask.


--EDIT--

Both links were dead even, see below for binaries

[nfitc1]

By "recently discovered flaw", are you talking about the multiple duplicate file names or the fact that the area between the file header and the file data is unknown?

[Aali]

By "recently discovered flaw", are you talking about the multiple duplicate file names or the fact that the area between the file header and the file data is unknown?

It's not unknown, LGP Tools tries to write it, and messes it up. Specifically, underscores and minus signs as the first or second character of the filename will result in a broken lookup table (thats what it is) because LGP Tools doesn't know that these characters should get some special treatment when it comes to the lookup table. (As can be seen in lgp.h)

[Millenia]

Can SOMEBODY please provide binaries? I'm a 3d artist, not a programmer, damn it

[nfitc1]

So I looked through it. You're saying that it's based entirely on the first two letters of the name of the file?

Also, your parts here:

Code: [Select]

158 toc.unknown1 = 14;
159 toc.unknown2 = 0;
160 toc.unknown3 = 0; won't crash the game (I don't think), but I believe they have something to do with the files themselves. That unknown1 may be something like file properties while 2 and 3 are together one value having to do with duplicate file names.
It'll work with battle.lgp because it only uses the name of the file and they're all unique. Have you tried this with the others?

[Aali]

The lookup table is based on the first two letters, yes.

I have yet to see an LGP archive where those unknowns don't look like that, I'm not sure FF7 is even using them. Duplicate filenames could never work because FF7 always opens files by name only. (So if you really did have two files with the same name, it would only ever be able to find the first one)

[nfitc1]

I have yet to see an LGP archive where those unknowns don't look like that, I'm not sure FF7 is even using them. Duplicate filenames could never work because FF7 always opens files by name only. (So if you really did have two files with the same name, it would only ever be able to find the first one)

While that first unknown is most commonly 14, I've also seen it as 11 and even 3 on one file (I think). I'm not convinced it's being used either.

magic.lgp has duplicates all over the place. Open it in a hex editor. The two of the first three file names are "a.rsd". There are over 650 duplicate names (not to be confused with 650 files with the same name). This could possibly be explained that there are thousands of directories referenced by name in that same file. It's likely that some part of that table is referencing each file to one of those directories.

[Aali]

Hmm, it does seem like I'll have to rethink my approach to this. FF7 seems to look for some files using two names, presumably one is used for the lookup table, while the other is the actual filename from the TOC.

Unfortunately this means that some of the original information was lost in the conversion to lgp archives, since there's no way to figure out which name is actually used to access a file, save for the first two characters used in the lookup table (and even then you have a conflict between k/l and _/-).

Expect a new version up soon that will handle this properly. For now, don't try to repack magic.lgp (field, char, battle and menu should work fine, the others I have not tested)

:EDIT:

Turns out my function for computing lookup values was still a bit off, numbers and periods were not handled properly. With that fixed, my previous theory about the lookup table still holds, the two first characters of the filename are used to find the file, nothing else. Unfortunately, that leaves the duplicate filenames mystery without a reasonable explanation. If only the filename is used to to find them, FF7 will simply open whichever file comes first in the TOC, leaving the other file inaccessible.

:EDIT #2:

Good news everyone!^TM LGP files that do have subfolders provide a neat table of what goes where Give me some time with this and soon enough these tools will be able to create/extract any kind of LGPs

:EDIT #3:

Okay so it doesn't say exactly what goes where, it only specifies the folder name whenever there's a name conflict. Still, it's enough to (re)create a 1:1 copy of any existing LGP just from filesystem data.

:EDIT #4:

Only magic.lgp has name conflicts and the unlgp tool can now handle that properly, while the lgp tool is still missing some things. I also kind-of figured out what the last unknown means (unknown 2 & 3 is the offset into the conflict table), it's only ever 11 or 14, where 11 means it's a text file and 14 means it's a binary file. The game does not care about this field however, so just setting it to 14 for everything is fine.

[nfitc1]

Two questions:
1. How are 2/3 handled? Every file that shares a name with other file(s) also share this value.

Ex:
There are four "01.rsd" files in magic.lgp and they each have a 2/3 of 2. There are also four "01.p" files and they each have a 2/3 of 3. How does this get resolved exactly?

2. I'm also not certain that the unknown 1 is binary/text. The file deathlv5.hrc in magic.lgp was the first "11" file I found and it's the only .hrc in the magic.lgp. It is text, that's true. But then all the rsds would be text too and most of the .hrc files in char.lgp are "14" files. I do agree that the game doesn't care about this value, however.

It's not that I don't believe you, I just want to make sure I understand it all.

[Aali]

Well I just figured the binary/text detection method was a bit crappy While a bunch of text files are set to 14 as you say, I have not found any binary files set to 11.

unknown2/3 are actually one 16-bit value, an offset into the conflict table. Every offset into the conflict table has two or more path names, and each path has a TOC reference. If the path matches with the file you're trying to open, use that TOC entry, otherwise try the next path.

The conflict table is stored after the lookup table, starting with one 16-bit value indicating the number of conflicts present. For each of these, you first read another 16-bit value, this tells you how many paths are present for this conflict, then you read that many paths and their TOC indices (another 16-bit value).

It looks like this in the file:

[Number of conflicts (Let's say it's 2)][Number of paths for conflict 1 (Also 2)][<path>][<TOC index>][<path>][<TOC index>][Number of paths for conflict 2 (This one is 3)][<path>][<TOC index>][<path>][<TOC index>][<path>][<TOC index>]

FF7 then stores this information in a table of 4096 possible conflicts, each with a maximum of 255 paths.

[Aali]

Check the repository for release 0.2b

This one should be able to unpack and repack any LGP archive in the game, including the elusive magic.lgp

As an added bonus, you can now arrange files however you want in whatever subfolders you want, save for the conflicts in magic.lgp which have to stay in the right folder for FF7 to find them. (So, for example, you could arrange battle.lgp or char.lgp so that each separate model has its own subfolder, this will not affect the resulting LGP in any way)

[Fleet Command]

Hi, Aali. Thanks for the new tool. I can't wait to compile and try it out. (Or at least I can't wait to find out what it is, how to compile it and how to try it out.) I'd highly appreciate if you give some insight as to the programming language with which the tool is written with or how to compile it. I'm looking at it and it is certainly neither of the programming languages that I know. (Yes, it looks like a bred of C but it is neither C# nor C++ Builder.)

[Aali]

It's just regular C. It was written for a linux environment, but it will work fine in cygwin. You could also make some small adjustments to make it work with MSVC/whatever windows compiler you might have.

[Fleet Command]

So, let me get this straight: I should rewrite your code and then compile it in Windows and do that every time you release a new version? Or is there a way to compile your code, as is, and then run it in Windows? Then, where does this Cygwin come into play?

By the way, I have question that might look dumb, so forgive me if I'm such an idiot, but why did you ever create a native Linux tool for a Windows game in the first place?

Thanks in advance.

[Aali]

Rewrite? You could probably make it windows-compatible with a single preprocessor macro

Cygwin is a complete unix environment for windows, so pretty much anything that compiles on linux will work with cygwin.
It would be quite silly to install cygwin just to run this one thing though, so if you don't have it, don't bother.

Why dev this on linux? It was convenient for me.

[nfitc1]

By the way, I have question that might look dumb, so forgive me if I'm such an idiot, but why did you ever create a native Linux tool for a Windows game in the first place?

'cause any of the good windows-based development platforms cost oodles of cash. Linux and Unix can compile C/C++ code just as well for free from just a simple text file.
I only have the 2008 .NET suite because I'm fortunate enough to work at a higher-education institution and we get MAJOR discounts on such software like that. There's no way I'd buy that professional license trash when the educational one works just fine for my needs.

[Fleet Command]

Rewrite? You could probably make it windows-compatible with a single preprocessor macro

Cygwin is a complete unix environment for windows, so pretty much anything that compiles on linux will work with cygwin.
It would be quite silly to install cygwin just to run this one thing though, so if you don't have it, don't bother.

Why dev this on linux? It was convenient for me.

Thanks a bunch. I'll see what I can do. If I managed, do you mind if share the binaries with the rest of Qhimm community under Attribution-Share Alike-No Derivative Work terms?

By the way, I have question that might look dumb, so forgive me if I'm such an idiot, but why did you ever create a native Linux tool for a Windows game in the first place?

'cause any of the good windows-based development platforms cost oodles of cash. Linux and Unix can compile C/C++ code just as well for free from just a simple text file.

I'm afraid you are wrong. Visual Studio 2008 Express edition, CodeGear Delphi Lite and CodeGear C++ Builder Lite are all available free of charge. These free environment are fully capable of developing Windows applications.

Their only differences with their commercial editions are (1) lack advanced components such packages for developing Oracle database applications, Microsoft Office applications, MIDAS applications, 64-bit applications, mobile device applications, web applications (PHP or ASP.NET) as well as (2) advanced features such as source control, server development environments, profiling, advanced database connectivity, etc..

There are also other open-source development environments. We have lots of Java-based IDEs and compilers. There are myriads of C breeds.

Aali's answer however, was the logical and the acceptable one. Convenience is very important in software development. It simple takes millenia to develop a simple application with an inconvenient environment.

[Aali]

You can do anything you want with it

I would like to point out that Visual C++ Express lacks MFC, which is bad news for some people.
However if you're starting up a new project I don't see why on earth you'd choose to use MFC instead of a more portable API.

[arew264]

Eclipse + CDT + MinGW FTW!
I find this convenient because the same setup can be had on Linux, where it will use good ole GCC.

It is true that development is generally faster on Linux, if only because in many ways C/C++ in Windows are... Microsoft-ized. They add "safe" variants of the standard C library functions and then yell at you for not using them, and preprocessor directives must be used way more aggressively. Or such was my impression.
On the other hand, Linux development libraries can be installed quickly and painlessly, and you can have a full development environment inside of 10 minutes. None of this "ensure that your copy of Windows is genuine and download the Windows SDK" nonsense.

[Fleet Command]

  Spread your Anti-Microsoft or Anti-Linux propaganda somewhere else!  It is against the forum rules to verbally abuse either Microsoft or Unix-based products.

Besides, this thread is about Aali's LGP tool. If you have any question, suggestion or commentary about anything related to it, you are more than welcome to post. But anything else, especially verbal abuse, fancruft propaganda and the troubles of using pirated copies of software, is unwelcome.

[arew264]

That wasn't verbal abuse, that was simple statement of fact. Other programmers will say the same thing. I have respect for Microsoft, if only because Windows, as much as some people dislike it, is an excellent operating system with an uncompromising approach to usability. I love Linux, but it isn't for everyone. Windows has achieved that.

The question was raised as to why this tool hasn't been developed for Windows, and I was corroborating previous statements that development is easier on Linux. It is; that's the simple fact. If you think stating that is verbal abuse, then you're going to be offended for most of your life. I may have been harsher than I intended when I mentioned the SDK, but it's entirely true. To develop a Linux application, you download a few megabytes of development libraries and compilers. To develop on Windows, you download a DVD image (if I remember correctly, it was a full size DVD image) and use it to install the Windows SDK. This is a significant hinderance to development.
The other thing I left unsaid is that I find if easier to use the GCC MinGW compiler (basically GCC ported to Windows) to develop Windows applications than it is to develop them using Microsoft's tools when you plan to use good ole C rather than Microsoft Visual Basic or C#.

Now, if this is straight up C, I could look into porting it to Windows and possibly posting binaries if that would be useful.

EDIT: Also...

If you have any question, suggestion or commentary about anything related to it, you are more than welcome to post. But anything else, especially verbal abuse, fancruft propaganda and the troubles of using pirated copies of software, is unwelcome.

I could almost accuse you of verbal abuse for accusing me of being a verbal abuser, a propagandist, and a pirate in the same statement. I made it quite clear that DEVELOPMENT is easier on Linux. I never recommended it; I never said it was superior to Windows in any other way.
In fact, my first statement was a recommendation for an alternative development environment for Windows.

[obesebear]

I don't think there is any doubt that you didn't verbally abuse anything. 

@FleetCommand, are there really forum rules against verbally abusing Microsoft or Unix?  I honestly never read the rules    I just lurked 6 months before posting.

Back to on topic, arew264, you said you think you could make a binary of this?

[Kranmer]

i have just built this and it works great (on XP SP3 anyway), i dont seem to be getting any errors in the app.log now looking for files that are there in flevel.lgp, thanks alot for this Aali it will be especially usefull now palmer is at the import+export stage.

EDIT:-

LGP 0.3b binaries compiled by me can be gotten here
http://www.mediafire.com/?iyvidgtm3y2

[m35]

Also just built these with MinGW on WinXP SP3 without errors. Didn't even need MSYS. Executed both lgp.exe and unlgp.exe without any arguments and they also ran without errors.

Code: [Select]

C:\lgp\make
cc -Wall  lgp.c -o lgp
cc -Wall  unlgp.c -o unlgp

C:\lgp\lgp
Usage: lgp <directory> <archive>

C:\lgp\unlgp
Usage: unlgp <archive>
Uploaded the 0.2b binaries here. They will be available as long as Google Page Creator is still available (it was supposed to shut down last month).

I also want to ensure no one is of the the misconception that VS Express, or any Windows dev environments, can be used without cost. They are dependent on other software that has a price tag of at least $100USD.

Edit:Oops, I am incorrect. My binary needed cygwin.dll. kranmer binary doesn't, which is much nicer.

[Aali]

You should probably bundle the required .dlls as well.
I don't think people with cygwin already installed are going to need those binaries