Author Topic: ReWriting the FF7 Engine (Read 21289 times)

Darkdevil · « **Reply #25 on:** 2004-06-28 16:35:06 »

Well I created a clip-art Veiwer in Vb6 this morning in my 6th Form induction day...Other than that, I have ne means of doing ny programming as i have no applications to do so...

Goku7 · « **Reply #26 on:** 2004-06-28 17:40:59 »

Quote from: mirex

But if you really want to, then i could give you few hints. Are you skilled in programming ? If not then that is the best way to go. C++, Pascal or Visual Basic (which is not much recommended, but works ) ... choose any and go for it. Try to program something that messes with files, anything.

By the way, Mirex....what about the code I posted earlier up on the page? Granted, a lot of variables and stuff are just being used as placeholders for the real data types and such that would be used, but would that not be a start for getting direct access to specific models without decompressing the LGPs during run-time of this hypothetical re-written engine?

Looking back through some other threads, it appears there are a lot of talk about offsets to stuff. But what are the offsets relative to? The beginning of the individiual .p files?

Actually, perhaps the iidea behind the earlier code I posted could demystify some data tables that haven't been figured out, they could contain offsets that are only valid when looking at the compressed code of the LGP file, and not the .p files themselves.... resulting in a funny parallel:

LGP file == PSX data that is not stored in the normal CD filesystem
offset tables == coordinates for PSX CD Direct-Sector reads.

Heck, no one's even told me if what I posted up there or here is old ground or not....

ficedula · « **Reply #27 on:** 2004-06-28 18:06:53 »

LGP files aren't themselves compressed in any way. The FF7 engine can read data straight out of them if necessary ... for that matter, so can the Remake engine (not that I've touched that in ages). LGP files just provide a way of packaging up lots of files into one big chunk, so the OS (Windows) doesn't need to bother about creating 20,000 files itself.

The offsets we're talking about are generally relative to the beginning of the individual file in question (such as a P file), yes. I don't think you should give too much thought to the LGP file, since it's essentially just a container, nothing relies on being present in a particular location within the LGP (proof: you can add new data to an LGP file using LGP Tools and FF7 doesn't care *where* it is - even if it's now at the end of the archive when it was originally at the beginning - so long it's there and listed in the contents of the LGP).

Anyway. Yes, most tools copy (not decompress) the data from the LGP into memory and then use it from there, rather than reading direct from disk, but we don't *have* to do that. There's little reason not to, though, the end effect should always be the same.

halkun · « **Reply #28 on:** 2004-06-28 22:51:02 »

Quote from: Goku7

LGP file == PSX data that is not stored in the normal CD filesystem

PC Data != PSX data.

More importantly:
Raw Psy-Q data != FF7-PSX data != FF7-PC data.

The PSX deals with raw Psy-Q data. (TIM, RSD, SEQ, STR, TOD, HRC, etc etc etc) These are munged and stored in special formats for FF7-PSX. (BIN, LZS). The PC version has conversions of the PSY-Q data into PC native data. (SEQ=>MID, STR=>AVI, Binary HRC=>Plain Text HRC)

The only thing that should be platform independant is the core of the kernel. It takes a lot of bit twiddeling to get the data into something that the platfrom can easily handle and that the user can understand.

Goku7 · « **Reply #29 on:** 2004-06-29 00:13:03 »

First off -- Long time no see, Fice.

Quote from: ficedula

LGP files aren't themselves compressed in any way. The FF7 engine can read data straight out of them if necessary ... for that matter, so can the Remake engine (not that I've touched that in ages). LGP files just provide a way of packaging up lots of files into one big chunk, so the OS (Windows) doesn't need to bother about creating 20,000 files itself.

Oh, so they're NOT compressed....then that should make the re-writting of the engine that much easier, as then it would just need to deal with offsets w/ no real "decompression" per se.

Quote from: ficedula

The offsets we're talking about are generally relative to the beginning of the individual file in question (such as a P file), yes. I don't think you should give too much thought to the LGP file, since it's essentially just a container, nothing relies on being present in a particular location within the LGP (proof: you can add new data to an LGP file using LGP Tools and FF7 doesn't care *where* it is - even if it's now at the end of the archive when it was originally at the beginning - so long it's there and listed in the contents of the LGP).

You know....that makes sense.

After I posted the code earlier, it hit me that there just might be a table of contents in the LGP file. On the other hand, I didn't bother to mention it because I had this feeling that all this was indeed old ground.

Either way, thanks for confirming my suspicion, LOL.

I guess that the existance of a table of contents in the LGP files would be how the current FF7 engine could cope with drastically modified LGP files. This would mean that instead of having a table of contents hard-coded into the .exe file (which is what my code example assumed), it would perform the seekg commands using variables previously loaded with offset values from the table of contents.

But therein lies a potential problem, though: The offsets needed to find the table of contents may be hardcoded into the .exe file anyway. In all probability, if there are hardcoded offsets that determine the table of contents' beginning byte and ending byte in the LGP file, it would put a limit on how many files each LGP file can hold and still be recognized by FF7, due to the tunnel vision on the size of the contents listing that the hard-coded offset values are causing....right?

Quote from: ficedula

Anyway. Yes, most tools copy (not decompress) the data from the LGP into memory and then use it from there, rather than reading direct from disk, but we don't *have* to do that. There's little reason not to, though, the end effect should always be the same.

At least my train of thought was on the right track. Yay!

It also gave me an idea regarding how to find where the battle animations are stored. Have you (or anyone else for that matter) tried to do a memory dump of the video card RAM during a battle?

My thinking is this -- most likely FF7 sent the animation data to VRAM instead of the system RAM along with everything else -- and probably as a series of struct arrays as well. Furthermore, even IF it is sent to system RAM first, it is inevitable that it will be transferred to VRAM at some point, and thus we can be sure it's hiding in there.

Assuming we can get a dump of the the VRAM, we could compare the information in there with the data in the LGP files, and eliminate anything that corresponds to data we already know to not be the battle animations. Any sequences of data that don't appear to be reconizeable to us should have its corresponding sequence in the LGP files noted for later reference as a possibility for the animation sequences.

If there's only one unique set of sequences of unrecognized data, then....well there ya go!!

Quote from: halkun

PC Data != PSX data.

More importantly:
Raw Psy-Q data != FF7-PSX data != FF7-PC data.

The PSX deals with raw Psy-Q data. (TIM, RSD, SEQ, STR, TOD, HRC, etc etc etc) These are munged and stored in special formats for FF7-PSX. (BIN, LZS). The PC version has conversions of the PSY-Q data into PC native data. (SEQ=>MID, STR=>AVI, Binary HRC=>Plain Text HRC)

The only thing that should be platform independant is the core of the kernel. It takes a lot of bit twiddeling to get the data into something that the platfrom can easily handle and that the user can understand.

I know that, but when I wrote what I wrote, it was supposed to be a metaphor, not a literal statement that the code was the same bit for bit, 'cause its not. Just that the philosophy in how the engine probably looks at how the code is stored is probably the same in both versions.

I.E., those PSX direct sector reads are analogous to the PC version taking an LGP file and directly grabbing the code it needs from it, and then decompressing, using the offsets the same way the PSX uses a list of coordinates to do a direct-sector read from the disk.

I probably should have been a bit more concise when I wrote that, LOL.

halkun · « **Reply #30 on:** 2004-06-29 08:49:25 »

Quote from: Goku7

My thinking is this -- most likely FF7 sent the animation data to VRAM instead of the system RAM along with everything else -- and probably as a series of struct arrays as well. Furthermore, even IF it is sent to system RAM first, it is inevitable that it will be transferred to VRAM at some point, and thus we can be sure it's hiding in there.

Nope, it's most definatly not in there....

The video ram hold textures, texture caches, frame buffers, and palettte information. You can dump the VRAM from a PSX emulator and see for yourself...

Because the code is PSX native there's also a nother reason why haveing data in VRAM is silly.

The PSX has *no direct access to VRAM*

You can't put data into vram using a memory address as the video memory is not directly attached to the PSX's CPU bus. You have to send data using a GPU packet (Command + arguments) like "Draw triangle", "set viewport", or "load texture" That's why TIM files are used so often. They are pre-formed data that the GPU can accept.

You either put the data into a GPU register, or stream via the GPU's DMA hook. (Meaning you allow the GPU Direct Memory Accees to the PSX's main RAM, but you can't do it the other way around) The GPU has no concept of "Animation" or anything like that. It't can't even do math. That's the GTE's job and that's on another bus all togeather

Goku7 · « **Reply #31 on:** 2004-06-29 16:17:45 »

I was refering to the PC version......

ficedula · « **Reply #32 on:** 2004-06-29 16:28:23 »

Unfortunately still isn't going to happen. Modern graphics cards can upload animations and geometry to VRAM, but for that to be *any* use whatsoever the card needs to support hardware T&L at an absolute minimum (and realistically, it needs to have other features too). When FF7 came out, they didn't, so it's not written to upload animations to video cards.

It's a fairly recent development, allowing you to do that, anyway. PC users haven't been able to explicitly upload vertex data to the card for that long, nevermind the actual animations ... all possible now, mind you. If you have a recent Radeon/Geforce, of course, and a game written to support it...!

Back to the older stuff...

Quote from: Goku7

First off -- Long time no see, Fice.

Thank you

I've been busy with my job ... not a new job anymore, but still, first full time job. Warning: jobs eat your time.

Quote

But therein lies a potential problem, though: The offsets needed to find the table of contents may be hardcoded into the .exe file anyway. In all probability, if there are hardcoded offsets that determine the table of contents' beginning byte and ending byte in the LGP file, it would put a limit on how many files each LGP file can hold and still be recognized by FF7, due to the tunnel vision on the size of the contents listing that the hard-coded offset values are causing....right?

Nope. The only thing 'hard coded' is the point at which the table of contents *begins* (essentially the first piece of data in the file), not its length.

That said, the file format uses 16-bit integers for some of the data, so while the file size of an individual LGP could go up to at least 2GB, you'd be limited in practice to ~65,500 files inside it. Perhaps double that if you played some nasty tricks on the indexes.

Of course, that's fairly academic; even battle.lgp doesn't come anywhere near that number of files, so you've got plenty of room to add your own files.

Darkdevil · « **Reply #33 on:** 2004-06-29 17:01:04 »

IIRC the most files in an lgp is close to 11,000

Its is either the feil or battle LGP file....im going for the latter.

Goku7 · « **Reply #34 on:** 2004-06-29 22:02:20 »

Quote from: ficedula

Unfortunately still isn't going to happen. Modern graphics cards can upload animations and geometry to VRAM, but for that to be *any* use whatsoever the card needs to support hardware T&L at an absolute minimum (and realistically, it needs to have other features too). When FF7 came out, they didn't, so it's not written to upload animations to video cards.

It's a fairly recent development, allowing you to do that, anyway. PC users haven't been able to explicitly upload vertex data to the card for that long, nevermind the actual animations ... all possible now, mind you. If you have a recent Radeon/Geforce, of course, and a game written to support it...!

Hmm.....I see. However.....

If I'm not mistaken, whatever animations that are about to be drawn on the screen are first transfered to the framebuffer, which should be in the VRAM, so technically the data would in one form or another pass thru the VRAM before going on screen. But, it just occured to me that the data might have already been processed into something other than its original form by that time, so maybe a VRAM dump might not work as I planned.

IIRC, when the case is that no HW TnL is available, then the CPU and system RAM must then caculate the Geometry Transformation instructions before passing it to the framebuffer in the VRAM....so doing a system RAM dump might still be an acceptible lead?

Of course, we'd have to do this on a system with the minimum RAM necessary to get the game to run, as the last thing we need is to go through 1/4 to 1/2 a gig of data looking for the animations.

Basically, I'm saying that since we haven't been able to locate where the animation data is stored, we let the game do it for us, and then we take a picture of what it grabbed from the LGP, and then we go back and cross-reference the data there with what's in the LGP.

There is another remaining point that we ought to remember. Thanks to what halkun dug up regarding what FF7pc was compiled from, we know that FF7pc was written in C++. Therefore, we must think within the limitations of that language when trying to reconstruct how FF7 can access and decode battle operations, etc, and that includes file I/O, and memory storage.

That leaves us with commands like ".read"; ".get"; and ".getline" for loading the battle animation data into memory, if I'm not mistaken. And considering that the data is probably a series of values, it would have to be loaded into some sort of data array, which in turn means the entire data sequence for the animations would be occupying adjacent memory addresses, meaning a RAM dump just might happen to have the battle animation data all conveniently lumped together in one section of memory addresses, instead of being scattered all over the place. This would allow us to very quickly narrow down which sequence it is....I think.

Then again, I do have to admit I've only taken an introductory course in C++, so maybe I'm being too naive or something in regards to how the thing could be coded........

Quote from: ficedula

Thank you I've been busy with my job ... not a new job anymore, but still, first full time job. Warning: jobs eat your time.

Well, congrats on that. If all goes well, I also just might get a full time job in a computer store.....I'm just waiting for them to review my application/resume, and make a decision....

Quote from: ficedula

Nope. The only thing 'hard coded' is the point at which the table of contents *begins* (essentially the first piece of data in the file), not its length.

Well, in that case, the beginning of the table of contents isn't hard-coded as an offset either.

But how would they figure out where it ends?

.....oh, duh. There's probably a delimiter character in there. Man, do I feel stupid.

Quote from: ficedula

That said, the file format uses 16-bit integers for some of the data, so while the file size of an individual LGP could go up to at least 2GB, you'd be limited in practice to ~65,500 files inside it. Perhaps double that if you played some nasty tricks on the indexes.

Of course, that's fairly academic; even battle.lgp doesn't come anywhere near that number of files, so you've got plenty of room to add your own files.

Great. I had the right conclusion (there's a limit to how many files you can put in an LGP file), but got to it by a completely wrong line of reasoning....what are the chances of that?

But still....who in their right mind would need to have 65,500 files for a program?

Quote from: Darkdevil

IIRC the most files in an lgp is close to 11,000

Its is either the feil or battle LGP file....im going for the latter

11,000?! Ok, the programmers need to be taken into the backyard and shot one by one on behalf of the KISS principle.....

Darkdevil · « **Reply #35 on:** 2004-06-30 08:14:26 »

The exact number.

halkun · « **Reply #36 on:** 2004-06-30 08:42:32 »

Keep in mind they are 11,000 really tiny files. some are just a small 32x32 4 bit pictures and such.

Cyberman · « **Reply #37 on:** 2004-06-30 15:56:05 »

Quote from: Goku7

11,000?! Ok, the programmers need to be taken into the backyard and shot one by one on behalf of the KISS principle.....

That's quite unrealistic
Here is a dosage of reality here. Take a game like NWN you have say 3500 scripts 1200 models etc etc. Now each model consists of about 10 different files.. are you following me yet?

Let me change the way you look at things. If you were to see Windows Source code, think abou tthe number of components and different files that consist of windows. The programers wisely instead of having 11000 different files archived them into a singular file they could extract the data from, however originally they did NEED the 11000 files because they had to be made.

They didn't just WISH the LGP contents into being

A group of artists and scriptors were busily making the contents of that. Hence 11000 is not rediculous.

What they did do was KISS they centralized the data and kept information of where things were.

I'm sure FF8 has similar volumes of data in it (and I'm making slow progress on finding the sections of data heh, I have the world textures found already I just need to find the topographic data).

An example is one FF7 Battle mode on the PS1l for cloud.
The one file contains 113 sections
Section 1 consists of 23 bones six of which are joints. That means it has 17 models with vertex pools triangles quads with colored vertices and triangles and quads with textures.
Section 97 is the texture data
Section 98-113 are the weapon models.
Just one single file in reality consists of more than a hundred smaller files originally.
More importantly they HAD to have the 100+ original files and you can't just compile them to one file and throw them out (it will loose information).

There are 300+ enemys the average enemy has 10 bones lets say (no texture) that's 1 HRC and 10 RSD files per enemy, that doesn't include the animation information (there are at least 1 animation but generally there are 5) then a TIM is always include (irregardless if it's used). that's 16 at minimum original files. Well over 4000+ files just for battle models alone.

There is just TOO much information to make it super simple so the programers did the next best thing they archived it

Cyb

Goku7 · « **Reply #38 on:** 2004-07-03 01:06:48 »

First, I'm not knocking the fact that they archived it -- I think that's smart, and that more games should nowadays have as much stuff as they can be compressed w/o too much performance loss, instead of unloading 5,000 .bmp files for textures and raw .wav audio for sound.

But that aside, I realize that most of the time a game or powerful program's source code is made up of large numbers of files. However, when I said what I said, it didn't cross my mind that I should have considered those 11,000 files to be in the same vein as the source code, as you essentially pointed out.

However, I was also under the impression that even the final amount of files they ended up with in the LGPs was still needlessly split up past the point of simplicity -- like, for example, the fragments of previous threads on the files and how the engine uses them made it sound like the game had 300 separate files it had to access and load separately, JUST FOR CLOUD'S HEAD. That's what I thought was going on, and that's why I was saying that the programmers didn't seem to be following KISS.

I guess I was mistaken, though.....

trickstar66 · « **Reply #39 on:** 2004-07-03 01:34:17 »

They shouldn't just use compression but at least use an original compression tool. I was playing Return to Castle Wolfenstein recently and got bored so I started to poke about and found the data files. They were compressed in nothing but a .lzs file!! The files themselves weren't too great either. They were simple .tga and .wav files. The only thing that was compressed good were the 3D models. They were in a seperate folder from the compressed .lzs folders.

ficedula · « **Reply #40 on:** 2004-07-03 10:08:13 »

Why bother? Hard disk space is cheap nowadays, and compressing data too heavily just makes the game take too longer to load stuff. Light compression is OK: it's so quick to decompress that you save disk space without slowing down loading. I'd complain if a game applied something like full RAR to all the levels and took an extra minute to load up; Battlefield 1942 takes long enough to load as it is...

Messiah99 · « **Reply #41 on:** 2004-07-03 10:39:51 »

fice is right, gotta find a sweet spot. If you need to compress so things arent too large, then do, but you dont want to over-do it and end up with massive load times, Id rather a game be on several cds and take up a gig or gig and a half of hard drive space and load fast rather than be on one or two cds and take 6 or 7 hundred and load slow.

Alhexx · « **Reply #42 on:** 2004-07-03 19:42:19 »

Erm ... just a question...

I've asked if it would be possible that our team could do it.
But now it seems like we're seriously planning how to start that... am I right?

BTW: Welcome back, fice! :love: I've missed your posts here!!!

- Alhexx

halkun · « **Reply #43 on:** 2004-07-03 21:20:59 »

It seems like a game of tetris...

The blocks are kind of falling into place, if they align, well, cool but it depends on the falling blocks.

I'm documenting Gears as fast as I can. I'm hitting sort of a "hump" with the debug rooms. There's a lot of data go through and I have a lot of corrections to make on this guys doc about it. (I also have to change it enough to make it a proper dirivitve work.)

After that's done, I have to hop back into the kernel and rearrange that badly. I need to place the file formats from the end of my doc into "filesystems" as the PSX filesystem is now becoming more known. (It's overall fuctionality is also being revealed thanks to Cyb)

It's just such a pain to dump every debug room, read how it's supposed to work, and the compare it to the scripting code for the room. There are suprises, but it's so methodical and dull. I hit this point in my PSX doc when doinng the GTE. Didn't help I got a love letter from Sony about that time telling me that if I used Psy-Q, there were going to send a much of mad proctologists to my house with a really rusty drill.

If someone has a copy of the PSX version of the game on ISO, I have a PSX script editor that can edit the field scripting code and re-insert it back into the iso. This will make my life much easier as I need more people to help discover what on earth all those opcode arguments so. The Debug rooms are nothing but examples. That's it. I woudn't be suprised if they cut/pated parts of the debug rooms into the game proper for common tasks.

Woudn't hurt for someone to take my specs and write a script editor for the PC version. That's beyond my ability.

Anyway, I'm working on. Gears is still accessable via my WWW button

Cyberman · « **Reply #44 on:** 2004-07-03 22:55:05 »

Halkun:
I could help with tools unfortunately I don't have the PC version, I do have the PSX version. The idea thing would be to have a Virtual CDROM with shadow files (IE files that reside in global PC memory) that can be edited on the fly. Then mount that using the Playstation emulator. So boot check etc.

Ahhh well!

I still need to get the background battle scenes loading.

Allhex
From Halkun's docs I suggest following the original FF7's system. Break it up into modules. Using a kernal (core) with all the functionality the applications use. Think of it like KERNAL.DLL in win98. So Halkun's reversing of it is actually a better way to get there. If you understand what 'causes' something you can better run it, and edit it. This makes making content much easier.

To be blunt though it's not a matter of IF you can do it, that's entirely possible, the real question is.. WILL you do it. Many people have the ability but they don't do it in other words.

All
I'm contemplating making tools for FF7 but I need to know the underlying data and formats. I'm quite familiar with making compilor tools, so its not a big deal too me. Essentially if you want to do what you are thinking about you have to build these tools irregardless.

The preferable thing to do is encode the scripts into XML'ized stuff and compile from that (been there done that).

Marc · « **Reply #45 on:** 2004-07-06 01:04:11 »

I was reading this thread and I was thinking about the state of psx emulation right now.

Since you guys seem to be saying that the psx and pc games are pretty much the same except with some minor file conversions, shouldn't it be, in theory at least, possible to use the sound and graphic plugins as a basis for the engine ?

I mean most of the problems of the pc version are solved with these plugins such as 32 bit support, higher resolution, better movies rendering, better sound, etc. If you also factor in that there are opengl pluigns available it could also simplify croos-paltform compatibility no ?

Although this might be totally impossible. I have no programming knowledge whatsoever. It's just a thought.

Darkdevil · « **Reply #46 on:** 2004-07-06 08:17:21 »

The dudes got a point.

halkun · « **Reply #47 on:** 2004-07-06 08:55:50 »

As there are only two versions of the game, I don't think that's really neccasary. I'm still pluggin aling on the scripting language, however, you all have some new tools at your disposal to help me.

Cyberman · « **Reply #48 on:** 2004-07-06 19:12:01 »

Actually Halkun I'm making a small deviation to get a decent debugger interface working with Pixels PCSX debugger add on. Once that gets going ( 2 week delay my guess), I can focus on examing the FF7 interaction.

You wouldn't have a 'clean' listing of function calls commonly used in PS1 games would you?
I have a list of the bios function calls, I guess that would be a reasonable start. Anyone 2 weeks before I can get the debugging system working then I can start mangling FF7 data again.

Unfortunately it will be for Windows (need a good way to translate borlands dpr information into thier GTK equivalent to port to Linux).

Cyb

halkun · « **Reply #49 on:** 2004-07-06 23:20:22 »

Quote from: Cyberman

You wouldn't have a 'clean' listing of function calls commonly used in PS1 games would you?
I have a list of the bios function calls, I guess that would be a reasonable start. Anyone 2 weeks before I can get the debugging system working then I can start mangling FF7 data again.

I've never played with Psy-Q... ever. I have no idea what kind of calls that lib makes to the metal. When I was making my doc, sony was very very very very direct in telling me that no libs were go into my psx doc.

I don't even have the bios calls... (I used to)