Qhimm.com Forums
Miscellaneous Forums => Scripting and Reverse Engineering => Topic started by: Alhexx on 2004-06-24 14:35:26
-
I know there have been topics related to the before, however, I want to hear your opinion.
I don't want to knwo if it is possible to rewrite the engine; of course it is.
I want to know if you think that our team would be able to do this.
Would we be able to write a new engine that could use all FF7 files and play the game?
- Alhexx
-
Personally I think its gonna be a bitch of a task if it is started. I seem to recall that Fice started re-writing it and i couldn't see much difference in the screenies to the normal game. It will take alot of time, but If its going to happen, I would like to be a part of it.
Given a big enough team, alot of hard work, ****loads of spare time and real passion for this project I reckon it can be done.
-
rewriting the engine for modern platforms was a goal of mine too. I can't code very well, and absoluty suck at c++
We also don't have everything yet. I need to figure out the scripting commands and where they are in the PC field files. Right now I only have ther names and bearly any argument sentax. I bearly know what they do.
We also don't have the walkmeshes for the field files, not the cam movements or the battle animations. I'm still unifying the PSX and PC versions so we can have a common codebase.
BUT!
We have the source code directory listing so we can see how it all fits togeather. I'll probably make a Dia map of the program code files and probable functions inside. We can use that as a filler to start with the low-level kernel functions.
Me will have to make the kernel first, and that's the least understood out all of the modules. Keep in mid it *IS* a multitasking kernel with threading abilities.
But I will contine. I only ask you guys help...
Right now if someone can dig through the STARTMAP field file and try and find the tolkens listed in my scripting post and what they do. I would be greatful.
(I listed the entire STARTMAP script by the way)
-
A project, well worth the effort. It'd make the best game of all times work decently once again.
If you made it, perhaps it wouldn't be too complicated to enable/include movie and background filtering too? Since that could count as a full remake in terms of quality improvement :wink:
Makes me wish I wasn't just a beginner programmer :(
-
Know what ? I've been thinking about that recently. That would be great if we could do that. I know C++ and the DirectX API (by "to know", I do not mean I'm a pro or something like that, just that I have some knowledge of it and made some programs that use DirectX, such as a 2D RPG Engine - still under work), and would like to work in such project. The main problem is that I won't have much time during these holidays. And next year, I'll be in a Computer & Engineering university - no time for distraction. (yeah, I'm someone scared by time :lol:)
-
If anything, re-write it so that it's easier to create and impliment modifications without overwriting original models and material, and so that they actualy work.
As it is now, 3-d models really can't be edited too much. They have to retain the same information. It would be nice if completly new models could be implimented, models with mroe than like, 8 polys per part =P
-
Well Alhexx all things are possible
So you need to decide what it is you want.
My thought is go for portability. This means disembodying platform specific code from the main engine. I think Halkum has a good idea, recreate the base set of data. Now the PC version has a number of crashes and the PSX version has a few bugs.
The portability concept would allow for example Direct X or Open GL to be supported. Different Audio, and more importantly any movies could be in a format like XVID with an OGG wraper. Again non proprietary formats and data lend to portability. The orginal movie sequences are pure unobtainium for it so the old movies repacked should be fine.
This is what I want though, what goal do you want? Multiplatform compatibility and a functional game engine? Toolset for compiling FF data and working with it's engine?
Set goals, makes it easier to acomplish something I've heard. :)
Cyb
-
Cyberman: the movies should work fine in any format as it is. Mine are divx5 with mp3 audio and they work perfectly (and yes, I do own 3 original copies of the game, in case anyone wondered :D - I'm not joking here :oops:). We only need the filtering and perhaps higher color depth 8)
-
Here are forseeable problems and isssues:
1) The new engine should support both the PSX and PC datasets. The PC version is simply too rare while you can pick up the PSX verion new as a "greatest hits". The PSX disks however are not in ISO-9660 format and will not mount in linux. (At least not on my computer, I don't know how to mount a "Mode 2" disk) This means we *MAY* have to write a direct filesystem accessor within the new kernel. (Which, now that I think about it, we will have to do anyway as most of the datasets have to be extracted/decompressed/decoded to even be looked it) If not dual format right away, al least make the system modular enough where support can be added later
2) It should also be cross-platfrom, (Win Mac Linux) which causes some other problems.
3) The kernel is going to be difficult to code, as most of it's fuctionality is invisable
4) To keep from beating our heads aginst the wall, we must use a cross platform libary. Linux can't run DirectX, but windows can run OpenGL
5) The PSX sound data will have to be *re-rendered* on the fly. This includes both the SEQ music files and the sound data. (Which is why Square simply used MIDI and rendered WAV files in the PC version) Linux can't use MIDI out of the box. Timidity only comes with a piano voice. That's it. I had to download 128 meg of sound data from my keyboard just so I can have proper sound. It can use soundfonts in the PC version, but you have to look out for people who don't have the PC version availible to them.
6) The Duck codec is propiatary. It can't play on linux and you must have a licence to use it in you game. That's a legal issue we have to solve. Because duck seems to not even want us asking about it, as you need a password to even look at thier website. That poses a real problem.
7) There is no 7
8) Speaking of legal hurdals, a demo of *ORIGINAL CONTENT* will have to be created in order to make the new engine an original work. It needs to be able to stand on it's own without any support from Square at all.
-
Yes the team will be able to do it, time is only issue ... i'd say it will take few years to complete it (Yea i know im born pesimist : )
I can say that from my personal experience, i'm stuck in few projects and im just-in-the-middle ... complex things like this allways take a long time.
-
Hey Mirex, how long have we been working on this now ^_^
Look at your join date and mine, we've been here a while...
-
Here are forseeable problems and isssues:
5) The PSX sound data will have to be *re-rendered* on the fly. This includes both the SEQ music files and the sound data. (Which is why Square simply used MIDI and rendered WAV files in the PC version) Linux can't use MIDI out of the box. Timidity only comes with a piano voice. That's it. I had to download 128 meg of sound data from my keyboard just so I can have proper sound. It can use soundfonts in the PC version, but you have to look out for people who don't have the PC version availible to them.
Well, the PSF patch (http://aaronserv.dyndns.org/hosting/ffsf/download_7.html) should be up to that task.
Not to mention, the PSX-emu scene has a decent amount of attention given to Linux, so I wouldn't be surprised if there are Linux-compatible players out there, that can be adapted to use with the PSF patch....
-
8) Speaking of legal hurdals, a demo of *ORIGINAL CONTENT* will have to be created in order to make the new engine an original work. It needs to be able to stand on it's own without any support from Square at all.
If this is done anytime after the 14th of august (When I get Broadband) I will be able to send it in to PC Gamer, PCZone and PCFormat to see what they think of it *IF* thats what you mean.
-
Well, the thing is, ff7 and all the characters, etc, are also the INTELLECTUAL copyright of square. We'd have to be very specifi about this being a "Fan-project" and that Cloud, Sephiroth, etc are still copyright of Square. I'm sure that's a no-brainer, but still.
My point is, I don't think it'd be too wise to ditribute a demo to gaming magazines. I mean sure, plenty of mods go onto gaming disks, but this is above and beyond any normal mod. Basicalyl, y'all would ahve to be careful how you present it, that's all.
-
If this is done anytime after the 14th of august (When I get Broadband) I will be able to send it in to PC Gamer, PCZone and PCFormat to see what they think of it *IF* thats what you mean.
I highly doubt we will have *AYNTHING* soon. We can bearly swap models. We can't even create our own without crashing the engine. After four years of work, we are more at the beginning than the end.
-
I'm not sure about anyone else but I like making tools. IE need something to dump out a particular format, I can do that. It's something I like to do, heck I wrote a tool that took assembly for one processor and recoded for another processor (In PERL of all languages).
Reading in Mode 2 is a big issue period, maybe you need to seperate the CD/DISK IO out as a section instead of incorperation. Hmmm anyhow.
PSX sound engine is well emulated and documented, PeopsSound toy is a good example.
Your project will die by a thousand paper cuts if you don't break it down into seperate problems. Like the original game engine is. Perhaps that's the best method to follow, use the original game concept to run the game. 6 functional programs 1 kernal and 5 sub programs that run from the kernal.
In the PC version do they include all the characters as part of the movie sequence or are they like the PSX 3d characters over an animated background?
Cyb/Stephen
-
Here are forseeable problems and isssues:
1) The new engine should support both the PSX and PC datasets. The PC version is simply too rare while you can pick up the PSX verion new as a "greatest hits". The PSX disks however are not in ISO-9660 format and will not mount in linux. (At least not on my computer, I don't know how to mount a "Mode 2" disk) This means we *MAY* have to write a direct filesystem accessor within the new kernel. (Which, now that I think about it, we will have to do anyway as most of the datasets have to be extracted/decompressed/decoded to even be looked it) If not dual format right away, al least make the system modular enough where support can be added later.
I'm guessing that when you are refering to having to extract/decompress/decode the data files, you're refering to what both the PC and PSX versions of game have to do to access their files, since they're both using compressed formats, right?
If so, then if what I'm about to say is old ground that was already covered, feel free to beat me with the "OLD!" stick -- mainly because I haven't been very attentive in regards to this board's progress regarding the reverse-engineering of FF7pc. :roll:
But hear me out anyway, in case I wind up giving any of you ideas, even if this a bunch of old ground....heck, for all we know, this might allow us to get at the the specific models inside the LGP without needing to decompress to separate files first. :P
What if there is no actual large-scale decompression going on in the PC version (and possibly the PSX version)? I mean, we know that the PC version was coded in C++....so what if they decided to use C++ member functions like "seekp" and "seekg" to do non-sequencial read access?
Now think about this for a moment. If they tried to model on the PC how the PSX does its data lookups (direct sector reading, and not using CDFS format) -- which really only works when you can be positive that the data on the disc isn't gonna move around -- then using seekp and seekg to point toward specific bytes of whatever LGP file they are using, then they can essentially copy that portion of the file into a struct array, and then decompress it without needing to decompress the whole LGP file.
Now, this would also explain why things can go wacky when you try to alter the file and come up with a different size -- for example, let's say the game tries to pull up Cloud's battle model, so it goes to the appropriate LGP file and uses the "seekg" function.....I'll do some code off the top of my head to show what I mean:
(please note that all the data types, byte offsets, and file sizes have been grabbed out of thin air and do not accurately reflect what types they should be...this is more half-psuedocode, basically)
#include <iostream>
#include <fstream>
using namespace std;
//define model data structure
struct model
{
int Polys[InsertArraySizeHere];
int Vertices[InsertArraySizeHere];
int Textures[InsertArraySizeHere];
};
//function prototypes
void getPolyData(model *);
void getVertexData(model *);
void getTextureData(model *);
void decompressData(model *);
int main()
{
*ptrCloud = new model;
//grab cloud's poly, vertex and texture data
getPolyData(*ptrCloud);
getVertexData(*ptrCloud);
getTextureData(*ptrCloud);
//now decompress everything into memory -- and pray the
//struct that you've set up that will hold the decompressed
//data is large enough :)
decompressData(*ptrCloud);
return 0;
}
void getPolyData(model *cloud)
{
fstream file;
fstream *position;
fstream *end_struct;
//open file for random access of data
fstream file("battle.lgp", ios::in | ios::binary);
//jump to end of Cloud's model data, and flag it with a
//pointer
*endofstruct = file.seekg(40L, ios::beg);
//we know that Cloud's compressed poly data occupies bytes 20
//through 40, so we jump to byte 20
*position = file.seekg(-20L, ios::cur);
//increment 'till we hit the end of this model's compressed
//poly data, or the end of the LGP
do
{
for (int i = 0; position < end_struct; i++)
{
file.get(cloud.Polys[i]);
position++;
}
} while (!file.eof());
file.close();
}
void getVertexData(model *cloud)
{
blah blah blah...
}
void getTextureData(model *cloud)
{
yada yada yada.....
}
void decompressData(model *cloud)
{
*insert file bloating sounds here*
}
....using numbered offsets to set the read position to the correct part of the file -- or what it THOUGHT was the correct part of the file, if we are using a drastically changed file whose size is totally different, and would thus make the game convulse if it doesn't have a way of dynamically recalculating the byte offsets in a changed file.
Anyway...thoughts on my code? Oldness, or am I on to something? :P
-edit-
Ack, the comments got all out of line when it resized the text boarders. Fixed it -- I think.... :wink:
-2nd Edit-
Did general code cleanup and formatting that I felt like doing. :P
-
If this is done anytime after the 14th of august (When I get Broadband) I will be able to send it in to PC Gamer, PCZone and PCFormat to see what they think of it *IF* thats what you mean.
I highly doubt we will have *AYNTHING* soon. We can bearly swap models. We can't even create our own without crashing the engine. After four years of work, we are more at the beginning than the end.
A good point. Well made too.
-
Well, actually, we *can* create new FF7 Models that the engine does accept. However, this has only been tested with untextured files, but with a little bit of hacking (I hope I will have time for this after my army time...) we'll be able to create new textured models, too.
Of course, the other file formats are a big problem. Battle animations, cameras and so on have to be hacked first.
- Alhexx
-
Well 'we' know scene.bin contains all the nifty little battle beasties yes we do!
the battle engine is a beats to itself is it not? Since it's loaded by the kernal module a good debugger would be handy at that point indubidably.
Cyb
-
Uhhh I would gladly try to work on creating P files that FF7 engine accepts, only if i had some free time. Too many projects are on my hands.
-
Uhhh I would gladly try to work on creating P files that FF7 engine accepts, only if i had some free time. Too many projects are on my hands.
Mirex, If you could teach me how to do stuff like that, I will be able to do some stuff as Im doing nothing untill september now...Im out of school so i have ALOT of free time.
-
Mirex, If you could teach me how to do stuff like that...
Teaching this kind of hobby would take even more time imho.
Nothing personal.
-
Oh, Ok.
-
But if you really want to, then i could give you few hints. Are you skilled in programming ? If not then that is the best way to go. C++, Pascal or Visual Basic (which is not much recommended, but works ;) ) ... choose any and go for it. Try to program something that messes with files, anything.
-
Well I created a clip-art Veiwer in Vb6 this morning in my 6th Form induction day...Other than that, I have ne means of doing ny programming as i have no applications to do so...
-
But if you really want to, then i could give you few hints. Are you skilled in programming ? If not then that is the best way to go. C++, Pascal or Visual Basic (which is not much recommended, but works ) ... choose any and go for it. Try to program something that messes with files, anything.
By the way, Mirex....what about the code I posted earlier up on the page? Granted, a lot of variables and stuff are just being used as placeholders for the real data types and such that would be used, but would that not be a start for getting direct access to specific models without decompressing the LGPs during run-time of this hypothetical re-written engine?
Looking back through some other threads, it appears there are a lot of talk about offsets to stuff. But what are the offsets relative to? The beginning of the individiual .p files?
Actually, perhaps the iidea behind the earlier code I posted could demystify some data tables that haven't been figured out, they could contain offsets that are only valid when looking at the compressed code of the LGP file, and not the .p files themselves.... resulting in a funny parallel:
LGP file == PSX data that is not stored in the normal CD filesystem
offset tables == coordinates for PSX CD Direct-Sector reads.
Heck, no one's even told me if what I posted up there or here is old ground or not....
-
LGP files aren't themselves compressed in any way. The FF7 engine can read data straight out of them if necessary ... for that matter, so can the Remake engine (not that I've touched that in ages). LGP files just provide a way of packaging up lots of files into one big chunk, so the OS (Windows) doesn't need to bother about creating 20,000 files itself.
The offsets we're talking about are generally relative to the beginning of the individual file in question (such as a P file), yes. I don't think you should give too much thought to the LGP file, since it's essentially just a container, nothing relies on being present in a particular location within the LGP (proof: you can add new data to an LGP file using LGP Tools and FF7 doesn't care *where* it is - even if it's now at the end of the archive when it was originally at the beginning - so long it's there and listed in the contents of the LGP).
Anyway. Yes, most tools copy (not decompress) the data from the LGP into memory and then use it from there, rather than reading direct from disk, but we don't *have* to do that. There's little reason not to, though, the end effect should always be the same.
-
LGP file == PSX data that is not stored in the normal CD filesystem
PC Data != PSX data.
More importantly:
Raw Psy-Q data != FF7-PSX data != FF7-PC data.
The PSX deals with raw Psy-Q data. (TIM, RSD, SEQ, STR, TOD, HRC, etc etc etc) These are munged and stored in special formats for FF7-PSX. (BIN, LZS). The PC version has conversions of the PSY-Q data into PC native data. (SEQ=>MID, STR=>AVI, Binary HRC=>Plain Text HRC)
The only thing that should be platform independant is the core of the kernel. It takes a lot of bit twiddeling to get the data into something that the platfrom can easily handle and that the user can understand.
-
First off -- Long time no see, Fice. :D
LGP files aren't themselves compressed in any way. The FF7 engine can read data straight out of them if necessary ... for that matter, so can the Remake engine (not that I've touched that in ages). LGP files just provide a way of packaging up lots of files into one big chunk, so the OS (Windows) doesn't need to bother about creating 20,000 files itself.
Oh, so they're NOT compressed....then that should make the re-writting of the engine that much easier, as then it would just need to deal with offsets w/ no real "decompression" per se.
The offsets we're talking about are generally relative to the beginning of the individual file in question (such as a P file), yes. I don't think you should give too much thought to the LGP file, since it's essentially just a container, nothing relies on being present in a particular location within the LGP (proof: you can add new data to an LGP file using LGP Tools and FF7 doesn't care *where* it is - even if it's now at the end of the archive when it was originally at the beginning - so long it's there and listed in the contents of the LGP).
You know....that makes sense.
After I posted the code earlier, it hit me that there just might be a table of contents in the LGP file. On the other hand, I didn't bother to mention it because I had this feeling that all this was indeed old ground. :P Either way, thanks for confirming my suspicion, LOL.
I guess that the existance of a table of contents in the LGP files would be how the current FF7 engine could cope with drastically modified LGP files. This would mean that instead of having a table of contents hard-coded into the .exe file (which is what my code example assumed), it would perform the seekg commands using variables previously loaded with offset values from the table of contents.
But therein lies a potential problem, though: The offsets needed to find the table of contents may be hardcoded into the .exe file anyway. In all probability, if there are hardcoded offsets that determine the table of contents' beginning byte and ending byte in the LGP file, it would put a limit on how many files each LGP file can hold and still be recognized by FF7, due to the tunnel vision on the size of the contents listing that the hard-coded offset values are causing....right?
Anyway. Yes, most tools copy (not decompress) the data from the LGP into memory and then use it from there, rather than reading direct from disk, but we don't *have* to do that. There's little reason not to, though, the end effect should always be the same.
At least my train of thought was on the right track. Yay! :P
It also gave me an idea regarding how to find where the battle animations are stored. Have you (or anyone else for that matter) tried to do a memory dump of the video card RAM during a battle?
My thinking is this -- most likely FF7 sent the animation data to VRAM instead of the system RAM along with everything else -- and probably as a series of struct arrays as well. Furthermore, even IF it is sent to system RAM first, it is inevitable that it will be transferred to VRAM at some point, and thus we can be sure it's hiding in there. :roll:
Assuming we can get a dump of the the VRAM, we could compare the information in there with the data in the LGP files, and eliminate anything that corresponds to data we already know to not be the battle animations. Any sequences of data that don't appear to be reconizeable to us should have its corresponding sequence in the LGP files noted for later reference as a possibility for the animation sequences.
If there's only one unique set of sequences of unrecognized data, then....well there ya go!! :wink:
PC Data != PSX data.
More importantly:
Raw Psy-Q data != FF7-PSX data != FF7-PC data.
The PSX deals with raw Psy-Q data. (TIM, RSD, SEQ, STR, TOD, HRC, etc etc etc) These are munged and stored in special formats for FF7-PSX. (BIN, LZS). The PC version has conversions of the PSY-Q data into PC native data. (SEQ=>MID, STR=>AVI, Binary HRC=>Plain Text HRC)
The only thing that should be platform independant is the core of the kernel. It takes a lot of bit twiddeling to get the data into something that the platfrom can easily handle and that the user can understand.
I know that, but when I wrote what I wrote, it was supposed to be a metaphor, not a literal statement that the code was the same bit for bit, 'cause its not. Just that the philosophy in how the engine probably looks at how the code is stored is probably the same in both versions.
I.E., those PSX direct sector reads are analogous to the PC version taking an LGP file and directly grabbing the code it needs from it, and then decompressing, using the offsets the same way the PSX uses a list of coordinates to do a direct-sector read from the disk.
I probably should have been a bit more concise when I wrote that, LOL. ;)
-
My thinking is this -- most likely FF7 sent the animation data to VRAM instead of the system RAM along with everything else -- and probably as a series of struct arrays as well. Furthermore, even IF it is sent to system RAM first, it is inevitable that it will be transferred to VRAM at some point, and thus we can be sure it's hiding in there.
Nope, it's most definatly not in there....
The video ram hold textures, texture caches, frame buffers, and palettte information. You can dump the VRAM from a PSX emulator and see for yourself...
Because the code is PSX native there's also a nother reason why haveing data in VRAM is silly.
The PSX has *no direct access to VRAM*
You can't put data into vram using a memory address as the video memory is not directly attached to the PSX's CPU bus. You have to send data using a GPU packet (Command + arguments) like "Draw triangle", "set viewport", or "load texture" That's why TIM files are used so often. They are pre-formed data that the GPU can accept.
You either put the data into a GPU register, or stream via the GPU's DMA hook. (Meaning you allow the GPU Direct Memory Accees to the PSX's main RAM, but you can't do it the other way around) The GPU has no concept of "Animation" or anything like that. It't can't even do math. That's the GTE's job and that's on another bus all togeather
-
I was refering to the PC version......
-
Unfortunately still isn't going to happen. Modern graphics cards can upload animations and geometry to VRAM, but for that to be *any* use whatsoever the card needs to support hardware T&L at an absolute minimum (and realistically, it needs to have other features too). When FF7 came out, they didn't, so it's not written to upload animations to video cards.
It's a fairly recent development, allowing you to do that, anyway. PC users haven't been able to explicitly upload vertex data to the card for that long, nevermind the actual animations ... all possible now, mind you. If you have a recent Radeon/Geforce, of course, and a game written to support it...!
Back to the older stuff...
First off -- Long time no see, Fice.
Thank you ;) I've been busy with my job ... not a new job anymore, but still, first full time job. Warning: jobs eat your time.
But therein lies a potential problem, though: The offsets needed to find the table of contents may be hardcoded into the .exe file anyway. In all probability, if there are hardcoded offsets that determine the table of contents' beginning byte and ending byte in the LGP file, it would put a limit on how many files each LGP file can hold and still be recognized by FF7, due to the tunnel vision on the size of the contents listing that the hard-coded offset values are causing....right?
Nope. The only thing 'hard coded' is the point at which the table of contents *begins* (essentially the first piece of data in the file), not its length.
That said, the file format uses 16-bit integers for some of the data, so while the file size of an individual LGP could go up to at least 2GB, you'd be limited in practice to ~65,500 files inside it. Perhaps double that if you played some nasty tricks on the indexes.
Of course, that's fairly academic; even battle.lgp doesn't come anywhere near that number of files, so you've got plenty of room to add your own files.
-
IIRC the most files in an lgp is close to 11,000
Its is either the feil or battle LGP file....im going for the latter.
-
Unfortunately still isn't going to happen. Modern graphics cards can upload animations and geometry to VRAM, but for that to be *any* use whatsoever the card needs to support hardware T&L at an absolute minimum (and realistically, it needs to have other features too). When FF7 came out, they didn't, so it's not written to upload animations to video cards.
It's a fairly recent development, allowing you to do that, anyway. PC users haven't been able to explicitly upload vertex data to the card for that long, nevermind the actual animations ... all possible now, mind you. If you have a recent Radeon/Geforce, of course, and a game written to support it...!
Hmm.....I see. However.....
If I'm not mistaken, whatever animations that are about to be drawn on the screen are first transfered to the framebuffer, which should be in the VRAM, so technically the data would in one form or another pass thru the VRAM before going on screen. But, it just occured to me that the data might have already been processed into something other than its original form by that time, so maybe a VRAM dump might not work as I planned.
IIRC, when the case is that no HW TnL is available, then the CPU and system RAM must then caculate the Geometry Transformation instructions before passing it to the framebuffer in the VRAM....so doing a system RAM dump might still be an acceptible lead?
Of course, we'd have to do this on a system with the minimum RAM necessary to get the game to run, as the last thing we need is to go through 1/4 to 1/2 a gig of data looking for the animations.
Basically, I'm saying that since we haven't been able to locate where the animation data is stored, we let the game do it for us, and then we take a picture of what it grabbed from the LGP, and then we go back and cross-reference the data there with what's in the LGP.
There is another remaining point that we ought to remember. Thanks to what halkun dug up (http://forums.qhimm.com/viewtopic.php?t=3052) regarding what FF7pc was compiled from, we know that FF7pc was written in C++. Therefore, we must think within the limitations of that language when trying to reconstruct how FF7 can access and decode battle operations, etc, and that includes file I/O, and memory storage.
That leaves us with commands like ".read"; ".get"; and ".getline" for loading the battle animation data into memory, if I'm not mistaken. And considering that the data is probably a series of values, it would have to be loaded into some sort of data array, which in turn means the entire data sequence for the animations would be occupying adjacent memory addresses, meaning a RAM dump just might happen to have the battle animation data all conveniently lumped together in one section of memory addresses, instead of being scattered all over the place. This would allow us to very quickly narrow down which sequence it is....I think.
Then again, I do have to admit I've only taken an introductory course in C++, so maybe I'm being too naive or something in regards to how the thing could be coded........
Thank you ;) I've been busy with my job ... not a new job anymore, but still, first full time job. Warning: jobs eat your time.
Well, congrats on that. If all goes well, I also just might get a full time job in a computer store.....I'm just waiting for them to review my application/resume, and make a decision....
Nope. The only thing 'hard coded' is the point at which the table of contents *begins* (essentially the first piece of data in the file), not its length.
Well, in that case, the beginning of the table of contents isn't hard-coded as an offset either. ;)
But how would they figure out where it ends?
.....oh, duh. There's probably a delimiter character in there. Man, do I feel stupid. (http://mapage.noos.fr/gc12/x3dfx/kaboom.gif)
That said, the file format uses 16-bit integers for some of the data, so while the file size of an individual LGP could go up to at least 2GB, you'd be limited in practice to ~65,500 files inside it. Perhaps double that if you played some nasty tricks on the indexes.
Of course, that's fairly academic; even battle.lgp doesn't come anywhere near that number of files, so you've got plenty of room to add your own files.
Great. I had the right conclusion (there's a limit to how many files you can put in an LGP file), but got to it by a completely wrong line of reasoning....what are the chances of that? :P
But still....who in their right mind would need to have 65,500 files for a program?
IIRC the most files in an lgp is close to 11,000
Its is either the feil or battle LGP file....im going for the latter
11,000?! Ok, the programmers need to be taken into the backyard and shot one by one on behalf of the KISS principle.....
-
(http://img59.photobucket.com/albums/v179/darkdevil177/untitled.jpg)
The exact number.
-
Keep in mind they are 11,000 really tiny files. some are just a small 32x32 4 bit pictures and such.
-
11,000?! Ok, the programmers need to be taken into the backyard and shot one by one on behalf of the KISS principle.....
That's quite unrealistic
Here is a dosage of reality here. Take a game like NWN you have say 3500 scripts 1200 models etc etc. Now each model consists of about 10 different files.. are you following me yet?
Let me change the way you look at things. If you were to see Windows Source code, think abou tthe number of components and different files that consist of windows. The programers wisely instead of having 11000 different files archived them into a singular file they could extract the data from, however originally they did NEED the 11000 files because they had to be made.
They didn't just WISH the LGP contents into being ;) A group of artists and scriptors were busily making the contents of that. Hence 11000 is not rediculous.
What they did do was KISS they centralized the data and kept information of where things were.
I'm sure FF8 has similar volumes of data in it (and I'm making slow progress on finding the sections of data heh, I have the world textures found already I just need to find the topographic data).
An example is one FF7 Battle mode on the PS1l for cloud.
The one file contains 113 sections
Section 1 consists of 23 bones six of which are joints. That means it has 17 models with vertex pools triangles quads with colored vertices and triangles and quads with textures.
Section 97 is the texture data
Section 98-113 are the weapon models.
Just one single file in reality consists of more than a hundred smaller files originally.
More importantly they HAD to have the 100+ original files and you can't just compile them to one file and throw them out (it will loose information).
There are 300+ enemys the average enemy has 10 bones lets say (no texture) that's 1 HRC and 10 RSD files per enemy, that doesn't include the animation information (there are at least 1 animation but generally there are 5) then a TIM is always include (irregardless if it's used). that's 16 at minimum original files. Well over 4000+ files just for battle models alone.
There is just TOO much information to make it super simple so the programers did the next best thing they archived it ;)
Cyb
-
First, I'm not knocking the fact that they archived it -- I think that's smart, and that more games should nowadays have as much stuff as they can be compressed w/o too much performance loss, instead of unloading 5,000 .bmp files for textures and raw .wav audio for sound.
But that aside, I realize that most of the time a game or powerful program's source code is made up of large numbers of files. However, when I said what I said, it didn't cross my mind that I should have considered those 11,000 files to be in the same vein as the source code, as you essentially pointed out.
However, I was also under the impression that even the final amount of files they ended up with in the LGPs was still needlessly split up past the point of simplicity -- like, for example, the fragments of previous threads on the files and how the engine uses them made it sound like the game had 300 separate files it had to access and load separately, JUST FOR CLOUD'S HEAD. That's what I thought was going on, and that's why I was saying that the programmers didn't seem to be following KISS.
I guess I was mistaken, though.....
-
They shouldn't just use compression but at least use an original compression tool. I was playing Return to Castle Wolfenstein recently and got bored so I started to poke about and found the data files. They were compressed in nothing but a .lzs file!! The files themselves weren't too great either. They were simple .tga and .wav files. The only thing that was compressed good were the 3D models. They were in a seperate folder from the compressed .lzs folders.
-
Why bother? Hard disk space is cheap nowadays, and compressing data too heavily just makes the game take too longer to load stuff. Light compression is OK: it's so quick to decompress that you save disk space without slowing down loading. I'd complain if a game applied something like full RAR to all the levels and took an extra minute to load up; Battlefield 1942 takes long enough to load as it is...
-
fice is right, gotta find a sweet spot. If you need to compress so things arent too large, then do, but you dont want to over-do it and end up with massive load times, Id rather a game be on several cds and take up a gig or gig and a half of hard drive space and load fast rather than be on one or two cds and take 6 or 7 hundred and load slow.
-
Erm ... just a question...
I've asked if it would be possible that our team could do it.
But now it seems like we're seriously planning how to start that... am I right?
BTW: Welcome back, fice! :love: I've missed your posts here!!!
- Alhexx
-
It seems like a game of tetris...
The blocks are kind of falling into place, if they align, well, cool but it depends on the falling blocks.
I'm documenting Gears as fast as I can. I'm hitting sort of a "hump" with the debug rooms. There's a lot of data go through and I have a lot of corrections to make on this guys doc about it. (I also have to change it enough to make it a proper dirivitve work.)
After that's done, I have to hop back into the kernel and rearrange that badly. I need to place the file formats from the end of my doc into "filesystems" as the PSX filesystem is now becoming more known. (It's overall fuctionality is also being revealed thanks to Cyb)
It's just such a pain to dump every debug room, read how it's supposed to work, and the compare it to the scripting code for the room. There are suprises, but it's so methodical and dull. I hit this point in my PSX doc when doinng the GTE. Didn't help I got a love letter from Sony about that time telling me that if I used Psy-Q, there were going to send a much of mad proctologists to my house with a really rusty drill.
If someone has a copy of the PSX version of the game on ISO, I have a PSX script editor that can edit the field scripting code and re-insert it back into the iso. This will make my life much easier as I need more people to help discover what on earth all those opcode arguments so. The Debug rooms are nothing but examples. That's it. I woudn't be suprised if they cut/pated parts of the debug rooms into the game proper for common tasks.
Woudn't hurt for someone to take my specs and write a script editor for the PC version. That's beyond my ability.
Anyway, I'm working on. Gears is still accessable via my WWW button
-
Halkun:
I could help with tools unfortunately I don't have the PC version, I do have the PSX version. The idea thing would be to have a Virtual CDROM with shadow files (IE files that reside in global PC memory) that can be edited on the fly. Then mount that using the Playstation emulator. So boot check etc.
Ahhh well!
I still need to get the background battle scenes loading.
Allhex
From Halkun's docs I suggest following the original FF7's system. Break it up into modules. Using a kernal (core) with all the functionality the applications use. Think of it like KERNAL.DLL in win98. So Halkun's reversing of it is actually a better way to get there. If you understand what 'causes' something you can better run it, and edit it. This makes making content much easier.
To be blunt though it's not a matter of IF you can do it, that's entirely possible, the real question is.. WILL you do it. Many people have the ability but they don't do it in other words.
All
I'm contemplating making tools for FF7 but I need to know the underlying data and formats. I'm quite familiar with making compilor tools, so its not a big deal too me. Essentially if you want to do what you are thinking about you have to build these tools irregardless.
The preferable thing to do is encode the scripts into XML'ized stuff and compile from that (been there done that). (http://www.geocities.com/cyberman_phillips/Cybs_Projects.html)
-
I was reading this thread and I was thinking about the state of psx emulation right now.
Since you guys seem to be saying that the psx and pc games are pretty much the same except with some minor file conversions, shouldn't it be, in theory at least, possible to use the sound and graphic plugins as a basis for the engine ?
I mean most of the problems of the pc version are solved with these plugins such as 32 bit support, higher resolution, better movies rendering, better sound, etc. If you also factor in that there are opengl pluigns available it could also simplify croos-paltform compatibility no ?
Although this might be totally impossible. I have no programming knowledge whatsoever. It's just a thought.
-
The dudes got a point.
-
As there are only two versions of the game, I don't think that's really neccasary. I'm still pluggin aling on the scripting language, however, you all have some new tools at your disposal to help me.
-
Actually Halkun I'm making a small deviation to get a decent debugger interface working with Pixels PCSX debugger add on. Once that gets going ( 2 week delay my guess), I can focus on examing the FF7 interaction.
You wouldn't have a 'clean' listing of function calls commonly used in PS1 games would you?
I have a list of the bios function calls, I guess that would be a reasonable start. Anyone 2 weeks before I can get the debugging system working then I can start mangling FF7 data again.
Unfortunately it will be for Windows (need a good way to translate borlands dpr information into thier GTK equivalent to port to Linux).
Cyb
-
You wouldn't have a 'clean' listing of function calls commonly used in PS1 games would you?
I have a list of the bios function calls, I guess that would be a reasonable start. Anyone 2 weeks before I can get the debugging system working then I can start mangling FF7 data again.
I've never played with Psy-Q... ever. I have no idea what kind of calls that lib makes to the metal. When I was making my doc, sony was very very very very direct in telling me that no libs were go into my psx doc.
I don't even have the bios calls... (I used to)
-
You wouldn't have a 'clean' listing of function calls commonly used in PS1 games would you?
I have a list of the bios function calls, I guess that would be a reasonable start. Anyone 2 weeks before I can get the debugging system working then I can start mangling FF7 data again.
I've never played with Psy-Q... ever. I have no idea what kind of calls that lib makes to the metal. When I was making my doc, sony was very very very very direct in telling me that no libs were go into my psx doc.
I don't even have the bios calls... (I used to)
Hmmm.. I suppose if I will then need to use a derivation system to create C level function call listings. IE when you find a call to a subroutine find out where it goes/does then give it a name and guessed parametric reference. Should work. I plan on storing this data in an XML database so I can compare it with other information. I think I have the Yaroze function list.. and I think I have the PS1 bios list as well. Hmmm.
I should get to work on that soon.
Cyb
-
You are assuming that there are standard calls that are commonly used, there isn't
Though it is true that many game companies use Psy-Q libs to make games ...LoadTIM(), playSEQ()... These are very often tweaked and stripped for the individual application. The PSX does not use shared libaries like DLLs, a PSX executable is a static COFF-type. I don't think you are going to find entry points very easily as they can be anywere, and custom within the executable.
The FF7 kernel is amazing because it manages threading and several sub programs, each with a scripting language, all without a Memory manager. (FF7's MMU is software(!)) This reqired many of the libs to be tweaked to run in such an environment.
However there is a strict programming rule that the PSX executable must use the BIOS fuctions. You can't talk directly to the metal. This is why PS1 games work on a PS2 even with a massivly different architecture. This is also why Virtual Game station ran so well. Connectix wrote thier own BIOS.
Also it's why Bleem was only 512K. It caught BIOS calls and made DirectX calls.
-
I thought Connectix was a JIT emulator.
-
You are assuming that there are standard calls that are commonly used, there isn't
Though it is true that many game companies use Psy-Q libs to make games ...LoadTIM(), playSEQ()... These are very often tweaked and stripped for the individual application. The PSX does not use shared libaries like DLLs, a PSX executable is a static COFF-type. I don't think you are going to find entry points very easily as they can be anywere, and custom within the executable.
The FF7 kernel is amazing because it manages threading and several sub programs, each with a scripting language, all without a Memory manager. (FF7's MMU is software(!)) This reqired many of the libs to be tweaked to run in such an environment.
However there is a strict programming rule that the PSX executable must use the BIOS fuctions. You can't talk directly to the metal. This is why PS1 games work on a PS2 even with a massivly different architecture. This is also why Virtual Game station ran so well. Connectix wrote thier own BIOS.
Also it's why Bleem was only 512K. It caught BIOS calls and made DirectX calls.
That explains why I never saw a program use the TLB hardware in the PS1.. odd that. I suppose if they had things would be much more difficult (LOL).
Well if they use the BIOS I can deal with the bios calls easily enough. That at least will give an idea of what they are doing at that level.
Cyb
-
On top of which ^_^
Yaroze and and Psy-Q are totaly different beasts. Yaroze allows no access to the CD-ROM at all. Also the executible format is different It was an "ecoff" bin and not compatable with a PSX-EXE type coff file. The yaroze libs are massively different and made by Sony. Psy-Q was deveoped by Psygonsis.
You also can't boot a Yaroze ecoff from a cd-rom. The BIOS would halt the disk.
The TLB was used for caching purposes when needed. Because all the memory segemnts were hard wired togeather, depenending on what memory mirror you worte to, that would engaguge and disengauge the cache. That's really it.
PSX also doen't have a math co-prossesr either. The GTE is a fixed-point matrix trasformation engine. It also is horrably buggy and that's why the texture perpectives suck so badly. Many games relied on the buggy GTE and sony coudn't fix it later on.
-
...we know that FF7pc was written in C++. Therefore, we must think within the limitations of that language when trying to reconstruct how FF7 can access and decode battle operations, etc, and that includes file I/O, and memory storage.
That leaves us with commands like ".read"; ".get"; and ".getline" for loading the battle animation data into memory, if I'm not mistaken...
C++ doesn't really have any limitations. Its not uncommon for most video games to have a lot of the operations that need to run very very fast (thousands of times per seconds, in some cases) written in ASM (assembly) blocks.
.read, .get, and .getline are not really effecient for the very specific and repetative purposes of a lot of the FF7 files. I would be more surprised if they didn't write thier own low-level file and memory IO functions than if they did.
They teach that in intermediate C++ classes. Its a fairly common practice as far as I can tell.
Remember that C++ and ASM pretty much go hand-in-hand, and you can write ASM directly into your C++ programs.
But then, Square programmers may also be a fairly lazy lot. So who knows?
And in case anyone is wondering, PSX games are programmed in a version of C++ as well -hooking into a propriatary Sony hardware API.[/b]
-
Hey isn't the decompression algorithm used in FF7 patented?
-
Maybe you're thinking of the much-disputed LZW patent? (read: "The GIF bitch")
-
Maybe you're thinking of the much-disputed LZW patent? (read: "The GIF b*tch")
Which has expired, I wrote a compressor that used it as a college project lot of fun, but tottally useless :)
Unisys had the patent on it and decided to 'exercise' the patent, they got a rude awakening as people deleted the standard unix compress off there systems and used something else instead. "Yes we are compliant .. we don't use your software at all or a variant we deleted it, you can come and look if you wish."
sfx1999
LSZ compression is used which is a very different animal.
Cyb