Qhimm.com Forums

Final Fantasy Forums => Scripting and Reverse Engineering => Topic started by: paul on 2015-12-12 16:09:11

Title: [FF7 PC] Full source code reversing project?
Post by: paul on 2015-12-12 16:09:11
Would anyone be interested in a project like this: https://github.com/OpenRCT2/OpenRCT2

But for FF7 PC? Basically its a hook DLL where by each game function is slowly replaced with C/C++ until the full source of the game is obtained.
 
Title: Re: [FF7 PC] Full source code reversing project?
Post by: Kaldarasha on 2015-12-12 20:41:27
I got a similar idea. Is it possible to replace certain elements on the fly trough a other application. I'm mainly interested in a way to replace the models trough models with Wight painted meshes.

I also would love it if we could replace the menu and text boxes.
Title: Re: [FF7 PC] Full source code reversing project?
Post by: halkun on 2015-12-12 21:33:51
Once of my little achievements was to extract a partial source code list from the executable

http://wiki.qhimm.com/view/FF7/Technical/Source

Mind using that as kind of a template :)

There are two sides to this source. The game side and the PsyQ (driver) side. Its neat to see how it's put togeather

I have a cooler looking breakdown here ----> http://forums.qhimm.com/index.php?topic=13814.0
Title: Re: [FF7 PC] Full source code reversing project?
Post by: Covarr on 2015-12-13 02:33:23
Reverse engineering is great and I really like this idea, but if I see any links to copyrighted data such as decompiled source code, it will be deleted and warnings issued, or bans for repeat offenses. I know you're enough to know where the line is and not to cross it, Paul, but it needs said nonetheless.
Title: Re: [FF7 PC] Full source code reversing project?
Post by: Shard on 2015-12-13 02:53:28
Honestly, you're better off writing your own engine in 2016 rather than decompiling code from 1995-1996. Take advantage of new technology. This is what QGears is aiming to do, except it's not very active right now.
Title: Re: [FF7 PC] Full source code reversing project?
Post by: ficedula on 2015-12-13 08:52:20
It's not that bad an idea; in a sense, it's how Ultrasound and some other mods work (replace just this function - to change/add sound effects - leave the rest of the game intact).

Not sure I'd count on ever rewriting the whole thing that way, though.
Title: Re: [FF7 PC] Full source code reversing project?
Post by: paul on 2015-12-13 14:45:40
Honestly, you're better off writing your own engine in 2016 rather than decompiling code from 1995-1996. Take advantage of new technology. This is what QGears is aiming to do, except it's not very active right now.

This would be the best base for something like QGears though. For example when the world map functions are fully reversed then you can replace/extend it in any way you'd like. And you have a full working game. Otherwise everything has to be 100% re-implemented from day 1 to have a fully working game.

Also I'd assume by this point many people already know what a lot of functions in the exe do from all of the previous reversing work?
Title: Re: [FF7 PC] Full source code reversing project?
Post by: ergonomy_joe on 2015-12-19 07:01:34
Funny, decompiling Final Fantasy VII is what I have been doing for some times now.
The PC version I mean, and the patch version 1.02.
I succeeded in reversing most of the minigames (except condor) and most of the main systems (except battle) so far.
It's not easily readable (I use their addresses to name the functions and the global so far, and do not plan to rename them until all is done), but you can get a lot of information from it.
Since it's the most comprehensible of the lot I wanted to post the code to the roller-coaster mini game, but I guess I won't be allowed to do it here right ?
(though I don't understand why posting reversed data is less infringing than posting reversed code)
Title: Re: [FF7 PC] Full source code reversing project?
Post by: paul on 2015-12-19 11:25:36
RE: (though I don't understand why posting reversed data is less infringing than posting reversed code).

I don't know why either since reversed data is a 1:1 copy of the copyrighted data, yet the reversed code is an interpretation of machine code back to something that's more human readable. I.e asm -> c, the c you've come up with can't be covered by copyright.

What format is your data in? I assume you have real or pesudo C code of the reversed functions? My plan was to do something like the RC2 project where we patch the EXE to load a DLL. Then this DLL replaces all of the known functions with a reimplementation. Since the graphics output uses a "driver" system thats probably the easiest part. Perhaps you've already reversed this and the functions for loading resources?

If you ever use IRC you could join the qhimm channel to discuss further?
Title: Re: [FF7 PC] Full source code reversing project?
Post by: ergonomy_joe on 2015-12-19 12:51:47
I'll try to clarify a little how I do:
.I translate from disassembled code to C code by hand, function after function (they are easy to spot in the ASM code)
.I then use --what I believe is-- the compiler originaly used, Visual C++ 5.0, to create an object file
.I then use an original tool to compare this object file to FF7.EXE to spot any translation error

Here is sample, the UPDATE callback to the Coast shooting mini-game:
Code: [Select]
//coaster[UPDATE][callback]
void C_005E8E7E(struct t_aa0 *bp08) {
C_005E8D03(0, 0, 0, 1.0f);//coaster:clear buffers
//%%% check ending condition? %%%
if(D_00C3F75C * 4  > D_00C3F894 - 0x10 || D_00C3F774 == 1) {//else 005E8ECB
C_005E988B();//sound related(6)
C_005E8E0B(bp08);//to mainDispatcher for coaster
return;
}
//%%% %%%
C_005E8F9B(bp08);//coaster:next frame
C_0041A21E(bp08);//Refresh input driver?
if(C_00660EC0(0, bp08)) {//G_DRV_88:BeginScene
C_00666DA3(bp08);//calls "instance:reset"
C_00666DC0(bp08);//calls "dx_sfx:reset heaps(1)"
C_00666DDD(bp08);//reset "deferred heap"
//-- refresh without display --
while(D_00C3F6EC + 1.0f < D_00C3F6E8) {
D_009014A8 = 0;
C_005E9051(bp08);//coaster.refresh
D_00C3F6EC += 1.0f;
}
//-- refresh with display --
D_009014A8 = 1;
C_005E9051(bp08);//coaster.refresh
D_00C3F6EC += 1.0f;
//-- --
C_00660EEB(bp08);//G_DRV_8C:EndScene
}
C_005E8E06();//<empty>
}

You can find others amples on my (not very good) blog concerning decompilation: http://magnetiktank.blogspot.jp/
Title: Re: [FF7 PC] Full source code reversing project?
Post by: paul on 2015-12-19 13:24:02
Wow pretty interesting stuff!

Also:

"To give you an idea of how far I went, let's say that except for the Battle system (the biggest of all), the Condor Fort system and part of the Menu system related to the battle system, I could reverse all the executable file to C source code (some parts harder to read than others) and recompile to a runnable file. Which means that the fun starts ... now !"

So you can compile your source and get a binary that runs the game - except for battles? Seems quite extreme to go to the level of completely binary compatibility!
Title: Re: [FF7 PC] Full source code reversing project?
Post by: DLPB on 2015-12-19 15:22:13
I need the submarine game, I'd find that very very interesting. See, from what i could tell, there is a TON of unused/dupe data in there.  And very strange goings on.  I think originally they were planning yo have 5 distinct difficulties with different time limits... but ran out of time.
Title: Re: [FF7 PC] Full source code reversing project?
Post by: Covarr on 2015-12-19 16:56:31
(though I don't understand why posting reversed data is less infringing than posting reversed code)
If you mean things like extracted/converted models, etc., we don't allow that to be posted here either.
Title: Re: [FF7 PC] Full source code reversing project?
Post by: KnifeTheSky77 on 2015-12-19 17:28:19
Wow, that is really impressive. How much time has it taken you to get to this point?

Where can I look at more of this code?

This 'reversed' code is really no different than recreating the backgrounds, which are allowed. You look at it, analyse it and recreate it by your own means.
Title: Re: [FF7 PC] Full source code reversing project?
Post by: Kaldarasha on 2015-12-19 17:39:47
Wow, that is really impressive. How much time has it taken you to get to this point?

Where can I look at more of this code?

This 'reversed' code is really no different than recreating the backgrounds, which are allowed. You look at it, analyse it and recreate it by your own means.

I do agree. It is like as if you would show a part of a picture but not the picture itself. Also the code by itself doesn't run the game.
Title: Re: [FF7 PC] Full source code reversing project?
Post by: Jaki on 2015-12-19 20:13:40
.
Title: Re: [FF7 PC] Full source code reversing project?
Post by: paul on 2015-12-19 22:49:31
Yea, keep dreaming. It's never going to happen!

What isn't going to happen?
Title: Re: [FF7 PC] Full source code reversing project?
Post by: ergonomy_joe on 2015-12-19 23:10:08
If you mean things like extracted/converted models, etc., we don't allow that to be posted here either.

I understand. That will be hard to discuss reverse-engineering if I can't post some source code though. Maybe one function at a time is acceptable ?

So you can compile your source and get a binary that runs the game - except for battles? Seems quite extreme to go to the level of completely binary compatibility!

It builds fine, and it runs (I have to change two or three things in the source to make it run on Win 8 though). Funny when you think I'm using a build environment from 1997. As for the extremity, that's because it's the only way to be sure my code has the closest shape to the original source code.
It took a couple of years so far.

I need the submarine game, I'd find that very very interesting. See, from what i could tell, there is a TON of unused/dupe data in there.  And very strange goings on.  I think originally they were planning yo have 5 distinct difficulties with different time limits... but ran out of time.

This one is interesting: all the game's data (except the textures) is embedded in the source code.
What kind of "strange goings" are you referring to ?
Title: Re: [FF7 PC] Full source code reversing project?
Post by: Kaldarasha on 2015-12-19 23:25:37
the ff7 exe was freely shared by Eidos so there shouldn't be peoblem with sharing codes about it.
Title: Re: [FF7 PC] Full source code reversing project?
Post by: paul on 2015-12-19 23:36:00
ergonomy_joe do you actually plan to share your source at all? I'm quite keen to check it out and see how it runs :). Ideally creating a github project?
Title: Re: [FF7 PC] Full source code reversing project?
Post by: KnifeTheSky77 on 2015-12-20 00:08:10
ergonomy_joe should be able to share code that he has personally written, character for character, out of his own brain.

It is really interesting to see the logic underneath, even for the game over/insert disc. Those address-for-var names make it just a little harder to digest :)
Title: Re: [FF7 PC] Full source code reversing project?
Post by: ergonomy_joe on 2015-12-20 01:16:10
The conversation concerning the reversed source code has moved to private for now.
But to answer the questions:
.I would like to post my work somewhere
.I am aware of the legal issues

In the meantime I will try to update my blog with bits of code.
Especially, some of you maybe aware of the fact that the ORIGINAL EXE includes 3 different graphic drivers (Direct3D hardware accelerated, software, [broken]OpenGL) plus the option to load the driver as an external DLL. The sofware part is really interesting: it is a complete STATE OF THE ART (for 1997) sofware rendering library. I find it hard to beleive that EIDOS developped it only for the FF7 port, but I couldn't find traces of it in other games of that era. Anyway, this part of the code is really fun (mostly inlined ASM) and I'd like to make some post about it.

BTY, I might not be able to post anything during weekdays so don't blame me for not answering please
Title: Re: [FF7 PC] Full source code reversing project?
Post by: paul on 2015-12-20 01:26:06
The conversation concerning the reversed source code has moved to private for now.
But to answer the questions:
.I would like to post my work somewhere
.I am aware of the legal issues

In the meantime I will try to update my blog with bits of code.
Especially, some of you maybe aware of the fact that the ORIGINAL EXE includes 3 different graphic drivers (Direct3D hardware accelerated, software, [broken]OpenGL) plus the option to load the driver as an external DLL. The sofware part is really interesting: it is a complete STATE OF THE ART (for 1997) sofware rendering library. I find it hard to beleive that EIDOS developped it only for the FF7 port, but I couldn't find traces of it in other games of that era. Anyway, this part of the code is really fun (mostly inlined ASM) and I'd like to make some post about it.

BTY, I might not be able to post anything during weekdays so don't blame me for not answering please

I think what you've managed to achieve is pretty outstanding! Your method of verification is quite interesting too. Btw perhaps SW renderer could have been used in Tomb Raider ports?
Title: Re: [FF7 PC] Full source code reversing project?
Post by: Kaldarasha on 2015-12-20 07:23:50
Square did made the port not eidos. Eidos was chosen to publish the game because they had a name by PC players. Halkun has more info about that.
Title: Re: [FF7 PC] Full source code reversing project?
Post by: syntax error on 2015-12-26 20:45:00
Where I live its legal to rewrite c++ from assembler, because its your own work as long as you don't have been given a zip file of Squaresoft or Eidos code.

The FF VII engine is that big, that you better do it semi automatic like with
The state of the art multi platform open source decompiler:
https://github.com/uxmal/reko/
Title: Re: [FF7 PC] Full source code reversing project?
Post by: Vgr on 2015-12-26 23:31:45
I think where you live really doesn't matter, it's probably more like Japan's rules that matter. I'm not a lawyer though, so take this with a grain of salt.

That being said, I think disassembling and manually rewriting the code isn't illegal anywhere. Am still not a lawyer though, so don't take my word for it.
Title: Re: [FF7 PC] Full source code reversing project?
Post by: paul on 2015-12-26 23:50:13
Decompiler still almost needs a rewrite to obtain clean readable code, which is what ergonomy_joe has already done for 80% of the engine.
Title: Re: [FF7 PC] Full source code reversing project?
Post by: Vegeta_Ss4 on 2016-01-18 07:08:54
You can find others amples on my (not very good) blog concerning decompilation: http://magnetiktank.blogspot.jp/

Hey i think you have a great idea. And let me tell you have done a wonderful job.

Just by looking at your "lgp lib", seams very familiar with my lgp class i wrote from the wiki data and myst6re docs.

I'm specially interested in all hardcoded functions which access and modify directly some Savemap vars that can't be reversed from the script file.

So, if you guys are gonna start a new project count on me.
Title: Re: [FF7 PC] Full source code reversing project?
Post by: paul on 2016-01-19 16:27:41
That dude already has reversed it to the point of it compiles again and field works etc :)
Title: Re: [FF7 PC] Full source code reversing project?
Post by: Dark_Ansem on 2016-02-06 22:28:33
any update on this?
Title: Re: [FF7 PC] Full source code reversing project?
Post by: Tom on 2016-05-01 06:52:00
Any update on what ergonomy_joe is doing, from what I can see on his blog, he has most of the game working except the battle3d module!  Congrats

As to what I have been doing, is trying to figure out a bit battle stuff and reversing the battle3d module
Title: Re: [FF7 PC] Full source code reversing project?
Post by: sithlord48 on 2016-05-01 12:40:01
From my understanding Edios was chosen because they had a project they had already ported to the pc from psx . A little game called Tomb Raider, at that time there was no other psx -> pc ports 
Title: Re: [FF7 PC] Full source code reversing project?
Post by: Tom on 2016-05-01 14:41:07
You think the company that did the port still has the source somewhere on some forgotten disc?
Title: Re: [FF7 PC] Full source code reversing project?
Post by: Tom on 2016-07-04 13:51:40
Any update on what ergonomy_joe has done?  I have not seen anything on his blog for a while
Title: Re: [FF7 PC] Full source code reversing project?
Post by: Kaldarasha on 2016-07-04 16:49:06
I saved this discussion well in my head and from time to time I check his blog as well. While it sad there is no progress but what can we do with the current information? It would be great if we could start to replace the modules through dll's even if they do nothing yet, but this could be a big step for moders.
Title: Re: [FF7 PC] Full source code reversing project?
Post by: paul on 2016-07-05 11:22:12
Auto decompilers simply do not work. This will result in a horrible mess that is unlikely to compile, if it does it will probably just crash. The real work done here is in understanding and documenting the internal game structures, not just the function and var names.
Title: Re: [FF7 PC] Full source code reversing project?
Post by: BesideTheVoid on 2016-07-07 16:36:51
You think the company that did the port still has the source somewhere on some forgotten disc?
One would think that by now it would have been leaked, or at least its fate would have been leaked, but who knows. Squaresoft made a whole studio to create FF7 and it was torn apart after the game was done (see http://q-gears.sourceforge.net/index.phtml?content=4 (http://q-gears.sourceforge.net/index.phtml?content=4)), but there is not a word anywhere AFAIK about what happened to EIDOS' copy of the code. Many fans and modders have been pining to know. Hopefully an insider who knows can post the info anonymously someday. I hope leaks about that will be more likely now that Square-Enix cares more about the remake now.
Title: Re: [FF7 PC] Full source code reversing project?
Post by: Tom on 2016-07-19 13:16:49
Auto decompilers simply do not work. This will result in a horrible mess that is unlikely to compile, if it does it will probably just crash. The real work done here is in understanding and documenting the internal game structures, not just the function and var names.


The gears document has a lot of stuff in it but with an auto decompiler maybe you can put together working stuff together with what you know from the gears doc.  I think thats how ergonomy_joe has been doing it.  Still, very interesting project and hope to see more!
Title: Re: [FF7 PC] Full source code reversing project?
Post by: paul on 2016-07-20 07:35:19
The gears document has a lot of stuff in it but with an auto decompiler maybe you can put together working stuff together with what you know from the gears doc.  I think thats how ergonomy_joe has been doing it.  Still, very interesting project and hope to see more!

No seriously you can't, read my last post. He has been doing it by reading the asm, converting to C by hand and compiling with the SAME compiler used on the real game. Then compares the ASM output is the same. You simply can't automate this and get meaningful source code at all.

Even if it worked you'd have 100,000's of vars and functions that overlap etc and have no meaning until someone looks at each one in turn and figures out what its doing.

E.g something like:

void* g1 = 0xdeadbeef;
void* g2 = 0xcafebabe;

int F12343()
{
  int v1 = *g1;
  int v2 = v1 + *g2;
  return v2;
}

Is useless when the real code would probably have been like:

struct Player
{
 int x,y;
};

struct World
{
 int x,y,w,h;
};

Player* gPlayer;
World* gWorld;

int GetPlayerWorldPos()
{
 return gPlayer->x + gWorld + x;
}
Title: Re: [FF7 PC] Full source code reversing project?
Post by: Tom on 2016-07-21 13:14:49
If you have code that compiles, even if you don't know how any of the variables work, just by having compilable code you can add in bits of code that change the variables and see how the game responds and name them appropriately, ex comment out a function, see what breaks etc etc.

Also, the way he is going it is great, I would like to learn how he does it.
Title: Re: [FF7 PC] Full source code reversing project?
Post by: paul on 2016-07-22 22:52:22
I've just told you how he does it :P Having compliable code that is unreadable and unstructured is useless, changing asm is easier in that respect. The process of updating them all would literally take years of work.

Title: Re: [FF7 PC] Full source code reversing project?
Post by: KnifeTheSky77 on 2016-07-22 23:49:57
Building a house out of toothpicks isn't a great idea. If you are willing to spend THAT kind of time, you'd be better off just recreating that game in some engine and recreating the assets in modern formats. Don't even both trying to convert scripts, rewrite them yourself
Title: Re: [FF7 PC] Full source code reversing project?
Post by: Tom on 2016-07-23 16:17:34
Building a house out of toothpicks isn't a great idea. If you are willing to spend THAT kind of time, you'd be better off just recreating that game in some engine and recreating the assets in modern formats. Don't even both trying to convert scripts, rewrite them yourself
Please don't say such things! Its far worse to recreate the game in a different engine and use modern formats.  First of all, there is no advantage in using modern formats to store the games data.  You gain nothing and give yourself a nightmare worth of converting.  Also please don't just throw around the just "rewrite them yourself",  I have attempted to do that with the Midgar Conversion Project: http://forums.qhimm.com/index.php?topic=15744.0 AND with Finishing Touch: http://forums.qhimm.com/index.php?topic=16211.0 both times I realized it is utterly impossible to rewrite or convert the scripts since it takes about 5 days to a week to convert a field working 2-3 hours a day.  Given there are only around 50 weeks in a year and there are over 700 fields ehh I wouldn't bother.
The fastest solution to get the game working is to re-create the engine that was used to run the game in the first place so you only have to worry about the engine and not the data.  Early versions of Q-Gears accomplished this and progress was going smooth until suddenly the code was rewritten.
What ergonomy_joe is doing is great and is the proper way to do it because fighting with the engine is a winnable battle, fighting with the data is overwhelming and a certain loss.

I want  to find out how he does it and try to do it myself.  How does he get the games ASM?  How do you split the games code into files that you can compare?  Then once I recompile it with the compiler used to make the game what do I compare?
Title: Re: [FF7 PC] Full source code reversing project?
Post by: KnifeTheSky77 on 2016-07-23 18:27:15
He disassembles the .exe and spends hours upon hours digging through assembler language, reverse engineering 'chunks' of it at a time, handwriting that into human readable code. It certainly isn't an automated process, and it definitely can't be a fast/enjoyable process.

A heavily tooled engine with a closely tie-ed in scripting system and model/asset manager would do you much better in the long run. Best case scenario: you have approximated very old and provably buggy code.

Using modern formats means easy usage in modern day software.

Just my opinion though. If you are hell bent on manually parsing assembler language, you might want to get a PhD in asm first :P

Edit: Unity offers a free version of their stuff for personal use --> https://store.unity.com/products/unity-personal (https://store.unity.com/products/unity-personal)  :mrgreen:
Title: Re: [FF7 PC] Full source code reversing project?
Post by: Tom on 2016-07-24 08:50:23
Not interested in unity for rendering triangles and pictures...
The closest to what we want is OGRE and SDL, ff7 needs a renderer, nothing more.  And using modern formats is not in any way easy because you have to manually convert the data to the "modern" formats which is not fast/enjoyable either.  The amount of LUA you would need for the field scripts in the game exceeds the amount of C code that makes the game run. 
For an example nmkin_1 which is a simple field compared to many many others is 1867 lines.
nivgate which is a Nibelheim gate field is 2874 lines and its still a non complex field.
I don't even want to think about the mayor's minigame in Midgar Tower how big it would be...

Lets say on average the each field has 1500 lines of code which is generous.  Multiply that by 700, and you got 1,050,000 lines.  Not counting savemap scripts, battle scripts, battle AI scripts and world map scripts.  Lets face it people, including myself have tried to convert the games data to alternative formats like LUA for use with new engines and have failed many times.  Lets try something different and target the engine this time.

Not implying anywhere that unity is not good, its great for making games and physics simulation but not suitable with that we can do here.
Title: Re: [FF7 PC] Full source code reversing project?
Post by: paul on 2016-07-24 10:43:59
What do you mean by target teh engine? What are you even trying to do? The only way to have a flexible engine is to make a new one either from scratch or using unity/unreal/whatever else.
Title: Re: [FF7 PC] Full source code reversing project?
Post by: Tom on 2016-07-24 10:46:57
We have Q-Gears engine Akari made that works with the original formats, 0.13 i think was the version?  It interprets the field opcodes like the original one did.  Theres also the new Q-Gears engine but that uses LUA and needs manual conversion to work
Title: Re: [FF7 PC] Full source code reversing project?
Post by: KnifeTheSky77 on 2016-07-24 18:17:36
I think you are missing my points here. Unity is heavily documented and incredibly flexible. It leaves you with a lot of head room. If you want to fulfill qgears' mission statement, you'll have to rewrite a lot of it and suffer inflexibility/no documentation in the process. You'd also need intimate knowledge of how ff7 engine works under the hood.

It is because qgears is beholden to ff7's reversed scripting that those scripts are so long. Using unity, you'd have tons of method to call upon that would greatly shorten scripts. This is not an apples to apples comparison.
Title: Re: [FF7 PC] Full source code reversing project?
Post by: Zervox on 2016-07-24 19:44:20
saying most/all projects trying to use another engine failed, can happen because of a numbers of factors.
-limited programming/engine knowledge.
-poor structuring
-badly interconnected system making them hard to understand and/or use
-too much focus on having 100% replication of how FF7 did it.
-limited time
-loss of interest/little interest shown(makes the project seem not worthwhile to complete)

Although not the same game, it still applies.
Afterall by comparison before I kinda lost the interest in my own programming something to emulate FF8 was simply time consumption but in approximately 2 months thanks to all data/information/tools for FF8 I implemented.
-Full angelscript real time compile for whenever I changed a player's limit break damage calculation or spell damage/heal/effects
-Behaviour Tree based battle AI
-Spell and junction system(both extremely flexible and capable of executing each their own damage/effect script)
-GFs and Limitbreaks
-All monster stats and level/stats scaling
-world exported from FF8 converted to heightmap then added to terrain system(3D).
-2D backscreens with 3d player ontop(as expected as a copy of these games, with multiple layers support allowing animated layers to be grouped,like curtains blowing the back and stop(emulating wind has stopped))
-Character UI, party switching, junction system, item list.
-Worldmap battle encounter, regional monster tables

Again this relies too much on
-how much time you are willing/can spend on it
-how much knowledge you have around programming and what areas you are capable of finishing.
-You have others helping you in the areas you can not do yourself
-good design and readability of your code(doing solid prework and ideas of the systems can help alot in how you will connect them)

lines of code for programmers means absolutely nothing, it is what those lines you have does which matters(I've seen alot of people writing huge amount of code which could've been handled in less than half the lines).

using opcodes and guesswork on what to change to see if it breaks or not, to me sounds like a hard way of programming when going to replicate/reversing projects.
Quote
Lets say on average the each field has 1500 lines of code which is generous.  Multiply that by 700, and you got 1,050,000 lines.
That is asuming you need 50% of the lines avoiding duplicate functions/code in the scripts(which probably almost every field shares in someway).

KnifeTheSky77 is correct about Unity.
Title: Re: [FF7 PC] Full source code reversing project?
Post by: Tom on 2016-07-25 15:26:57
The field script I wrote for that port were very much optimized from their originals, I had removed GOTOs from nearly everywhere and replaced them with IF/ELSEs and structured code, functions like locking the player and menu and making player invisible were grouped.  The fields do NOT use opcodes but LUA commands.  We don't "change an opcode and see what happens"

Code: [Select]
    FFVII.Data.triangleId = entity_manager:get_entity( "Cloud" ):get_move_triangle_id()
    if (FFVII.Data.triangleId == FFVII.Data.expectedTriangleId) then
      if (FFVII.Data.progress_game < 7) then
        entity_manager:player_lock( true )
        -- field:menu_lock( true )
        background2d:scroll_to_position( -96 * 3, 60 * 3, Background2D.SMOOTH, 1.066667 )
        script:request( Script.ENTITY, "Cloud", "scene_part_1", 6 )
        FFVII.Data.progress_game = 7
        script:request_end_sync( Script.ENTITY, "Biggs", "scene_part_2", 6 )
        script:wait( 0.2 )
        script:request_end_sync( Script.ENTITY, "Jessie", "scene_part_3", 6 )
        script:wait( 0.333333 )
        script:request_end_sync( Script.ENTITY, "Biggs", "scene_part_4", 5 )
        script:wait( 0.266667 )
        script:request_end_sync( Script.ENTITY, "Cloud", "scene_part_6", 6 )
        script:wait( 0.266667 )
        script:request_end_sync( Script.ENTITY, "Barret", "scene_part_8", 6 )
        script:wait( 0.4 )
        -- music:execute_akao( 0x20, 52, 64 )
        script:wait( 0.266667 )
        -- music:execute_akao( 0x20, 32, 64 )
        script:request( Script.ENTITY, "DoorLeft", "open", 6 )
        script:request( Script.ENTITY, "DoorRight", "open", 6 )
        script:wait( 0.333333 )
        script:request( Script.ENTITY, "Jessie", "scene_part_10", 6 )
        script:wait( 1 )
        script:request( Script.ENTITY, "Biggs", "scene_part_11", 6 )
        script:wait( 0.2 )
        script:request( Script.ENTITY, "Wedge", "scene_part_12", 6 )
        script:wait( 0.5 )
        script:request_end_sync( Script.ENTITY, "Barret", "scene_part_13", 6 )
        -- Prepare map change
        script:request( Script.ENTITY, "Barret", "scene_part_15", 6 )
        script:wait( 1 )
        -- field:movie_set( 20 )
        FFVII.Data.bMoviePlaying = 1
        -- field:play_movie()
        load_field_map_request("ffvii_md1_2", "md1_1_Director_on_update_addr_1044")
        entity_manager:player_lock( false )
        -- field:menu_lock( false )
      end
    end

That is what field script looks like.  And yes its very documented, there is a page that contains these commands and another one explaining what they do.  https://github.com/q-gears/q-gears/wiki/Field-Script-Command-Reference
The main problem that keeps Q-Gears behind is the lack of C++ programming on the side of the engine / converter.  However if we were to use Akari's Engine the conversion would be cut to bits because we would just use the original data anyways.

Remember: All the data conversion must be done programmatically because we can not distribute converted data!  This means each user will have to have a program that converts all the original data to the XYZ's Engine Data and the conversion can not be done by hand.  Unless off course the engine can use the original data.
Title: Re: [FF7 PC] Full source code reversing project?
Post by: paul on 2016-07-25 20:01:59
If I had time to spend on it I would just get the field module 100% working, but I'd probably also design it so it can work with FF7-9 since a lot of the core functionality is the same.
Title: Re: [FF7 PC] Full source code reversing project?
Post by: Zervox on 2016-07-25 20:54:48
Remember: All the data conversion must be done programmatically because we can not distribute converted data!  This means each user will have to have a program that converts all the original data to the XYZ's Engine Data and the conversion can not be done by hand.  Unless off course the engine can use the original data.

You do not need a program for every user ,this can and is done in alot of other reverse engineered engines so that when you start up the project for the first time it converts data.
This does of course rely on either having preconverted scripts for each asset(lets say field scripts, and yes these hand converted scripts you can distribute) or something which can interpret and convert then save.

some reversed projects I've seen does both, it will emulate and use original scripts or assets until converter works fully/a replacement script(can be done 1 script or 1 asset at a time) is made.
Title: Re: [FF7 PC] Full source code reversing project?
Post by: Tom on 2016-07-26 13:36:12
I have been told that I can not distribute converted scripts even if I have hand converted them ...
Title: Re: [FF7 PC] Full source code reversing project?
Post by: paul on 2016-07-26 17:23:42
Even if you could whats the point with out the sounds/fields etc? Also its a grey area and would get you C&D'ed for "porting" the scripts.
Title: Re: [FF7 PC] Full source code reversing project?
Post by: Zervox on 2016-07-26 18:35:16
I've yet to see any of the projects I am following that reverses other games to see a C&D for converting/porting scripts to emulate/mimic/alter behaviour.
Afterall FF games does not have any ownership of the behavior their scripts executes, only the code directly(which only prevents you to copy paste, which you aren't doing when you convert by hand, you simply mimic the behavior of the script), if you write something which does the same thing, they really can't do anything about it.

paul: the point is that sounds,fields and models when formats are fully known can be automatically converted on first run per user(this is allowed as you aren't distributing anything) usually this is done by other projects by asking the user to find/select original game data folders/archives.

and even if converting opcodes/scripts might be hard there are multiple cases where this is also converted, even per script based scenarios.

Most countries I can think of also do not allow patent or directly copyrights code in that sense(unless you actually copied and distributed the original file, which you aren't doing).
I'd seriously start to wonder how people who only does texture touch ups for games(not making new similar ones, but only tweaks the original textures) is able to distribute their versions, afterall this to me seems to be exactly what modders who only increases the resolution of the original FMV/Textures(or uses original and adds filters to them then distribute them), details of models are doing.
Title: Re: [FF7 PC] Full source code reversing project?
Post by: paul on 2016-07-26 18:52:03
For games that are still being sold this does happen, not distributing any data is the only way to be 100% safe, unless of course you are a well vexed lawyer :P
Title: Re: [FF7 PC] Full source code reversing project?
Post by: Kaldarasha on 2016-07-26 20:52:44
Didn't Eidos themselves published a fully workable FF7.exe? So won't it be legitimate to share code of it?
Title: Re: [FF7 PC] Full source code reversing project?
Post by: Zervox on 2016-07-26 22:04:07
For games that are still being sold this does happen, not distributing any data is the only way to be 100% safe, unless of course you are a well vexed lawyer :P

You aren't distributing any data belonging to them, you are distributing converted code to make the data work, still requiring every user to launch it and link to original data folders/archives for the conversion to convert original data that they already have by having bought the game.
Title: Re: [FF7 PC] Full source code reversing project?
Post by: paul on 2016-07-26 22:23:41
You aren't distributing any data belonging to them, you are distributing converted code to make the data work, still requiring every user to launch it and link to original data folders/archives for the conversion to convert original data that they already have by having bought the game.

So was the chrono trigger remake, 100% remade assets/scripts/code and they got C&D. Its a heavily grey area and the same laws don't apply everywhere. Might as well just make it use the original data files rather than risk being shut down IMO.

Title: Re: [FF7 PC] Full source code reversing project?
Post by: Tom on 2016-07-26 22:52:10
I totally agree with Paul, we can't take such risks.  It's better to use original scripts because it's not possible for a program to covert them to a higher level language automatically and distributing files will get us shut down and we can't have that happen.
Title: Re: [FF7 PC] Full source code reversing project?
Post by: Zervox on 2016-07-27 01:01:45
"100% remade assets/scripts/code"
Great so they resorted to plagiarism. they full duplicated every scene and behavior and required no original copy ownership for a user to play it, see the problem here?

"It's better to use original scripts because it's not possible for a program to covert them to a higher level language automatically"
Yet there are projects which do, some of the Fallout 2 reverse engineered projects actually do support converting scripts.

It's fine not wanting to take risk to be 100% sure, but the Chrono Trigger case was quite obviously outside of the gray area and inside the area of just blatantly copying it.

I am not going to argue which is better and what to do, just wanting to state that there are ways of doing things which is perfectly legal, and as I said, won't work unless a good sum of planning is done beforehand.

It's still a discussion which I enjoy.  :) ;)
Title: Re: [FF7 PC] Full source code reversing project?
Post by: paul on 2016-07-27 11:57:39
"It's better to use original scripts because it's not possible for a program to covert them to a higher level language automatically"

It is possible, its not just not easy to do since it means writing a decompiler that produces structured code.
Title: Re: [FF7 PC] Full source code reversing project?
Post by: Tom on 2016-07-27 14:05:41
I don't know of any decompilers that produce structured runnable code at the moment.  IDA code is not compilable.  So is Boomerang code not compileable.  However I know that paid services that decompile code and manually fix issues do work.
Title: Re: [FF7 PC] Full source code reversing project?
Post by: paul on 2016-07-27 16:09:53
Try almost any C# decompiler, native code is another matter. And simple script byte code is certainly decompilable. And the topic here is the simple script byte code, I already said many posts ago that decompling native code is a lost cause :P
Title: Re: [FF7 PC] Full source code reversing project?
Post by: KnifeTheSky77 on 2016-07-27 20:40:26
It is possible, its not just not easy to do since it means writing a decompiler that produces structured code.

Maybe you should get back to work on that?? :):)
Title: Re: [FF7 PC] Full source code reversing project?
Post by: Tom on 2016-07-27 21:07:26
True that or Akari engine 0.14
Title: Re: [FF7 PC] Full source code reversing project?
Post by: paul on 2016-07-28 07:01:01
Maybe you should get back to work on that?? :):)

I guess because I lost interest due to needless complexity. :)
Title: Re: [FF7 PC] Full source code reversing project?
Post by: Tom on 2016-07-28 12:42:17
Complexity about which part?  Needless, can we make it simpler the decompiler?
Title: Re: [FF7 PC] Full source code reversing project?
Post by: paul on 2016-07-29 11:10:37
The whole decompiler itself is a needless complexity VS using the original file formats directly.
Title: Re: [FF7 PC] Full source code reversing project?
Post by: KnifeTheSky77 on 2016-07-30 04:25:39
On the bright side, a lot was reversed in the process. I imagine there is still quite a bit to be discovered
Title: Re: [FF7 PC] Full source code reversing project?
Post by: Tom on 2016-07-30 20:43:54
I agree with Paul, converting the data was bad idea,  using the original would allow rapid development focussing on the mechanics and cool parts and not files and directories :)
Title: Re: [FF7 PC] Full source code reversing project?
Post by: Hellbringer616 on 2016-07-30 21:14:58
OpenMW went the original data route, seems to make the most sense to me. They have support for other formats too, so that would fix issues with limitations of the old format would it not?
Title: Re: [FF7 PC] Full source code reversing project?
Post by: paul on 2016-07-31 00:09:05
I'm thinking maybe making the github repo a compliable version of the 0.14 source might be a good idea. Probably the code will need modernizing now though. I'm making the assumption that implementing the missing field opcodes is enough to get it working.
Title: Re: [FF7 PC] Full source code reversing project?
Post by: Tom on 2016-07-31 11:14:58
First of all we need to get the game working, we shouldn't even be talking about bypassing the games limitations until it works as the original did.  In the 0.14 version most op codes are implemented, enough to run the first mission
Title: Re: [FF7 PC] Full source code reversing project?
Post by: paul on 2016-07-31 13:57:41
Seems 0.14 source is no where to be found
Title: Re: [FF7 PC] Full source code reversing project?
Post by: Kranmer on 2016-07-31 18:21:22
Seems 0.14 source is no where to be found
You can still get it at the original location
https://sourceforge.net/p/q-gears/code/ci/default/tree/media/version_old/
0.14 src is the 2nd from bottom file named "2007-05_q-gears_v0.14r_src.zip"
Title: Re: [FF7 PC] Full source code reversing project?
Post by: KnifeTheSky77 on 2016-07-31 18:38:59
Why regress all the way back to 2007 codebase? Looks like Ogre has a new version out, api docs are still wack
Title: Re: [FF7 PC] Full source code reversing project?
Post by: paul on 2016-07-31 19:01:17
Because as discussed in this thread the old version uses the original formats and is far more progressed than then current "generic" version. Also Ogre is a massive dependency to carry, it takes 20 mins to compile latest version compared with about 3 mins of the old.
Title: Re: [FF7 PC] Full source code reversing project?
Post by: Tom on 2016-08-05 08:54:37
Exactly, paul is right, there is a lot of progress in that version on the original formats
Title: Re: [FF7 PC] Full source code reversing project?
Post by: Zervox on 2016-08-05 09:42:50
compile ogre as a library or as a dll that way you do not have to recompile Ogre source every time you make changes to the things which can be handled through inheritance or components.
Title: Re: [FF7 PC] Full source code reversing project?
Post by: paul on 2016-08-05 14:38:23
Ogre is still far more complex than SDL2 when FF7 engine its self is so simple. On things like Travis-CI you do have to compile it everytime or come up with somewhere to store it because its so large. Where as compling SDL2 on travis-CI takes 1 min. This does matter in open source projects that need free CI.


Edit: Oh btw the 20mins figure is for travis to build QGears when ogre is prebuilt!
Title: Re: [FF7 PC] Full source code reversing project?
Post by: Hellbringer616 on 2016-08-09 01:03:34
What about modability? I'm unfamiliar with SDL2 but ogre accepts a wide range of formats does it not? Would make it easier than trying to maintain an old undocumented format.

Not thay I'm promoting we convert anything to a new format, more like OpenMW where it accepts the old format as well as new formats.
Title: Re: [FF7 PC] Full source code reversing project?
Post by: KnifeTheSky77 on 2016-08-09 06:23:14
This proj should be approached from a feasibility pov at this point, go for it you madmen
Title: Re: [FF7 PC] Full source code reversing project?
Post by: paul on 2016-08-09 07:04:27
Yeah loading a a png or whatever is easy enough, seems the code will need some clean up though. Its still using SDL1, immediate mode OpenGL etc. Another striking issue is everything being global :)
Title: Re: [FF7 PC] Full source code reversing project?
Post by: Tom on 2016-12-18 20:13:47
Paul any progress with that?

Anyone else got any news on FF7 source code decompilation or remake?
Title: Re: [FF7 PC] Full source code reversing project?
Post by: paul on 2016-12-18 22:45:47
I believe re-creating just the field module via hooks in to the existing game is probably a viable starting point. That way if nothing else is completed at least there is a standalone end-to-end working field.
Title: Re: [FF7 PC] Full source code reversing project?
Post by: Tom on 2017-01-25 16:26:14
Any updates on that or similar projects?
Title: Re: [FF7 PC] Full source code reversing project?
Post by: ergonomy_joe on 2018-10-11 04:20:22
I uploaded two parts from my decompilation work:
- the coaster minigame: https://github.com/ergonomy-joe/ff7-coaster
- the world map engine: https://github.com/ergonomy-joe/ff7-worldmap

The code once compiled with visual studio 97 is almost the same as the original code.
But this time I made some arrangement:
.this project build with Visual Studio 2008 (although it may be easy to build it with more recent versions)
.part of the original code is included as LIB files: the main library FF7LIB, the sound library FF7SND and the menu library FF7MENU (I may release as source code later but for now you have to use the libraries).
.I patched a little the original code so that it runs in a window (with a glitch though)
.both projects work as standalone. I included an original .cpp file in both case to launch both engines. Respectively CoasterMain.cpp and FF7WorldMap.cpp.
.you need to build/link with DirectX SDK version 5.0. I didn't try with newer versions. It may work too, I don't know.

Enjoy !