[FF7 PC] Full source code reversing project?

[Vgr]

I think where you live really doesn't matter, it's probably more like Japan's rules that matter. I'm not a lawyer though, so take this with a grain of salt.

That being said, I think disassembling and manually rewriting the code isn't illegal anywhere. Am still not a lawyer though, so don't take my word for it.

[paul]

Decompiler still almost needs a rewrite to obtain clean readable code, which is what ergonomy_joe has already done for 80% of the engine.

[Vegeta_Ss4]

You can find others amples on my (not very good) blog concerning decompilation: http://magnetiktank.blogspot.jp/

Hey i think you have a great idea. And let me tell you have done a wonderful job.

Just by looking at your "lgp lib", seams very familiar with my lgp class i wrote from the wiki data and myst6re docs.

I'm specially interested in all hardcoded functions which access and modify directly some Savemap vars that can't be reversed from the script file.

So, if you guys are gonna start a new project count on me.

[paul]

That dude already has reversed it to the point of it compiles again and field works etc

[Dark_Ansem]

any update on this?

[Tom]

Any update on what ergonomy_joe is doing, from what I can see on his blog, he has most of the game working except the battle3d module!  Congrats

As to what I have been doing, is trying to figure out a bit battle stuff and reversing the battle3d module

[sithlord48]

From my understanding Edios was chosen because they had a project they had already ported to the pc from psx . A little game called Tomb Raider, at that time there was no other psx -> pc ports 

[Tom]

You think the company that did the port still has the source somewhere on some forgotten disc?

[Tom]

Any update on what ergonomy_joe has done?  I have not seen anything on his blog for a while

[Kaldarasha]

I saved this discussion well in my head and from time to time I check his blog as well. While it sad there is no progress but what can we do with the current information? It would be great if we could start to replace the modules through dll's even if they do nothing yet, but this could be a big step for moders.

[paul]

Auto decompilers simply do not work. This will result in a horrible mess that is unlikely to compile, if it does it will probably just crash. The real work done here is in understanding and documenting the internal game structures, not just the function and var names.

[BesideTheVoid]

You think the company that did the port still has the source somewhere on some forgotten disc?

One would think that by now it would have been leaked, or at least its fate would have been leaked, but who knows. Squaresoft made a whole studio to create FF7 and it was torn apart after the game was done (see http://q-gears.sourceforge.net/index.phtml?content=4), but there is not a word anywhere AFAIK about what happened to EIDOS' copy of the code. Many fans and modders have been pining to know. Hopefully an insider who knows can post the info anonymously someday. I hope leaks about that will be more likely now that Square-Enix cares more about the remake now.

[Tom]

Auto decompilers simply do not work. This will result in a horrible mess that is unlikely to compile, if it does it will probably just crash. The real work done here is in understanding and documenting the internal game structures, not just the function and var names.

The gears document has a lot of stuff in it but with an auto decompiler maybe you can put together working stuff together with what you know from the gears doc.  I think thats how ergonomy_joe has been doing it.  Still, very interesting project and hope to see more!

[paul]

The gears document has a lot of stuff in it but with an auto decompiler maybe you can put together working stuff together with what you know from the gears doc.  I think thats how ergonomy_joe has been doing it.  Still, very interesting project and hope to see more!

No seriously you can't, read my last post. He has been doing it by reading the asm, converting to C by hand and compiling with the SAME compiler used on the real game. Then compares the ASM output is the same. You simply can't automate this and get meaningful source code at all.

Even if it worked you'd have 100,000's of vars and functions that overlap etc and have no meaning until someone looks at each one in turn and figures out what its doing.

E.g something like:

void* g1 = 0xdeadbeef;
void* g2 = 0xcafebabe;

int F12343()
{
  int v1 = *g1;
  int v2 = v1 + *g2;
  return v2;
}

Is useless when the real code would probably have been like:

struct Player
{
 int x,y;
};

struct World
{
 int x,y,w,h;
};

Player* gPlayer;
World* gWorld;

int GetPlayerWorldPos()
{
 return gPlayer->x + gWorld + x;
}

[Tom]

If you have code that compiles, even if you don't know how any of the variables work, just by having compilable code you can add in bits of code that change the variables and see how the game responds and name them appropriately, ex comment out a function, see what breaks etc etc.

Also, the way he is going it is great, I would like to learn how he does it.

[paul]

I've just told you how he does it Having compliable code that is unreadable and unstructured is useless, changing asm is easier in that respect. The process of updating them all would literally take years of work.

[KnifeTheSky77]

Building a house out of toothpicks isn't a great idea. If you are willing to spend THAT kind of time, you'd be better off just recreating that game in some engine and recreating the assets in modern formats. Don't even both trying to convert scripts, rewrite them yourself

[Tom]

Building a house out of toothpicks isn't a great idea. If you are willing to spend THAT kind of time, you'd be better off just recreating that game in some engine and recreating the assets in modern formats. Don't even both trying to convert scripts, rewrite them yourself

Please don't say such things! Its far worse to recreate the game in a different engine and use modern formats.  First of all, there is no advantage in using modern formats to store the games data.  You gain nothing and give yourself a nightmare worth of converting.  Also please don't just throw around the just "rewrite them yourself",  I have attempted to do that with the Midgar Conversion Project: http://forums.qhimm.com/index.php?topic=15744.0 AND with Finishing Touch: http://forums.qhimm.com/index.php?topic=16211.0 both times I realized it is utterly impossible to rewrite or convert the scripts since it takes about 5 days to a week to convert a field working 2-3 hours a day.  Given there are only around 50 weeks in a year and there are over 700 fields ehh I wouldn't bother.
The fastest solution to get the game working is to re-create the engine that was used to run the game in the first place so you only have to worry about the engine and not the data.  Early versions of Q-Gears accomplished this and progress was going smooth until suddenly the code was rewritten.
What ergonomy_joe is doing is great and is the proper way to do it because fighting with the engine is a winnable battle, fighting with the data is overwhelming and a certain loss.

I want  to find out how he does it and try to do it myself.  How does he get the games ASM?  How do you split the games code into files that you can compare?  Then once I recompile it with the compiler used to make the game what do I compare?

[KnifeTheSky77]

He disassembles the .exe and spends hours upon hours digging through assembler language, reverse engineering 'chunks' of it at a time, handwriting that into human readable code. It certainly isn't an automated process, and it definitely can't be a fast/enjoyable process.

A heavily tooled engine with a closely tie-ed in scripting system and model/asset manager would do you much better in the long run. Best case scenario: you have approximated very old and provably buggy code.

Using modern formats means easy usage in modern day software.

Just my opinion though. If you are hell bent on manually parsing assembler language, you might want to get a PhD in asm first

Edit: Unity offers a free version of their stuff for personal use --> https://store.unity.com/products/unity-personal 

[Tom]

Not interested in unity for rendering triangles and pictures...
The closest to what we want is OGRE and SDL, ff7 needs a renderer, nothing more.  And using modern formats is not in any way easy because you have to manually convert the data to the "modern" formats which is not fast/enjoyable either.  The amount of LUA you would need for the field scripts in the game exceeds the amount of C code that makes the game run. 
For an example nmkin_1 which is a simple field compared to many many others is 1867 lines.
nivgate which is a Nibelheim gate field is 2874 lines and its still a non complex field.
I don't even want to think about the mayor's minigame in Midgar Tower how big it would be...

Lets say on average the each field has 1500 lines of code which is generous.  Multiply that by 700, and you got 1,050,000 lines.  Not counting savemap scripts, battle scripts, battle AI scripts and world map scripts.  Lets face it people, including myself have tried to convert the games data to alternative formats like LUA for use with new engines and have failed many times.  Lets try something different and target the engine this time.

Not implying anywhere that unity is not good, its great for making games and physics simulation but not suitable with that we can do here.

[paul]

What do you mean by target teh engine? What are you even trying to do? The only way to have a flexible engine is to make a new one either from scratch or using unity/unreal/whatever else.

[Tom]

We have Q-Gears engine Akari made that works with the original formats, 0.13 i think was the version?  It interprets the field opcodes like the original one did.  Theres also the new Q-Gears engine but that uses LUA and needs manual conversion to work

[KnifeTheSky77]

I think you are missing my points here. Unity is heavily documented and incredibly flexible. It leaves you with a lot of head room. If you want to fulfill qgears' mission statement, you'll have to rewrite a lot of it and suffer inflexibility/no documentation in the process. You'd also need intimate knowledge of how ff7 engine works under the hood.

It is because qgears is beholden to ff7's reversed scripting that those scripts are so long. Using unity, you'd have tons of method to call upon that would greatly shorten scripts. This is not an apples to apples comparison.

[Zervox]

saying most/all projects trying to use another engine failed, can happen because of a numbers of factors.
-limited programming/engine knowledge.
-poor structuring
-badly interconnected system making them hard to understand and/or use
-too much focus on having 100% replication of how FF7 did it.
-limited time
-loss of interest/little interest shown(makes the project seem not worthwhile to complete)

Although not the same game, it still applies.
Afterall by comparison before I kinda lost the interest in my own programming something to emulate FF8 was simply time consumption but in approximately 2 months thanks to all data/information/tools for FF8 I implemented.
-Full angelscript real time compile for whenever I changed a player's limit break damage calculation or spell damage/heal/effects
-Behaviour Tree based battle AI
-Spell and junction system(both extremely flexible and capable of executing each their own damage/effect script)
-GFs and Limitbreaks
-All monster stats and level/stats scaling
-world exported from FF8 converted to heightmap then added to terrain system(3D).
-2D backscreens with 3d player ontop(as expected as a copy of these games, with multiple layers support allowing animated layers to be grouped,like curtains blowing the back and stop(emulating wind has stopped))
-Character UI, party switching, junction system, item list.
-Worldmap battle encounter, regional monster tables

Again this relies too much on
-how much time you are willing/can spend on it
-how much knowledge you have around programming and what areas you are capable of finishing.
-You have others helping you in the areas you can not do yourself
-good design and readability of your code(doing solid prework and ideas of the systems can help alot in how you will connect them)

lines of code for programmers means absolutely nothing, it is what those lines you have does which matters(I've seen alot of people writing huge amount of code which could've been handled in less than half the lines).

using opcodes and guesswork on what to change to see if it breaks or not, to me sounds like a hard way of programming when going to replicate/reversing projects.

Lets say on average the each field has 1500 lines of code which is generous.  Multiply that by 700, and you got 1,050,000 lines.

That is asuming you need 50% of the lines avoiding duplicate functions/code in the scripts(which probably almost every field shares in someway).

KnifeTheSky77 is correct about Unity.

[Tom]

The field script I wrote for that port were very much optimized from their originals, I had removed GOTOs from nearly everywhere and replaced them with IF/ELSEs and structured code, functions like locking the player and menu and making player invisible were grouped.  The fields do NOT use opcodes but LUA commands.  We don't "change an opcode and see what happens"

Code: [Select]

    FFVII.Data.triangleId = entity_manager:get_entity( "Cloud" ):get_move_triangle_id()
    if (FFVII.Data.triangleId == FFVII.Data.expectedTriangleId) then
      if (FFVII.Data.progress_game < 7) then
        entity_manager:player_lock( true )
        -- field:menu_lock( true )
        background2d:scroll_to_position( -96 * 3, 60 * 3, Background2D.SMOOTH, 1.066667 )
        script:request( Script.ENTITY, "Cloud", "scene_part_1", 6 )
        FFVII.Data.progress_game = 7
        script:request_end_sync( Script.ENTITY, "Biggs", "scene_part_2", 6 )
        script:wait( 0.2 )
        script:request_end_sync( Script.ENTITY, "Jessie", "scene_part_3", 6 )
        script:wait( 0.333333 )
        script:request_end_sync( Script.ENTITY, "Biggs", "scene_part_4", 5 )
        script:wait( 0.266667 )
        script:request_end_sync( Script.ENTITY, "Cloud", "scene_part_6", 6 )
        script:wait( 0.266667 )
        script:request_end_sync( Script.ENTITY, "Barret", "scene_part_8", 6 )
        script:wait( 0.4 )
        -- music:execute_akao( 0x20, 52, 64 )
        script:wait( 0.266667 )
        -- music:execute_akao( 0x20, 32, 64 )
        script:request( Script.ENTITY, "DoorLeft", "open", 6 )
        script:request( Script.ENTITY, "DoorRight", "open", 6 )
        script:wait( 0.333333 )
        script:request( Script.ENTITY, "Jessie", "scene_part_10", 6 )
        script:wait( 1 )
        script:request( Script.ENTITY, "Biggs", "scene_part_11", 6 )
        script:wait( 0.2 )
        script:request( Script.ENTITY, "Wedge", "scene_part_12", 6 )
        script:wait( 0.5 )
        script:request_end_sync( Script.ENTITY, "Barret", "scene_part_13", 6 )
        -- Prepare map change
        script:request( Script.ENTITY, "Barret", "scene_part_15", 6 )
        script:wait( 1 )
        -- field:movie_set( 20 )
        FFVII.Data.bMoviePlaying = 1
        -- field:play_movie()
        load_field_map_request("ffvii_md1_2", "md1_1_Director_on_update_addr_1044")
        entity_manager:player_lock( false )
        -- field:menu_lock( false )
      end
    end

That is what field script looks like.  And yes its very documented, there is a page that contains these commands and another one explaining what they do.  https://github.com/q-gears/q-gears/wiki/Field-Script-Command-Reference
The main problem that keeps Q-Gears behind is the lack of C++ programming on the side of the engine / converter.  However if we were to use Akari's Engine the conversion would be cut to bits because we would just use the original data anyways.

Remember: All the data conversion must be done programmatically because we can not distribute converted data!  This means each user will have to have a program that converts all the original data to the XYZ's Engine Data and the conversion can not be done by hand.  Unless off course the engine can use the original data.