Messages

26

Scripting and Reverse Engineering / Re: looking for source code to read battle models and animations

« on: 2019-05-13 19:55:24 »

I found Wiki for PSX format, but not PC, and it's very different. The battle models are split into multiple files. For example, for Cloud, the files are named rtaa, rtab, rtac, etc. Apparently the animations are always in xxda (e.g. rtda for Cloud). But it's not clear to me yet how to determine which of the files are vertex data vs. texture data vs. etc.

I guess I could try to re-reverse-engineer it, e.g. by comparing PC vs PSX. But if there's anyone that has info or code or any insight, it would help a lot.

BTW, I have found 3 tools that can read battle model data: Ifalna, Leviathan, and biturn. But I can't find source code for them. Bleh.

27

Scripting and Reverse Engineering / Re: Two ideas: Binary data schema and auto-reverse-engineering tool

« on: 2019-05-13 19:47:11 »

Thanks for looking at that. Very good point. So, kaitai is worthless for editing, then, until they add write/serialization support.

28

Scripting and Reverse Engineering / Re: Two ideas: Binary data schema and auto-reverse-engineering tool

« on: 2019-05-13 16:07:40 »

Was discussing this in Discord with antiquechrono and... check this out:

https://kaitai.io/

29

Scripting and Reverse Engineering / looking for source code to read battle models and animations

« on: 2019-05-13 15:03:19 »

Anyone know where I can find source code for reading battle models and animations?

The only program I know that can do this is Ifalna by Ficedula, but all the links I've found so far are either broken or only point to a binary.

30

FF7 Tools / Re: PyFF7: Python 3 tools to manipulate FF7 files

« on: 2019-05-13 12:19:54 »

I have a snapshot of a pretty darn recent version of Aali's lgp utility (binary and source) here:

https://github.com/picklejar76/kujata/tree/master/lgp-0.5b

31

General Discussion / Re: FF7 Remake Teaser

« on: 2019-05-10 20:36:17 »

Would you rather live in a universe where Square just did a remaster, or where they're doing a remake?

The way I see it, the latter provides more value and potential, in proportion to "how different" it is.

Remember, not only can we mod the old game to be more "modern", but we'll also be able to mod the remake to be "more like the original".

Aww, you wanted it to be turn-based? Don't worry, I guarantee you there will be a mod soon enough to make it turn-based.

Aww, you wanted the choice to reject Aerith's flower? And for that to affect the date mechanics? Don't worry, there will be a mod for that.

The great thing about having a DIFFERENT remake is, it gives us a huge spectrum of possibilities. So more people can be pleased eventually. You just have to wait for the mod you want. Or write it yourself!

32

FF7 Tools / Re: PyFF7: Python 3 tools to manipulate FF7 files

« on: 2019-05-06 16:56:37 »

BTW, you said you tried "ulgp" and it didn't work in some cases? Which specific utility did you try, and which version? The reason I ask is, there are at least 2 or 3 different lgp utilities out there, and different versions, and links can get outdated, etc.

Also... I forgot to say the most important thing: WELCOME! This community can always use more smart developers. (I'm not an official forum moderator or anyone special. Just one of many people who love the game and reverse engineering it. Every time I try to let it go, I keep coming back.)

33

FF7 Tools / Re: PyFF7: Python 3 tools to manipulate FF7 files

« on: 2019-05-06 16:43:41 »

BTW, the Python suite I mentioned didn't seem to have lgp utility.

But you could look at Aali's lgp utility source code and maybe find the answer there.

I can't remember where to get Aali's source code, but I made a snapshot copy of it here:
https://github.com/picklejar76/kujata/tree/master/lgp-0.5b/src

Hope this helps!

(Edit: I originally said this was Ficedula's, but it's Aali's.)

34

FF7 Tools / Re: PyFF7: Python 3 tools to manipulate FF7 files

« on: 2019-05-06 16:36:17 »

You should check out this:
https://github.com/cebix/ff7tools

It's a whole suite of Python code for reading tons of FF7 asset files.

Basically, I'd suggest:
Any Python Dev should extend the above Python project.
Any C dev should look at sithlord's ff7 toolkit.
Any JS dev should look at Kujata.

35

General Discussion / Re: [FF7PC-98/Steam] Multiple mods - The Reunion (R05c)

« on: 2019-05-06 16:27:35 »

Thanks for adding that link!

36

General Discussion / Re: [FF7PC-98/Steam] Multiple mods - The Reunion (R05c)

« on: 2019-05-06 11:43:33 »

Where is "The Reunion Database"?

37

General Discussion / Re: [FF7PC-98/Steam] Multiple mods - The Reunion (R05c)

« on: 2019-05-05 17:13:00 »

DLPB: I received a comment saying they felt like the Enemy Attack that is currently translated as "The Emperor" (official) or "The Pope" (Beacause) should really be "The Heirophant". If you want more details, let me know. I doubt you are still entertaining ideas on changes anymore, but just FYI.

38

Scripting and Reverse Engineering / Re: Two ideas: Binary data schema and auto-reverse-engineering tool

« on: 2019-05-01 02:15:19 »

codemann8: yeah, that hex editor "010 Editor" looks pretty sweet, thanks for sharing! here's a screenshot for others interested:

39

Scripting and Reverse Engineering / Re: Two ideas: Binary data schema and auto-reverse-engineering tool

« on: 2019-05-01 00:34:02 »

Another challenge to keep in mind: There's a difference between the "original data" and "serialized data". So there are really two different schemas we're talking about here.

As a simple example, suppose a game had an "array of all Weapon objects", and each Weapon had a name and an attack value, and suppose the total number of weapons is not set in stone (until file-write time).


The schema for the "original data" might be something simple like:

schema for "AllWeapons" object:

• weapons: List of Weapon

sub-schema for "Weapon" object:

• name: string
• attackPower: uint16

The schema for the "serialized data", however, might be:

schema for "all-weapons.bin" binary file:

• fileSize: uint64
• numWeapons: uint32
• weaponSections: sequence of ${numWeapons} WeaponSections
• @integrityCheck: ${_data}.length == ${fileSize}

sub-schema for "WeaponSection" (serialized/chunk)

• sectionSize: uint32
• nameLength: uint32
• name: string of length {$nameLength}
• attackPower: uint16
• @integrityCheck: ${_data}.length == ${sectionSize}

So the "schemas" are different. The latter "schema" has "size" information, but that's only needed for deserialization logic.

Also, the serialized data might be encoded with gzip encoding or lzs encoding, whereas the original data might not.

Having the schema of just the original data is not good enough, because there are multiple ways to serialize it.

Having the schema of just the serialized data is much better, but ideally the schema should also somehow define how it is deserialized into the original data.

Also, deserializers can often work by reading data "in order", but serializers often work backwards, for example, they often write the total file size at the beginning of the file, which is the last thing they know.

So the challenge is, how a schema can define all of this stuff, instead of just one format, or the other format, or the transformation in between.

And yeah, as you pointed out codemann8, each file will definitely has its own nuances, so it definitely needs to support more than just the primitive data types like uint8 and such.

But hey... I'm sure we could come up with something!

40

Scripting and Reverse Engineering / Two ideas: Binary data schema and auto-reverse-engineering tool

« on: 2019-04-30 14:56:27 »

Idea #1: Language-agnostic Binary data schema

So... most of our projects involve consuming, editing, and writing binary data like Final Fantasy game asset files.

This community has been awesome as far as bringing people together to reverse engineer the file formats, publish their findings on Wiki, and sharing source code.

So now, it would be super nice if we could go just one step further and to create actual schema definitions, in a structured format. "That way, we could auto-generate documentation. And most importantly, I wouldn't have to develop code to parse the data, because it can just be parsed automatically, by my program, which by the way is written in... uh, my preferred programming language..."

Yeah. Exactly. That's why we don't have schema files. Everyone has different needs, prefers different programming languages, etc.

But still... just imagine the possibilities if we could agree on a language-agnostic schema. This is a common problem and there are solutions for it. For example, perhaps we could use one of these: https://en.wikipedia.org/wiki/Data_exchange#Data_exchange_languages

Even if the language-agnostic schema isn't super-friendly to your preferred programming language, it would not matter if there were utilities to generate documentation for you to read and understand it in a user-friendly way, and if there were utilities to translate the schema to data structures in your preferred programming language. Right?

Well, for new projects, yeah, that might be fantastic. But... for existing projects... well, suppose we did come up with such a plan and even developed the tools to generate docs and code in everyone's preferred language. Would we expect the authors of existing tools to re-factor their code to use the new auto-generated code? Probably not.

But, if you are such an author, then you probably have some good insight into what kind of features you would want such auto-generated code to have, in your preferred programming language. So, perhaps you would be interested in contributing ideas, or even code, to translating universal schema to "code in my preferred language".

And long-term, imagine the possibilities with future tools!

I'm sure this community has talked about this before numerous times, so my main question is, do we have a forum topic dedicated to this? If not, can we create one? And if there are any individual threads where this has been discussed at length, can you point to them?

Idea #2:

It would be cool to have a tool to auto-reverse-engineer new file formats. The tool could use a repository of known file formats / characteristics (like "magic numbers", other patterns to look for, etc.). It could even be interactive and show different interpretations to the user and let the user guide the tool to choose the right interpretation. Finally, it could output schemas for the user. What format should the schema be in, though? See idea #1 above. (See?! That's another reason we need it!)

41

General Discussion / Wiki

« on: 2019-04-29 20:35:11 »

I just signed up as a new member of the new *.ru Wiki site. It looks like my account was created successfully. And I can access edit pages (haven't tried to submit yet though) and I can see History pages. But, every time I click on a user link on a History page, I get a weird error like "user is not registered", even for Qhimm. Is this expected?

42

Scripting and Reverse Engineering / Re: Field IDs

« on: 2019-04-29 20:25:07 »

Interesting. BTW, we should update the Wiki with the stuff we're finding.

43

FF7 Tools / Re: [PC] Translate graphics models to glTF - Kujata (v1.0)

« on: 2019-04-27 23:30:22 »

BTW, I imagine that any modder working on models and animations would like to have a nice reference website that provides things like:

"Given a field model, show me a list of all field maps (flevels) and field scripts where the model is used."

"Given a field model animation, show me a list of all field maps (flevels) and field scripts where the animation is used."

I plan to update the Kujata website soon with some of this functionality.

44

FF7 Tools / Re: [PC] Translate graphics models to glTF - Kujata (v1.0)

« on: 2019-04-27 23:22:06 »

cmh175:

Q: "What's currently possible with this? ... Does this work with animations?"

A: It can translate FF7 field models to glTF, including animations, and including references to textures (which must be translated using a separate tool).

Example of translated output data: https://github.com/picklejar76/kujata-data/tree/master/data/field/char.lgp

Example of translated output visualized on a website: https://picklejar76.github.io/kujata-webapp/stage-viewer

The examples above do not include animations yet. But the tool can translate them. I have tested the translated glTF in a web browser using THREE.js, including animations, so the translation is pretty decent.

By the way, it uses Ifalna database (bundled with Kimera) to "guess" the default standing animation, and can optionally use the Ifalna database to allow you to include every single animation that is used by that model in the game, although that obviously makes the translated data huge.

Translating in the reverse direction (glTF back to FF7) is not yet supported, so it's not a complete modding solution yet, but I guess you could say it's about halfway there, and it already provides a lot of value.

45

Scripting and Reverse Engineering / Re: FF7 World/Field Map Scripts and Encounters

« on: 2019-04-21 22:46:02 »

Kujata also has world map metadata for wm0, wm1, etc., in these files:

coords: https://github.com/picklejar76/kujata-data/blob/master/metadata/field-id-to-world-map-coords.json

menu names: https://github.com/picklejar76/kujata-data/blob/master/metadata/wm-field-menu-names.json

links from fields to world map locations: https://github.com/picklejar76/kujata-data/blob/master/metadata/scene-graph.json

By the way, to get a little technical, there's a slight difference between the world map coordinates for entering and exiting a place. For example, when you exit a town from a field script to the world map, you typically appear at a very specific coordinate, slightly away from the town, and the main menu says your location is something like "Outside Kalm". However, to re-enter the town, you don't go to that exact same coordinate ("slightly away from the town"), you go to the location of the town itself. (And I assume you don't have to go to an exact coordinate, but more like a small area, perhaps triggered by a mesh triangle or a set of trigger lines.)

And of course there are the "special" scenarios where, instead of going to a specific coordinate, you go to something like "last field map location" or "relative location" (e.g. getting Yuffie, exiting Highwind, etc.)

46

Scripting and Reverse Engineering / Re: FF7 World/Field Map Scripts and Encounters

« on: 2019-04-19 20:20:36 »

I actually have a way to display field models in a webpage. The Kujata project translates the field graphics models and animations to glTF, and you can use a library like THREE.js to display the models and animations.

Whoa, I didn't see Makou Reactor's Mass Extract feature, until you just pointed out that it exists. Thanks a bunch! I'm going to try that right now!

47

Scripting and Reverse Engineering / Re: FF7 World/Field Map Scripts and Encounters

« on: 2019-04-19 19:35:59 »

Hey codemann8, your tool shows an image of the actual background. Are you generating the image directly from the tiles in the data, or are you getting it from some other outside source? (Those images would be really useful for a project I'm working on.)

48

Scripting and Reverse Engineering / Re: FF7 World/Field Map Scripts and Encounters

« on: 2019-04-11 00:26:16 »

Oooooh, very nice job, man!

How do you decide the boundaries of a group? Is it automatic, or are you adding the nodes manually, via the tool itself? Looks like the latter? And I'm guessing that each group is basically a section of a walkthrough/ guide? Like, if we compare to the Prima Strategy Guide book, you'd have like 1 group per chapter?

49

Scripting and Reverse Engineering / Re: FF7 World/Field Map Scripts and Encounters

« on: 2019-04-10 19:55:04 »

Hey codemann8 and DLPB, check this out. It's basically codemann8's world map image + nodes for every worldmap ID and field ID + links between them:

https://picklejar76.github.io/kujata-webapp/scene-graph.html

You can zoom and pan with your mouse. (Didn't test on mobile device, not sure how well that page works on a phone.)

Here's a screenshot in case it doesn't work in your browser:



The data is not clean, so you'll see a lot of "bugs" in the graph, but it's definitely "interesting".

Like codemann8 alluded to, it probably doesn't make sense to try to put all the nodes on a single graph like this, it would probably be better to split it into smaller sections. But here it is anyway. /shrug

50

Scripting and Reverse Engineering / Re: FF7 World/Field Map Scripts and Encounters

« on: 2019-04-10 19:35:54 »

You're exactly right, there are many scenarios that make this tricky: GameMoment, walkmesh reachability, etc. Other examples might be "whether you have a Gold Chocobo", "whether you have a key item in your inventory", "whether Vincent is in your party", "how many Fort Condor battles you've fought". Anyway, yeah, you might think of this as "extra dimensions", or maybe it could be implemented as "conditions" tagged to a link?

And yeah, there's a lot of trickiness involving how to connect field maps together. For example, initially, my graphs were connecting Kalm and Nibelheim together, because when your party reaches the inn in Kalm and when Cloud tells everyone about the back story, the game (field script) performs mapjumps between the Kalm and Nibelheim field flevels (flashbacks). Another problem is (I think you guys may have already pointed it out) some of the coordinates are "relative coordinates", like 33=Yuffie encounter, 45=Leaving Highwind, etc.