Author Topic: Where is all the data for the PS1 version of FF7 located?  (Read 2039 times)

MysticLord

  • *
  • Posts: 79
    • View Profile
I want to make some things like this but for FF7 when I'm done with the Ogre series.

I'm also curious if anyone has made any hacks that de-hardcode things, or expands tables, which can integrated with them, as well as the locations of names for digits (items, skills, materia) if it's uncompressed and can be easily integrated into the vlookup functions that generate previews.

I need explanations of each byte/bit too, or someone I can pester about the subject.

I don't care about animations or models, just data for things like skills, items, materia, enemies, encounter tables, and so on.
« Last Edit: 2021-05-20 16:51:37 by MysticLord »

nfitc1

  • *
  • Posts: 2981
  • I just don't know what went wrong.
    • View Profile
    • WM/PrC Blog
Re: Where is all the data for the PS1 version of FF7 located?
« Reply #1 on: 2021-05-20 16:59:58 »
I've got data for battle related stuff (except for encounter tables), but it's not super well organized. Every byte and most bits have documented functions too. Just tell me what you need and I'll work with you.

What de-hardcoding hack are you thinking about? It seems possible to un-hardcode some things with the right memory hooks. To my knowledge there are no such hacks for battle-related mechanics.

MysticLord

  • *
  • Posts: 79
    • View Profile
Re: Where is all the data for the PS1 version of FF7 located?
« Reply #2 on: 2021-05-20 17:35:08 »
Quote
I've got data for battle related stuff (except for encounter tables), but it's not super well organized. Every byte and most bits have documented functions too. Just tell me what you need and I'll work with you.

What de-hardcoding hack are you thinking about? It seems possible to un-hardcode some things with the right memory hooks. To my knowledge there are no such hacks for battle-related mechanics.
I want to make spreadsheet editors for everything, so uhhh, give me everything is the best way to start I guess... but we can start with whatever you want to start with. I really don't know much about hacking FF7.

Though it seems you're talking about battle mechanics code instead of data. That's not quite as easy to deal with using spreadsheets, but it can still be very useful. Either way it can wait until later, as it's not that useful without data unless you're talking about a catch-all spreadsheet for battle mechanic hacks. Still, that sort of thing is better used as a commented and labeled disassembly.

Give me the data related to battle stuff - general 1 sentence description of what each table does, what file it's in, starting address, ending address, number of entries, entry size if fixed and entry delimiting scheme if it's not. That should be enough to set up a spreadsheet that can be used for editing. If it requires decompressing or has a subfile scheme, tell me where to find that tool and parameters for using it, as well as any byte massaging (delete bytes, add bytes, etc) I need to do to use it.

Then a description of what you think each byte per entry does, and as much of a breakdown on valid inputs for each byte as you've got. If it's too complicated, just post your unedited notes and I'll fill in what I can then hand it over to you for editing my notes, and of course you can use it to test things.

If the table is also found intact in RAM - or the entries are - in a location that is always the same place (like 0x13c00, always), then post that, because it will enable hot-patching of save states for faster editing. If your emulator has a special header, post that too.
« Last Edit: 2021-05-20 17:45:13 by MysticLord »

nfitc1

  • *
  • Posts: 2981
  • I just don't know what went wrong.
    • View Profile
    • WM/PrC Blog
Re: Where is all the data for the PS1 version of FF7 located?
« Reply #3 on: 2021-05-20 19:36:31 »
Let's start with something simple like items. What format do you want the data in? Can you give me an example?

I think I see what you're getting at. I haven't done much PS1 emulator hacking, but since the RAM is always the same it shouldn't be difficult to find the data. I know the entire KERNEL.BIN (where all the battle-related object data is) is always present in memory during battle so it should be in a static place each time. It can't be too hard to find it.
« Last Edit: 2021-05-20 19:38:58 by NFITC1 »

MysticLord

  • *
  • Posts: 79
    • View Profile
Re: Where is all the data for the PS1 version of FF7 located?
« Reply #4 on: 2021-05-20 20:19:44 »
Quote
Let's start with something simple like items. What format do you want the data in? Can you give me an example?
1. File it's found in.
2. Starting address of data.
3. Ending address of data.
4. Number of entries of data if it's 1 entry per item, otherwise describe the scheme for delimiting it.
5. Number of bytes per entry if it's a fixed number per entry, otherwise describe the delimiting scheme for it.
6. Explanations of each byte.

Example of number 6:
Code: [Select]
0x00: Item ID

0x01: Item Weight

0x02: Item Power
    : Context depends on if item is weapon, armor, or shield.
    : All skills that are present on this item use this byte to modify their formulas.

0x03: Strength modifier
    : Signed byte.

0x04: Dexterity modifier
    : Signed byte.

...

0x0C: Additional effects
    : This byte determines the context of bytes 0x0D, 0x0E, and 0x0F.
    : Values
        0x00: null
        0x01: weapon attack - 0x0D = element, 0x0E = status effect, 0x0F = critical rate
        0x02: weapon healing - 0x0D = element, 0x0E = status effect, 0x0F = critical rate
        0x03: Use skill 100% of time - 0x0D = upper 8 bits of skill ID, 0x0E = lower 8 bits of skill ID, 0x0F = bit 0x80 sets detroy on use, other bits unknown

You don't have to type "0x" in front of all hex values, you can just say that all numbers are hex unless prefixed by a "d", or use the $ prefix for hex or what have you. As long as it's consistent, unambiguous, and you tell me your scheme it's all good.

I'm going to reread whatever you post a dozen times as I transcribe it into a spreadsheet and double/triple check it after that, so the only possible issue is something not being answered, inconsistencies, or mistakes. If the first two are present, I will definitely notice and ask for clarification. I might notice a mistake if it doesn't jibe with the data.

If it seems that something needs to be organized in it's own way (if - god help us - several bytes have values whose context is completely unique depending on several other values), then put that stuff with the byte that determines the context and leave a note for the stuff that very rarely changes context like "... unless byte 0x0c says otherwise".

So if byte 0x0c can change the meaning of bytes 0x00, 0x01, and 0x02 in a handful of circumstances, then add "see byte 0x0c for context dependent information" to bytes 0x00, 0x01, and 00x2; then identify in byte 0x0c's entry the circumstances where it does this.

Think of it as trying to reduce the duplications of explanations so you edit as few places as possible when you find an error or update something.

Quote
I think I see what you're getting at. I haven't done much PS1 emulator hacking, but since the RAM is always the same it shouldn't be difficult to find the data. I know the entire KERNEL.BIN (where all the battle-related object data is) is always present in memory during battle so it should be in a static place each time. It can't be too hard to find it.
IIRC pSX emulator has a header 0x2b0 bytes long for it's save states, and all others have footers (so the address doesn't change). Addresses shouldn't be any different if it's a footer.



RE save states: If the data is exactly the same format for the PC versions, post the same info for them too... though you should only need to specify things that differ, like file and starting address.

If the data is a subfile in another file, is only ever accessed by doing math on the header of the file, and you can reasonably expect the change the sizes of the subfiles previous to it, then include an explanation of the header. I understand what little endian means, and have basically a sophomore++ level of CS education.
« Last Edit: 2021-05-20 20:21:40 by MysticLord »

nfitc1

  • *
  • Posts: 2981
  • I just don't know what went wrong.
    • View Profile
    • WM/PrC Blog
Re: Where is all the data for the PS1 version of FF7 located?
« Reply #5 on: 2021-05-20 20:38:18 »
Item format is here: http://wiki.ffrtt.ru/index.php?title=FF7/Item_data
In fact, the first nine sections of the KERNEL.BIN entries should interest you: http://wiki.ffrtt.ru/index.php?title=FF7/Kernel/Kernel.bin
Most of that is my work so I'm the one to ask questions of on any of those sections.

The KERNEL.BIN gets unpacked into the same location in RAM each time the game is loaded. I know where the pointer to the PC values is (nine Dword values at 7BA070 point to the locations in memory where each of the sections end up), but it might not translate the same for PS1. PS1 might unload the KERNEL files into the same addresses each time.

MysticLord

  • *
  • Posts: 79
    • View Profile
Re: Where is all the data for the PS1 version of FF7 located?
« Reply #6 on: 2021-05-20 23:53:28 »
Quote
Item format is here: http://wiki.ffrtt.ru/index.php?title=FF7/Item_data
Has anyone tried zeroing out the first eight dummy bytes to see if anything happens?

Where can I find a listing of names for each item in order?

I assume it's 256 items, with a dummy item for empty slot.

I assume no one has a listing of item names, and I'll have to rip them myself to prevent mistakes.
1. Are they index-addressed fixed-length, or pointer-addressed null-terminated?
2. Are they ASCII or some other weird format that lacks the standard ASCII subset?
3. If it's pointer addressed, is there any weird stuff for duplicated names or entries without names?


Found item and materia digits on GameFAQs, in a Gameshark FAQ.

The Damage Calculation page links to a detailed Damage Formula page, but there's nothing there.

I assume that the power in this page is Item byte 0x0F ?

Quote
In fact, the first nine sections of the KERNEL.BIN entries should interest you: http://wiki.ffrtt.ru/index.php?title=FF7/Kernel/Kernel.bin
Most of that is my work so I'm the one to ask questions of on any of those sections.
Are there programs you recommend to extract the kernel subfiles and ungzip them?

Is there a GUI program that does one or both jobs?

If not, is there anything I can call in a python script to ungzip them and then gzip them later (I prefer to newb-proof things as much as I can)?

Has anyone written something I can reuse (open source because lol it's a python script I ain't compiling shit) to deal with gzips if the best scenario for users is making new scripts to deal with the kernel?


Cebix has tools to do this stuff, though they're python 2.7 and have some other dependencies.

https://github.com/cebix/ff7tools
Quote
The tools are entirely written in Python and have the following
dependencies:

 * Python >= 2.7.6
   The tools are not compatible with Python 3.

 * Pillow >= 5.1.1 (https://pypi.python.org/pypi/Pillow/)
   Used by the 'tim2png' and 'subending' tools.

 * NumPy >= 1.8.0 (http://www.numpy.org/)
   Used by the 'subending' tool.

 * PSXImager >= 1.0 (https://github.com/cebix/psximager)
   Used by the 'fixup' tool.

Most of the tools use the included 'ff7' package which must be in the
PYTHONPATH or reside in the same directory as the tools themselves.

Shouldn't be too hard to port the code I need to Python 3, and grab whatever license he uses.

Quote
The KERNEL.BIN gets unpacked into the same location in RAM each time the game is loaded. I know where the pointer to the PC values is (nine Dword values at 7BA070 point to the locations in memory where each of the sections end up), but it might not translate the same for PS1. PS1 might unload the KERNEL files into the same addresses each time.
If I were you, I'd just crack open a save state and search for the first few bytes of item data. If that doesn't work, search for a set of adjacent and very distinct bytes, first using XVI32's count function, then the search function when you get a small enough data set.
« Last Edit: 2021-05-22 03:57:17 by MysticLord »

nfitc1

  • *
  • Posts: 2981
  • I just don't know what went wrong.
    • View Profile
    • WM/PrC Blog
Re: Where is all the data for the PS1 version of FF7 located?
« Reply #7 on: 2021-05-24 16:26:08 »
Has anyone tried zeroing out the first eight dummy bytes to see if anything happens?

Yes I have. They are actually spacer/padding values because the Item data structure is derived from Action data structure. In order to make them loadable by the same method, the first eight bytes of Action data is just dummied out. I have verified this by reversing the PC executable extensively.

Where can I find a listing of names for each item in order?

I assume it's 256 items, with a dummy item for empty slot.

I assume no one has a listing of item names, and I'll have to rip them myself to prevent mistakes.
1. Are they index-addressed fixed-length, or pointer-addressed null-terminated?
2. Are they ASCII or some other weird format that lacks the standard ASCII subset?
3. If it's pointer addressed, is there any weird stuff for duplicated names or entries without names?


Found item and materia digits on GameFAQs, in a Gameshark FAQ.

I would have said use Wall Market, but lots of people are having problems with it because Power Packs 3 doesn't work with the current edition and it's hard to find now. :( I've removed that dependency for version 1.5, but that version is still too far from release-ready.

The Damage Calculation page links to a detailed Damage Formula page, but there's nothing there.

The actual link is here. For some reason this wiki doesn't like altering links even if it is to the same wiki.

I assume that the power in this page is Item byte 0x0F ?
Yes
Are there programs you recommend to extract the kernel subfiles and ungzip them?

Is there a GUI program that does one or both jobs?

If not, is there anything I can call in a python script to ungzip them and then gzip them later (I prefer to newb-proof things as much as I can)?

Has anyone written something I can reuse (open source because lol it's a python script I ain't compiling shit) to deal with gzips if the best scenario for users is making new scripts to deal with the kernel?


Cebix has tools to do this stuff, though they're python 2.7 and have some other dependencies.

https://github.com/cebix/ff7tools
Shouldn't be too hard to port the code I need to Python 3, and grab whatever license he uses.
If I were you, I'd just crack open a save state and search for the first few bytes of item data. If that doesn't work, search for a set of adjacent and very distinct bytes, first using XVI32's count function, then the search function when you get a small enough data set.

Wall Market, but see above. :(

MysticLord

  • *
  • Posts: 79
    • View Profile
Re: Where is all the data for the PS1 version of FF7 located?
« Reply #8 on: 2021-05-24 17:43:07 »
Quote
Yes I have. They are actually spacer/padding values because the Item data structure is derived from Action data structure. In order to make them loadable by the same method, the first eight bytes of Action data is just dummied out. I have verified this by reversing the PC executable extensively.
I noticed that in the other kernel.bin structures. Does it actually save enough space to justify the wasted bytes? Or is it an issue of speed?

Difference between the link that doesn't work and the one that does:
https://wiki.ffrtt.ru/index.php/DamageFormula
https://wiki.ffrtt.ru/index.php/FF7/DamageFormula

I have everything I need to do this, but I'm working on Ogre Battle MOTBQ and probably Tactics Ogre TKOL first. They don't have editors.

nfitc1

  • *
  • Posts: 2981
  • I just don't know what went wrong.
    • View Profile
    • WM/PrC Blog
Re: Where is all the data for the PS1 version of FF7 located?
« Reply #9 on: 2021-05-24 18:21:46 »
I noticed that in the other kernel.bin structures. Does it actually save enough space to justify the wasted bytes? Or is it an issue of speed?

I believe it's an issue of design. This is designed to work on a 32-bit environment so it likes to have sizes that can "cleanly" fit into 32-bit chunks. It can do other sizes, of course, but accessing is just easier if everything is "rounded off" to the nearest 32-bit measurement. It's a preference of the compiler, I'd imagine. By default it adds lots of padding to try to prevent memory leaks and the like.