Author Topic: [PSX/PC] General editor - Hades Workshop (0.40b)  (Read 265525 times)

Tirlititi

  • Freak
  • *
  • Posts: 518
  • Karma: 68
    • View Profile
Re: [FF9] General editor - Hades Workshop (0.35c)
« Reply #625 on: 2017-02-02 16:26:58 »
Hi Aidolu ! Welcome back :D
Unfortunatly, while it is surely possible to do it, I don't know how... The MIPS code could certainly access to the AP and exp of the characters (in battle - it wouldn't be usable in menu anyway -), but it would require the expertise of a good person who does RAM analysis to find which sequences to use to retrieve these values.

There's a list of known fields in the MIPS help that you can access in the spell effect codes. There are HP, stats, trance gauge, level and activate supporting abilities, among others, but there's no exp nor AP (and I think it's not in the same structure so you wouldn't find them even by trying every possible figures).

In order to increase the level, that's this sequence :
lbu:     $1 = $19[122] & 0xFF
addiu: $1 = $1 + 1
sb:      $19[122] = $1 & 0xFF

However, I would bet that it would have only a twisted effect to directly change the level value :
1) Max HP/MP will most likely increase, but not the other stats (strength...),
2) The exp doesn't change so it will need more time to level up once again after that,
3) It may even be possible that the level gets reverted back to normal at the end of the battle. In this case, the only noticeable difference will be the effectiveness of spells like Lvl 5 death or so...

Sorry, I kind of deserted the PSX version for the Steam version now... Well, when something can be done for both, I do it (copy/paste the spells of enemies, for instance), but MIPS is just too related to the PSX version to get improved.

aidolu12

  • Newbie
  • *
  • Posts: 4
  • Karma: 0
    • View Profile
Re: [FF9] General editor - Hades Workshop (0.35c)
« Reply #626 on: 2017-02-03 07:48:14 »
Hi Aidolu ! Welcome back :D
Unfortunatly, while it is surely possible to do it, I don't know how... The MIPS code could certainly access to the AP and exp of the characters (in battle - it wouldn't be usable in menu anyway -), but it would require the expertise of a good person who does RAM analysis to find which sequences to use to retrieve these values.

There's a list of known fields in the MIPS help that you can access in the spell effect codes. There are HP, stats, trance gauge, level and activate supporting abilities, among others, but there's no exp nor AP (and I think it's not in the same structure so you wouldn't find them even by trying every possible figures).

In order to increase the level, that's this sequence :
lbu:     $1 = $19[122] & 0xFF
addiu: $1 = $1 + 1
sb:      $19[122] = $1 & 0xFF

However, I would bet that it would have only a twisted effect to directly change the level value :
1) Max HP/MP will most likely increase, but not the other stats (strength...),
2) The exp doesn't change so it will need more time to level up once again after that,
3) It may even be possible that the level gets reverted back to normal at the end of the battle. In this case, the only noticeable difference will be the effectiveness of spells like Lvl 5 death or so...

Sorry, I kind of deserted the PSX version for the Steam version now... Well, when something can be done for both, I do it (copy/paste the spells of enemies, for instance), but MIPS is just too related to the PSX version to get improved.


Oh though MIPS where for both, I was thinking on passing to STEAM version when you just released the tool for it, but I had these problems with the tags for the colors and text filling, then I move from my country to another so until now I did not have time for it. Is it possible to edit the effects for the STEAM version, I will download it again (I bought it but it was installed on my PC from my hometown.)

Further than that, just to thank you again, this tool has improvised to impressive levels, and my mind to, to make better mods.

Tirlititi

  • Freak
  • *
  • Posts: 518
  • Karma: 68
    • View Profile
Re: [FF9] General editor - Hades Workshop (0.36)
« Reply #627 on: 2017-02-10 23:28:44 »
Update to v0.36:
- Added a Background Editor Tool. It allows to convert images into Steam background format files and then import them. More on that below.
- Added a CIL Macro to make the engine accept the converted backgrounds (in particular, for changing the resolution).
- Added support for the backgrounds that contain titles (Ice Cavern entrance, Qu's Marsh entrance, etc...).
- Fixed the CIL Macro "Unlock Ability Learning": it made the few stage battles at the start of the game bug.
- Improved the Script Editor (replaced the default variable names, added a few safety checks when parsing functions, added support for single-lines comments, ...).

So, the big new thing is the background editor. Thanks to it, upscaling and modifying backgrounds is possible. There are still a few flaws with the quality (people noticed seam lines or black pixels in-game). With a bit of work on the images, it can be perfected though.

A big hurray to Fraggoso who started to upscale and fix the images.

If you want to import custom backgrounds, you should first go to "CIL Code -> Macros", apply the "Custom Backgrounds" macro and save the Steam mod: it will make the engine compatible with the custom backgrounds (in particular if you change the resolution). If you don't do it first, do it last :p

So, the background editor gets available once you got to the "Environment -> Fields" panel. It has 3 panels itself :
- Converter -> From a few images (1 image per layer), create a Steam formatted file (.tex) of the background that can replace the default files in the game's assets.
- Mass Converter -> Same as converter, but it can do several fields consequently. Hopefully, the program doesn't run out of memory too soon (backgrounds are very RAM consuming).
- Mass Importer -> Take the .tex files and import them in the game's assets (in the StreamingAssets folder, p0data11.bin to p0data19.bin).

The converter lets you verify that the images are loaded correctly and that they match with the tilesets. There are several parameters:
1) Background Image: Select one of the layers of the background. All the layers must be in the same directory and be named
SomeName0.bmp
SomeName1.bmp
SomeName2.bmp
etc...
For multi-background fields (such as the Prima Vista's Cargo Room, that has 2 cameras and thus 2 images), it must be
SomeName0_0.bmp
SomeName0_1.bmp
...
SomeName1_0.bmp
...
Each of the bmp (or tga or tiff) must contain a layer which correspond to one of the field's tile block.
The Gimp add-on save-layers-tiff-24 is perfect for generating each layers with the right name from the .tiff images exported by HW.
2) Destination Folder: Where to save the Steam formatted file. The name of the .tex file is generated to match the background and should not be modified.
3) Use Game Tiling/Custom Tiling: Unavailable for now. In the future, this will allow to import entirely custom backgrounds, but for now, they need to match the normal ones (same number of layers, each of them placed at the same place in the whole image...).
4) Field: Choose the field corresponding to your background image.
5) Sort Layers by depth: The layers can be sorted either by their ID (the ID is used in the scripts, for instance) or by depth. Hades Workshop export the layers of the backgrounds by depth so you can usually let it checked.
6) Revert Layer Order: When checked, the foreground-most image is numbered "0" while it's the background-most image when unchecked. With the Gimp add-on, the order is reverted.
7) List of Tile Blocks: Select a few of them to verify that the layer match with the tile block. A greyed area is drawned where the layer is expected to be.
8) Resolution: By default, resolution is 32, meaning 32 pixels per tile. If you use upscaled images, modify this (resolution can go up to 123 for now, which is a bit less than 4x the Steam resolution and is already huge files ; a resolution of 64 seems honest for me).
9) Compression Quality: Select the compression algorithm to use. High quality is slower while low quality is faster. In any case, it doesn't change the file size of the output, only the time required to convert it.

Finally, there's also the case of backgrounds containing a title. Since those titles are present for every languages in the Steam version, you need to have layers for them in every languages as well (usually, it's 2 layers per language). The English (US) title layers should be at their usual places, and the other title layers must come after all the others in this order :
English (UK), Japanese, German, French, Italian, Spanish.
Again, the layers as they are exported by HW are in the correct order.

In the Mass Converter panel, you find the same parameters but the image file names are also indexed by the field ID (or are inside a folder indexed by the field ID). With the default settings, "Field0\Background1_2.tif" corresponds to the 3rd layer of the 2nd background of the 1st field (Prima Vista/Cargo Room). You may also use the indexing of the real IDs of the fields (used in the scripts among other places) instead of their order. With that option, Prima Vista/Cargo Room has an ID of 50 and thus should be placed in the folder "Field50" instead.
A window displays how many backgrounds were spotted and converted and whether there was a problem.

In the Mass Importer, you select a FF9_Launcher.exe. You can choose the one that is currently opened in HW, but the program may crash then after leaving the background editor (that's quite harmless if you weren't making a mod besides the backgrounds though).


One last thing, I've been told that this version crashes on start (when opening a game). But it works just fine for me, so can anyone confirm that it crashes or that it works correctly? Thank you.

I'll soon write a post about the current status of FF9 modding. What has been done and what can be done next.

Meru

  • Crazy poster
  • *
  • Posts: 106
  • Karma: 6
    • View Profile
Re: [FF9] General editor - Hades Workshop (0.36)
« Reply #628 on: 2017-02-11 00:36:55 »
Hey, Tirlititi, one of the WiP builds crashed. One that is made sometime after 2017-02-03. This 0.36 release seem to be working.

Anyway, thanks for all your work! I have a lot of stuff to do now. Expect me to bother you with "artifacts" again :D

Edit: I spoke too soon. This version does crashes, but after reading fields from PSX disc1 (US version), not steam version.
« Last Edit: 2017-02-11 01:05:25 by Meru »

Tirlititi

  • Freak
  • *
  • Posts: 518
  • Karma: 68
    • View Profile
Re: [FF9] General editor - Hades Workshop (0.36)
« Reply #629 on: 2017-02-11 12:05:52 »
Updated to fix that. There was indeed a weird behavior in the compiler that I didn't know. Thanks Meru :)

Meru

  • Crazy poster
  • *
  • Posts: 106
  • Karma: 6
    • View Profile
Re: [FF9] General editor - Hades Workshop (0.36)
« Reply #630 on: 2017-02-11 15:05:42 »
Update works past that point, but there seem to be another problem:
Open FFIX PSX -> Environment -> fields -> open synthesyst's "Manage" dialog and close it, switch to weapon shop (or possibly any field) then back to synthesyst -> crash.

Fraggoso

  • Crazy poster
  • *
  • Posts: 211
  • Karma: 10
    • View Profile
Re: [FF9] General editor - Hades Workshop (0.36)
« Reply #631 on: 2017-02-11 17:34:36 »
The newest Version still doesn't work for me... the older one still working. :/

Meru

  • Crazy poster
  • *
  • Posts: 106
  • Karma: 6
    • View Profile
Re: [FF9] General editor - Hades Workshop (0.36)
« Reply #632 on: 2017-02-11 19:42:56 »
Hey, Tirlititi, I seem to found a problem with the most recent HW.
Alexandria/Main street 1
It exports (and expects on import) 2 new tileblocks (page14-15) that look like this

This is the one that has a title. Selecting single language or all makes no difference. In wip build it wasn't the case though.
And judging by in-game results they are being used...???
« Last Edit: 2017-02-11 19:50:20 by Meru »

Tirlititi

  • Freak
  • *
  • Posts: 518
  • Karma: 68
    • View Profile
Re: [FF9] General editor - Hades Workshop (0.36)
« Reply #633 on: 2017-02-11 20:37:52 »
Hum... those are the English (UK) tiles. It seems that I don't handle them correctly.
The thing is, they are not used in-game: the US tiles are used instead of the UK for nearly all the titles (all except South Gate).

I'll try to fix that (and the other problems) but you can bypass it for now by copying the English (US) layers and replace those ill tiles by the US ones. If I understand correctly, what happens now is:
- The title is not present for UK so I export something incorrect (I meant to export a duplicate of the US but I made a mistake),
- When converting, the US title is imported and then the UK title is imported over it, so this fuzzy title is used in place of the US (the others should be correct).
So yeah, replacing the UK title with a duplicate of the US title should be ok. The only field for which it matters is exporting the title correctly.

Meru

  • Crazy poster
  • *
  • Posts: 106
  • Karma: 6
    • View Profile
Re: [FF9] General editor - Hades Workshop (0.36)
« Reply #634 on: 2017-02-11 21:18:18 »
Tirlititi, okay then I have a surprise for you. Here is what happens if I replace those layers as you said  :mrgreen:


Tirlititi

  • Freak
  • *
  • Posts: 518
  • Karma: 68
    • View Profile
Re: [FF9] General editor - Hades Workshop (0.36)
« Reply #635 on: 2017-02-11 22:29:34 »
Pft.
That no good surprise.

Remove the problematic images then. Just delete (or rename) the "Backgrounds10.tif" and "Background11.tif" and do it again. In the Mass Converter, there are warnings if a few images are missing, but I forgot to throw a warning in the Converter so it will tell you it did just fine without them (and it will be right, I tested, period, no more bad news about that  :-X ).

Fraggoso

  • Crazy poster
  • *
  • Posts: 211
  • Karma: 10
    • View Profile
Re: [FF9] General editor - Hades Workshop (0.36)
« Reply #636 on: 2017-02-12 15:38:50 »
The program works fine on my working laptop (from the company with Win 7). On my normal PC (win 10) it refuses to open neither the psx binarys nor the Launcher from the steamversion...
Is there any way I can give you and error log of some sorts?

Meru

  • Crazy poster
  • *
  • Posts: 106
  • Karma: 6
    • View Profile
Re: [FF9] General editor - Hades Workshop (0.36)
« Reply #637 on: 2017-02-12 17:05:45 »
Tirlititi
About these black dots: https://i.imgur.com/cDpUCJT.png
I'm not sure this is something I can work out on my side...
This is how HW's tileset looks like: https://i.imgur.com/QEY1WNv.png

Edit: also, I just noticed that this particular (green) layer has a clear signs of antialiasing. This could be the reason why these squares are slipping-in -> by being averaging with nearest pixels.
In that case the question is, why this layer gets scaling and/or antialiasing while other (tickets tent) - don't? Can we enforce that for all layers to (possibly) get a smoother picture?
My original layer does not have any half-transparent/antialiased pixels. Whatever opaque color or full transparency.

On a side note, I wasn't able to process this particular image fully with the errors "can't allocate memory" for a couple of last (biggest) layers. HW wasnt using more than 300megs of ram though.
The problem seem to be went away after I patched executable with largeaddressaware flag. Maybe just a coincidence... dunno.

Edit2: also any idea what would happen if we'll patch texture to be processed with trilinear mode instead of bilinear? (if anything). Also maybe patching it for point filtering mode may remove this processing of the green layer?
« Last Edit: 2017-02-12 17:23:20 by Meru »

Tirlititi

  • Freak
  • *
  • Posts: 518
  • Karma: 68
    • View Profile
Re: [FF9] General editor - Hades Workshop (0.36)
« Reply #638 on: 2017-02-12 17:21:48 »
Maybe it has something to do with premultiplied alpha...
The way FF9 handles the alpha is pretty bizarre ; they kind of still use the PSX way to do things in that regard. The best is to use image with only 2 alpha values: fully transparent and fully opaque (or half-opaque for the semi-transparent layers). I'm not saying it will fix the problem, but it worths the try. It's also a big argument for reworking entirely the engine about it, and get rid of the tiled format (this way, any alpha value could be used for any layer).

EDIT: About the "can't allocate memory" error, I also got it once and, by trying again right after, without even closing the program, it worked. I don't really know why it sometimes throws that error (maybe when requesting too much RAM space at once, it can be refused?).
« Last Edit: 2017-02-12 17:24:27 by Tirlititi »

Meru

  • Crazy poster
  • *
  • Posts: 106
  • Karma: 6
    • View Profile
Re: [FF9] General editor - Hades Workshop (0.36)
« Reply #639 on: 2017-02-12 17:24:16 »
Thats what I do. I only have fully transparent and fully opaque. Also pls see my edits above.

Tirlititi

  • Freak
  • *
  • Posts: 518
  • Karma: 68
    • View Profile
Re: [FF9] General editor - Hades Workshop (0.36)
« Reply #640 on: 2017-02-12 17:44:36 »
I don't know why this layer is green. I thought you colored it on purpose... It's not green in the Steam version, right?

I don't know about trilinear filtering, you're the one doing tests about it. Here are the different modifications done on the images:
1) As I explained to you, the tiles in the atlas are 36x36 instead of 32x32 and the 33x33 center region will be used by the 3D engine. So when importing the backgrounds, I need to somehow expand the 32x32 tiles into 36x36 tiles ; the center region is kept and the borders are averaged with the color of the tile next to it. In the mean time, I also gives color to the transparent pixels because they will have an impact on the DXT5 compression. Currently, I average the transparent pixels with the opaque pixels belonging to the boundary of the object near to it, but I tried a lot of other methods, and the default atlas doesn't do it like that.
2) The atlas gets DXT5-compressed. All the pixels are packed into 4x4 blocks with a few color variations in them. I use an external library for that (Squish) that is quite efficient. The compression quality parameter is used at this point to compute more fitting color variations in those 4x4 blocks.
3) The atlas is rendered by the game's 3D engine: the 33x33 square at the center of the tiles are used as textures for 3D quads (couple of triangles in fact) and the whole thing forms a kind of flat 3D model.
4) The rendering is scaled depending on the resolution. I have no idea of which algorithm is used there and I can't help you if you want to change it at this step.

Meru

  • Crazy poster
  • *
  • Posts: 106
  • Karma: 6
    • View Profile
Re: [FF9] General editor - Hades Workshop (0.36)
« Reply #641 on: 2017-02-12 17:55:49 »
I did colored it on purpose. Here is exact layer i used https://dl.dropboxusercontent.com/u/110558786/Report/Square28.7z

The rendering is scaled depending on the resolution:
what i am saying is, game has different approaches for different layers. Or at least it looks like it. "Tickets" tent and most of the textures looks like scaled with "point" method. They show hard edges and no artifacts. While some layers, like this green one seem to be processed differently. You can see it is "blended" into the background. These half-transparent green edges is not my doing nor it is related to DXT because it is not present in decompressed HW's output as I've shown on 2nd image.

So my theory is, these black dots are slipping in because of that "other processing" game does for this layer.
« Last Edit: 2017-02-12 18:17:39 by Meru »

Meru

  • Crazy poster
  • *
  • Posts: 106
  • Karma: 6
    • View Profile
Re: [FF9] General editor - Hades Workshop (0.36)
« Reply #642 on: 2017-02-12 19:17:55 »
Here is what i mean:

I create all layers like so - hard edges, no antialiasing:


Some layers look in game just like that even after adjusting to my resolution.

Some layers, on the other hand, start to look like this all on their own:


Now I have not checked all the layers out there, but incidentally layers that look like so also have a black dot issue.

Also as I described before, issues appear even on exporting and re-importing the same thing without modifications (grid issue being one of that)
There probably is a weird behavior in-engine, or even a bug, but original textures don't make it apparent because their tile paddings are precisely a copy of the tile's edge-pixel. (Or it could be Silicon's workaround for Unity's crappy tiling)

It is up to you of course what to do about that, but I think it is easier to create textures in a compatible way instead of figuring out what engine does there.
« Last Edit: 2017-02-12 19:32:18 by Meru »

Fraggoso

  • Crazy poster
  • *
  • Posts: 211
  • Karma: 10
    • View Profile
Re: [FF9] General editor - Hades Workshop (0.36)
« Reply #643 on: 2017-02-14 09:48:21 »
Hi Meru, are you creating all layers from scratch?
The engine is really nitpicking on the Layers. Slight variations done to the background layers opposed to all other layers can yield very bad results like gray/black seams and/or the dot's you're mentioning.

Kaldarasha

  • No life
  • *
  • Posts: 2254
  • Karma: 134
  • Prince of Model Editing
    • View Profile
Re: [FF9] General editor - Hades Workshop (0.36)
« Reply #644 on: 2017-02-14 11:26:19 »
A reason why those filtering does happen is probably that a few parts of the backgrounds are in fact 3D objects and the games filtering (bilinear or trilinear) does affect these objects.

Fraggoso

  • Crazy poster
  • *
  • Posts: 211
  • Karma: 10
    • View Profile
Re: [FF9] General editor - Hades Workshop (0.36)
« Reply #645 on: 2017-02-14 12:07:46 »
A reason why those filtering does happen is probably that a few parts of the backgrounds are in fact 3D objects and the games filtering (bilinear or trilinear) does affect these objects.

Nope, FFIX doesn't use any kind of 3D realtime models. Some 2D alpha effects like smoke is used but they not part of the static backgrounds and thus can't be filtered.

Kaldarasha

  • No life
  • *
  • Posts: 2254
  • Karma: 134
  • Prince of Model Editing
    • View Profile
Re: [FF9] General editor - Hades Workshop (0.36)
« Reply #646 on: 2017-02-14 16:16:26 »
Nope, FFIX doesn't use any kind of 3D realtime models. Some 2D alpha effects like smoke is used but they not part of the static backgrounds and thus can't be filtered.

There are some realtime models, but that's not what I meant. When I'm correct the problem does only occur on the layers, which can hide an object/character. I remember, that someone wanted to make a similar engine as FF7 by using Blender and encountered a problem with these layers, because he couldn't easily copy the way as Square did it for the game. So he placed the layers in the 'room' similar as an object. And that is what I assume the port team did with the BG's of FF9. That would explain why the texture filter is working on these layers.

Meru

  • Crazy poster
  • *
  • Posts: 106
  • Karma: 6
    • View Profile
Re: [FF9] General editor - Hades Workshop (0.36)
« Reply #647 on: 2017-02-14 16:32:37 »
Hi Meru, are you creating all layers from scratch?
The engine is really nitpicking on the Layers. Slight variations done to the background layers opposed to all other layers can yield very bad results like gray/black seams and/or the dot's you're mentioning.
Not sure what do you mean by "from scratch". You mean layer edges? Sometimes yes, sometimes no, but this doesn't matter. BTW the problem can be seen on some of your older screenshots of synthesist too.
Edit: of course for the final product all the layers must be redone. Leaving them as is and simply swapping backgrounds just makes things different kind of ugly from what it used to be with default backgrounds.

The dots are there because this is how HW creates textures. The problem is that these dots are outside of the range that must be shown on screen. I think that instead of trying to fix the engine, it is better mimic the way Silicon created textures. I dunno what the engine does there, nor I want to know, but these dots (and probably grid as well) look like a product of averaging with nearest pixels, like bilinear or better resampling or fxaa-style antialiasing (unlikely in this case since I think it works in screen-space only). When it happens that the nearest pixel is a black dot - sometimes we can see it on screen, when it happens to be imperfectly mirrored edge pixel - we see grid. The grid is very hard to spot if you deal with 64x64 tiles but quite easy with default ones.
« Last Edit: 2017-02-14 17:00:12 by Meru »

Fraggoso

  • Crazy poster
  • *
  • Posts: 211
  • Karma: 10
    • View Profile
Re: [FF9] General editor - Hades Workshop (0.36)
« Reply #648 on: 2017-02-14 18:09:32 »
Not sure what do you mean by "from scratch". You mean layer edges? Sometimes yes, sometimes no, but this doesn't matter. BTW the problem can be seen on some of your older screenshots of synthesist too.
Edit: of course for the final product all the layers must be redone. Leaving them as is and simply swapping backgrounds just makes things different kind of ugly from what it used to be with default backgrounds.

The dots are there because this is how HW creates textures. The problem is that these dots are outside of the range that must be shown on screen. I think that instead of trying to fix the engine, it is better mimic the way Silicon created textures. I dunno what the engine does there, nor I want to know, but these dots (and probably grid as well) look like a product of averaging with nearest pixels, like bilinear or better resampling or fxaa-style antialiasing (unlikely in this case since I think it works in screen-space only). When it happens that the nearest pixel is a black dot - sometimes we can see it on screen, when it happens to be imperfectly mirrored edge pixel - we see grid. The grid is very hard to spot if you deal with 64x64 tiles but quite easy with default ones.

I just wanted to know if you're doing the layer edges by hand instead of using a template (either PSX or Steam).
And yes, I also have these dots. It doesn't distracted me too much but they're there. :/

Meru

  • Crazy poster
  • *
  • Posts: 106
  • Karma: 6
    • View Profile
Re: [FF9] General editor - Hades Workshop (0.36)
« Reply #649 on: 2017-02-14 18:19:52 »
Fraggoso
Heh, I was too focused on the problem we were discussing ^^
The templates you are talking about I am using of course, but only for now. They will have to be redone by hand at some point. Maybe with vector masks (https://i.imgur.com/zeeepfQ.png) maybe something else, not sure yet.