Author Topic: [FF9PC] Background upscaling mod  (Read 14395 times)

Ze_PilOt

  • Fast newbie
  • *
  • Posts: 42
  • Karma: 8
    • View Profile
[FF9PC] Background upscaling mod
« on: 2019-01-08 20:55:40 »
Hi,

While in vacation, in December, I gave a try with SR-GAN to upscale FF9 background, following a discussion with some friends that the original background where lost, and that DNN could help finding back these lost assets.
So I took all the exported background and gave it a go..

https://imgur.com/gallery/K1R3wJZ

This is with one of the trained generator (ESRGRAN4x).
The main problem here is that it's trained with a standard photoset (usually BSDS100) that is not suitable for artwork.
I think some people gave it a go mixing it with a set trained with the Manga109 dataset, but again not suitable : Photo & drawing are very different from artworks.

A photo from the set :

A drawing from manga109 :


What we are trying to achieve :


Think of the process like this : The DNN is trained to find pattern is small resolution images, and use patterns from the high resolution images it was trained for to fill the gaps.

BSDS or Manga109 dataset will generate too soft, not very detailed images due to the very simple nature of the images (manga109) or over-detailed and artifacts because of wrongly interpreted patterns (BSDS-100).
Also, the final image we want should have a "hand-painted feel" that none of these dataset offer.
This is often the mistake when I see people giving a go with DNN : You have to train your network with the kind of data you want to manipulate. There is sadly no silver bullet.

I've spend a couple of days and trained a GAN with my own dataset : HD artworks from
https://www.neogaf.com/threads/the-lost-art-of-final-fantasy-ix-mama-robotnik-research-thread.551612/
as well as artwork from different games that kind of look like FF9 style, ie :
http://conceptartworld.com/news/dragon-age-inquisition-environment-concept-art/
https://alishermirzoev.artstation.com/
https://www.pinterest.fr/andreamassa/tomb-raider-art/
https://www.videogamesartwork.com/games/witcher-3-wild-hunt
https://kaidash.artstation.com/projects/6kP9W

A selection of about 110 HD artworks, resulting in a 18.000 images dataset once tiled & randomized (for a DNN, a flipped/rotated image is a brand new image).




After 3 days of training, I've managed to get a better result out of the box.


(left is the original Manga109 / ESGRAN4x mix), right is mine, after a day of training. It's even better now that it's fully trained)

The next step is to automate the process of upscaling the whole game.
I've started from the PSX images, as the PC version is very poor quality wise, due to the de-dithering filtering applied :


(left is PC, right is PSX).

All was extracted using Hades Workshop.

I've of course had to apply my own filtering for decreasing the dithering without loosing too much details, through a photoshop batch script.

Then the masking problem.

https://twitter.com/Ze_PilOt/status/1080914390306643968

The masking was quickly solve, but I was still unhappy with the border artifacts from the different layers.
So I rewrite part of Hades Workshop to export me more informations (animations, light/effect layers, ...) so I could recombine layers to have a full image for the GAN generator, and to de-composite it back for Re-inserting it for Hades.

I'm happy to say that all these problem are now solved, and I'm in the process of converting the first CD (222 backgrounds) to do a test mod.


All the work -trained generator, automation scripts (normally no human/manual intervention), Hades modifications- (except the dataset for training due to copyright issues) will be available as open source ASAP.


« Last Edit: 2019-01-08 21:02:39 by Ze_PilOt »

Fraggoso

  • Insane poster
  • *
  • Posts: 279
  • Karma: 13
    • View Profile
Re: [FF9PC] Background upscaling mod
« Reply #1 on: 2019-01-09 07:52:00 »
Hi, I'm also working on an Update to my original Waifu HD Backgrounds for FFIX.
What do you mean with rewritting HW to export more information? If you're extracting from PSX Source you just have to untick merge layers and you receive every layer (transparency, tiles and animations).

I've extraced all assets from PSX as well as Steam (as you need the light sources from Steam sadly) and I'm about 66% finished with everything.
I'll also release all the PSX and Steam assets as well as every Folder with their respective Field# so no one have to  worry about dubletts and Field#.

The seam issue is because the Steam Version uses one more pixel on each layer so they overlapp (the psx assets are seamless).
I resolved that with resizing the psx layers with 400% size & lanczos algorithm and get rid of the transperancy around the edges. Afterwards I just cut out the Layers on top of the new ESRGAN Image and voila -> No seam problems.

How did you solve it? Maybe your art/mask is better than mine. :)

Ze_PilOt

  • Fast newbie
  • *
  • Posts: 42
  • Karma: 8
    • View Profile
Re: [FF9PC] Background upscaling mod
« Reply #2 on: 2019-01-09 08:22:32 »
Hi, I'm also working on an Update to my original Waifu HD Backgrounds for FFIX.

Nice ! I've come across your mod searching if it was possible to re-inject the background into the game, seing your mod was a deciding factor :)

What do you mean with rewritting HW to export more information? If you're extracting from PSX Source you just have to untick merge layers and you receive every layer (transparency, tiles and animations).

I need more informations than that in order to generate images that will be handled properly by the GAN. It was trained with full images, and doesn't cope well with uncomplete layers & transparencies (ie. a door on a black background, it will try to find pattern between the black & the door that are not suitable for the final result).

In order to do that, I don't only need the animation layers, I also need to know that a layer (in the tiff file) IS an animation layer (vs. static background), or if the layer has parralax, and if so, what is the depth of that layer.
For lighting effects and other stuff, I also need to know if a layer a different blending mode.

I've extraced all assets from PSX as well as Steam (as you need the light sources from Steam sadly) and I'm about 66% finished with everything.

I would not like to duplicate works here.
I don't know if you do that manually or not, but I wrote scripts that does that automatically (by reading the script field id from both versions in order to match them properly), and that the correct layer from both versions (as your said : blending layers from steam, real background from the PSX version).

I will try to release these (python) scripts ASAP (probably thursday or this week-end). I can't pack the final result due to copyright issues, so I need some time to be sure that they are easy enough to handle.

I'll also release all the PSX and Steam assets as well as every Folder with their respective Field# so no one have to  worry about dubletts and Field#.

Yes, that is a good practice.

The seam issue is because the Steam Version uses one more pixel on each layer so they overlapp (the psx assets are seamless).

It's a classic layering issue. The main problem is that both version doesn't handle semi-transparent images, so the edges of each layer are sharp, instead of having some semi-transparent pixels in order to blend over.
Another issue is that, normally, the background layer shouldn't stop right where a layer on top start. That works on the PSX version because of the very sharp edges & low resolution, but I think they had some issues during the upscaling and used an extra pixel to avoid cover issues.

And on top of all that, their upscaling process had some black edge issues, related to bi-linear interpolation on single layer (that was I'm trying to avoid by recombining all the layers in a meaningful way first).

I resolved that with resizing the psx layers with 400% size & lanczos algorithm and get rid of the transperancy around the edges. Afterwards I just cut out the Layers on top of the new ESRGAN Image and voila -> No seam problems.

Actually, the transparency around the edge is not handle at all by the PC version. What is happening is that all pixels with a alpha < 127 result in a black alpha, and all pixels with an alpha <= 127 result in a white alpha.

I'm still unsure if I will use the PSX layers with a neighbooring algorythm, or something similar to what you did, but from early testing, both solutions seems workable.

How did you solve it? Maybe your art/mask is better than mine. :)

I'm not sure, but I think that the layer recombination is key to have a proper result.

ie .this kind of artefacts on the edge of the wood :



It's not due to the masking.
I've the same issue in photoshop is a perfect mask that I couldn't in-engine, but that I use for ground truth when testing.
It's due to the fact that the wood is interpolated near black pixels.
I've tried to extend the pixel borders to avoid that : I can reduce or increase the artifact effect depending of how it's done, but I'm only manage to get a perfect result when it's interpolated with a recombined image.

satsuki

  • Insane poster
  • *
  • Posts: 317
  • Karma: 51
    • View Profile
    • My softs (french only)
Re: [FF9PC] Background upscaling mod
« Reply #3 on: 2019-01-09 09:13:06 »
Could you share your trained model ? i'd like to test it with FFVII.
Thanks

Ze_PilOt

  • Fast newbie
  • *
  • Posts: 42
  • Karma: 8
    • View Profile
Re: [FF9PC] Background upscaling mod
« Reply #4 on: 2019-01-09 11:30:25 »
Could you share your trained model ? i'd like to test it with FFVII.
Thanks

Yes, I will share it with the automation scripts.

Ze_PilOt

  • Fast newbie
  • *
  • Posts: 42
  • Karma: 8
    • View Profile
Re: [FF9PC] Background upscaling mod
« Reply #5 on: 2019-01-09 20:14:43 »
Exemple of layer recombination, I'm able to output animated GIF :)

When this project is done, I will enhance the scripts to automate GIF creation of all the background, it's so nice !



Fraggoso

  • Insane poster
  • *
  • Posts: 279
  • Karma: 13
    • View Profile
Re: [FF9PC] Background upscaling mod
« Reply #6 on: 2019-01-10 11:37:15 »
Nice ! I've come across your mod searching if it was possible to re-inject the background into the game, seing your mod was a deciding factor :)

Great to hear! :D

I need more informations than that in order to generate images that will be handled properly by the GAN. It was trained with full images, and doesn't cope well with uncomplete layers & transparencies (ie. a door on a black background, it will try to find pattern between the black & the door that are not suitable for the final result).

Quote
In order to do that, I don't only need the animation layers, I also need to know that a layer (in the tiff file) IS an animation layer (vs. static background), or if the layer has parralax, and if so, what is the depth of that layer.
For lighting effects and other stuff, I also need to know if a layer a different blending mode.

That is already handled by Hades Workshop.
Or do you mean you wanted to automate a script to look over all layers instead of going hand by hand to decide if that#s an animation or transparency?
As I did it before hand by hand, I'm really fast on doing that on my own but if your scroipt works without issues, that's great!

Quote
I would not like to duplicate works here.
I don't know if you do that manually or not, but I wrote scripts that does that automatically (by reading the script field id from both versions in order to match them properly), and that the correct layer from both versions (as your said : blending layers from steam, real background from the PSX version).

Is this really working? if yes, I wished I had this beforehand! xD

Quote
It's a classic layering issue. The main problem is that both version doesn't handle semi-transparent images, so the edges of each layer are sharp, instead of having some semi-transparent pixels in order to blend over.

The Steamrelease use half transperancy on some. For example when Zidane and garnet flees from Steiner in the meeting room, the hatch under the table uses haölf transparency. Or the shield/sign that get knocked off in Alexandria.
I tried to replicate it but Tirlititi inserted only opaque as it gave us problems with seams.
I asked him to do another test with HW as I maybe know why that happened beforehand.

Quote
I'm still unsure if I will use the PSX layers with a neighbooring algorythm, or something similar to what you did, but from early testing, both solutions seems workable.
As long as you've one pixel over that should work. I decided against PSX masks as they're blocky as hell and it didn't really looked good with higher res Backgrounds. :/

Quote
ie .this kind of artefacts on the edge of the wood :



It's not due to the masking.
I've the same issue in photoshop is a perfect mask that I couldn't in-engine, but that I use for ground truth when testing.
It's due to the fact that the wood is interpolated near black pixels.
I've tried to extend the pixel borders to avoid that : I can reduce or increase the artifact effect depending of how it's done, but I'm only manage to get a perfect result when it's interpolated with a recombined image.

As long as you've one pixel more than the PSX masks, that will be completly gone, trust me. That's why i opted for Lanzos upres with 400% and going over some of my scripts to get rid of the half transparency.

If you need more information, let me know. :)

Can you post a comparison between your GAN and manga 109?

Ze_PilOt

  • Fast newbie
  • *
  • Posts: 42
  • Karma: 8
    • View Profile
Re: [FF9PC] Background upscaling mod
« Reply #7 on: 2019-01-10 12:25:22 »
That is already handled by Hades Workshop.

Yes, but I need these informations outside Hades Workshop.

Or do you mean you wanted to automate a script to look over all layers instead of going hand by hand to decide if that#s an animation or transparency?
As I did it before hand by hand, I'm really fast on doing that on my own but if your scroipt works without issues, that's great!

Yes, it's all automated. Even it could be fast to do manually, there are like 800 backgrounds + all the animations, I don't want to spend a week doing what a script can do in 15 min :)


Is this really working? if yes, I wished I had this beforehand! xD

It's really working. The git repository will come soonish.

The Steamrelease use half transperancy on some. For example when Zidane and garnet flees from Steiner in the meeting room, the hatch under the table uses haölf transparency. Or the shield/sign that get knocked off in Alexandria.
I tried to replicate it but Tirlititi inserted only opaque as it gave us problems with seams.
I asked him to do another test with HW as I maybe know why that happened beforehand.

Interesting !

Could you give me the field_id of that scene so I can check that myself ?

Can you post a comparison between your GAN and manga 109?


I will post a first version of the mod around the end of week.

Ze_PilOt

  • Fast newbie
  • *
  • Posts: 42
  • Karma: 8
    • View Profile
Re: [FF9PC] Background upscaling mod
« Reply #8 on: 2019-01-10 14:08:26 »
Here is an exemple with manga109 :



And the FF9-trained network :



Less noise, more details, and it also fix some errors due to the fact that the training set of manga109 doesn't match the kind of pattern we have in FF9

These details are twisted in manga109 :


Comparaison with the PC & PSX  background :


I think the fidelity to the PSX version is a bit better than manga109, while generated as much details as possible.

Ze_PilOt

  • Fast newbie
  • *
  • Posts: 42
  • Karma: 8
    • View Profile
Re: [FF9PC] Background upscaling mod
« Reply #9 on: 2019-01-11 10:15:21 »
I can already give you a list of image correspondance between PSX & PC (assuming that every CD was extracted in its own folder).

This is before layer extraction (as the layer extraction scripts are mixing PC & PSX layers, the match doesn't matter anymore)

It's a Json file, you can easily manipulate it and switch the path to your own.

https://pastebin.com/TKPs8KhQ
« Last Edit: 2019-01-11 12:23:29 by Ze_PilOt »

Fraggoso

  • Insane poster
  • *
  • Posts: 279
  • Karma: 13
    • View Profile
Re: [FF9PC] Background upscaling mod
« Reply #10 on: 2019-01-11 12:50:40 »
Oh you've a ton of dublets in your script.
If you want I can upload a dublete free Field ID per Field ID information.

Ze_PilOt

  • Fast newbie
  • *
  • Posts: 42
  • Karma: 8
    • View Profile
Re: [FF9PC] Background upscaling mod
« Reply #11 on: 2019-01-11 13:30:12 »
Oh you've a ton of dublets in your script.
If you want I can upload a dublete free Field ID per Field ID information.

These are all the layers exported by the batch option in Hades. I know that some background are used multiple times by some fields, but this is not the purpose of that file that only tell "for that image from Hades - PC, use that image of Hades - PSX".

Ze_PilOt

  • Fast newbie
  • *
  • Posts: 42
  • Karma: 8
    • View Profile
Re: [FF9PC] Background upscaling mod
« Reply #12 on: 2019-01-11 16:41:42 »

Fraggoso

  • Insane poster
  • *
  • Posts: 279
  • Karma: 13
    • View Profile
Re: [FF9PC] Background upscaling mod
« Reply #13 on: 2019-01-11 19:22:27 »
Do you really want to release a small slice of the whole mod?
How did you cope with incorrect layering if I may ask? I saw the picture of spindle wheel.
If you load the tiff from HW you may've noticed that the chair (or ta le) is positioned above the spindle wheel or at least some parts.

Another Example is the clock in the field of lindblum church.
One layer is above the clock animation and if you don't at least layer them correctly for the whole image before you use esrgan, you end up without animations at all.

Ze_PilOt

  • Fast newbie
  • *
  • Posts: 42
  • Karma: 8
    • View Profile
Re: [FF9PC] Background upscaling mod
« Reply #14 on: 2019-01-11 21:09:25 »
Do you really want to release a small slice of the whole mod?

Actually I'm planning to release a 0.1 version of the mod : All the backgrounds, but nothing manually checked.
As I'm writing this, The computer is currently recompressing all the bckg in .tex format for re-injection. Unless something went horribly wrong (and I bet it did, as it's the first try :), I should have something tomorrow.

The goal is to detect issues in the automation process (bad layering, artefacts, ...), fix the scripts, and iterate.
I really want to do as few manual operation as possible and concentrate on the important part : the best restoration possible of the images.  I foresee that denoising the images will need more polish (mostly for the skies & others gradients), but I already have some solutions in mind to do that properly.

How did you cope with incorrect layering if I may ask? I saw the picture of spindle wheel.
If you load the tiff from HW you may've noticed that the chair (or ta le) is positioned above the spindle wheel or at least some parts.

Another Example is the clock in the field of lindblum church.
One layer is above the clock animation and if you don't at least layer them correctly for the whole image before you use esrgan, you end up without animations at all.

I don't really know how HW handle the layering for animations, and I don't really care. I'm using HW for convenience, but it's just an extraction tool at that point. I'm exporting all the layers separated + all tiles informations, and I redo the layer compositing myself (automated with python scripting) with these informations.

The process is quite simple : I recompose all the static layers front to back, save that.
Then I isolate each layers of all the animated layers, and for each of of them, I put it in front of the static background.

I'm guess that the problem in HW is that some static layers are in front of the animated layers, so they cover partially the animation.

The only issue I could have is static layers where a front layer cover a back layer that have other pixel information. I don't check that (because there is no parallax or very few), but if it's an issue, it's easy to detect too (detecting a pixel at the same position than an existing one is an easy task)

From what I understand, you export everything (all bckg & animation layer) manually from HW !?
You should really wait for my scripts to avoid that tiresome process :(

It takes about 30 min to extracts all layers properly from the tif from HW, and another 45 min to recompose all the animation frames. Could be quicker (export + compositing + denoise + upscale + de-layering in a single script, and so avoid A TON of costly read/write operations), but as I'm debugging the process, I prefer have all steps well isolated.

in an hour and half, the computer does the work that (I guess) takes days to do manually.
What is very costly for the moment is the photoshop batch denoising. I really want to replace that with a openCV process, but I still need to tweak it to get results as good as photoshop.

I also can detect the duplicate background from different fields (I already know when two files have the exact same layers & dimensions), but I don't really skip these as it's already fast enough for my need as it is. Maybe I will do that if I need to iterate more.
I could also multi-threading everything except the upscaling...
« Last Edit: 2019-01-11 21:31:31 by Ze_PilOt »

Ze_PilOt

  • Fast newbie
  • *
  • Posts: 42
  • Karma: 8
    • View Profile
Re: [FF9PC] Background upscaling mod
« Reply #15 on: 2019-01-12 11:12:36 »
Version 0.1 of the Mod.

This is for testing purpose, please report any visual glitch :)

LINK REMOVED

Little gitch to solve first...
« Last Edit: 2019-01-12 11:41:16 by Ze_PilOt »

Ze_PilOt

  • Fast newbie
  • *
  • Posts: 42
  • Karma: 8
    • View Profile
Re: [FF9PC] Background upscaling mod
« Reply #16 on: 2019-01-13 20:44:20 »
We did a stream yesterday with the first version of the mod :

https://www.twitch.tv/videos/362737421

As you can see, some glitches (that's why it wasn't released publicly), but a proof that the whole script pipeline is working.

I'm now multithreading the scripts while solving these glitches, it's takes a couple minutes instead of half an hour on some steps.

Fraggoso

  • Insane poster
  • *
  • Posts: 279
  • Karma: 13
    • View Profile
Re: [FF9PC] Background upscaling mod
« Reply #17 on: 2019-01-13 20:53:52 »
It looks very promising if you can sort out the glitches and maybe, and that's only if the Vid isn't too compressed: Use ARGB on inserting instead of compressing the tex files.

Ze_PilOt

  • Fast newbie
  • *
  • Posts: 42
  • Karma: 8
    • View Profile
Re: [FF9PC] Background upscaling mod
« Reply #18 on: 2019-01-14 09:47:40 »
It looks very promising if you can sort out the glitches and maybe, and that's only if the Vid isn't too compressed: Use ARGB on inserting instead of compressing the tex files.

I'm pretty sure I can fix the glitches. I wasn't lucky with the test scenes I took, there was no black lines :)

Also I did a mistake with some background matching (altered version as normal version, already solved) and some bad layer composition (as I said previously : static layer covering stuff, I have solution for that).


Fraggoso

  • Insane poster
  • *
  • Posts: 279
  • Karma: 13
    • View Profile
Re: [FF9PC] Background upscaling mod
« Reply #19 on: 2019-01-14 09:51:28 »
I'm really impressed, great job. :)
If you need help with anything, let me know. I know the Fields very well. =)

Ze_PilOt

  • Fast newbie
  • *
  • Posts: 42
  • Karma: 8
    • View Profile
Re: [FF9PC] Background upscaling mod
« Reply #20 on: 2019-01-16 12:37:39 »
Ok, first version of the mod :

https://drive.google.com/file/d/1Gmy38tVHGF-zJqhuyyBn2SWhrsLc9Qz6/view?usp=sharing

Please report any glitch (bad version of a background / animation, ...) with a screenshot & the location (ie. Alexandria, Lindblum, ...)

kennybones

  • Fast newbie
  • *
  • Posts: 30
  • Karma: 2
    • View Profile
Re: [FF9PC] Background upscaling mod
« Reply #21 on: 2019-01-16 13:59:30 »
This isn't by any chance compatible with Memoria?

Fraggoso

  • Insane poster
  • *
  • Posts: 279
  • Karma: 13
    • View Profile
Re: [FF9PC] Background upscaling mod
« Reply #22 on: 2019-01-16 14:00:17 »
Congrats to your first Mod.
You should let it transfer into the release section.

Ze_PilOt

  • Fast newbie
  • *
  • Posts: 42
  • Karma: 8
    • View Profile
Re: [FF9PC] Background upscaling mod
« Reply #23 on: 2019-01-16 14:03:42 »
This isn't by any chance compatible with Memoria?

I will look into it as I want to bring back the FF IX font into the game :)

kennybones

  • Fast newbie
  • *
  • Posts: 30
  • Karma: 2
    • View Profile
Re: [FF9PC] Background upscaling mod
« Reply #24 on: 2019-01-16 14:06:20 »
Sweet! There are a few other mods too that isn't compatible with each other. Seems like there's some issues with the csharp.dll file, where each modder compiles their own file to work specific with their mod. While it needs adjustments to be able to work with multiple mods.

We sure could use a decent mod manager for FF9

Edit: Where do I report errors? Just loaded up a save from Cleyra, I see a few issues right away. It's a real troublesome area though, with a lot of masks.
But the backgrounds in general looks pretty darn good
« Last Edit: 2019-01-16 14:20:17 by kennybones »