Qhimm.com Forums

Final Fantasy 9 => Graphics => FF9 Graphical Releases => Topic started by: Ze_PilOt on 2019-01-08 20:55:40

Title: [FF9PC] Background upscaling mod
Post by: Ze_PilOt on 2019-01-08 20:55:40
Hi,

While in vacation, in December, I gave a try with SR-GAN to upscale FF9 background, following a discussion with some friends that the original background where lost, and that DNN could help finding back these lost assets.
So I took all the exported background and gave it a go..

https://imgur.com/gallery/K1R3wJZ

This is with one of the trained generator (ESRGRAN4x).
The main problem here is that it's trained with a standard photoset (usually BSDS100) that is not suitable for artwork.
I think some people gave it a go mixing it with a set trained with the Manga109 dataset, but again not suitable : Photo & drawing are very different from artworks.

A photo from the set :
(https://www.installer.com/photos/bsds-100.jpg)
A drawing from manga109 :
(http://www.manga109.org/image/cover_and_content/63.jpg)

What we are trying to achieve :
(https://i.imgur.com/NgD8NGF.jpg)

Think of the process like this : The DNN is trained to find pattern is small resolution images, and use patterns from the high resolution images it was trained for to fill the gaps.

BSDS or Manga109 dataset will generate too soft, not very detailed images due to the very simple nature of the images (manga109) or over-detailed and artifacts because of wrongly interpreted patterns (BSDS-100).
Also, the final image we want should have a "hand-painted feel" that none of these dataset offer.
This is often the mistake when I see people giving a go with DNN : You have to train your network with the kind of data you want to manipulate. There is sadly no silver bullet.

I've spend a couple of days and trained a GAN with my own dataset : HD artworks from
https://www.neogaf.com/threads/the-lost-art-of-final-fantasy-ix-mama-robotnik-research-thread.551612/
as well as artwork from different games that kind of look like FF9 style, ie :
http://conceptartworld.com/news/dragon-age-inquisition-environment-concept-art/
https://alishermirzoev.artstation.com/
https://www.pinterest.fr/andreamassa/tomb-raider-art/
https://www.videogamesartwork.com/games/witcher-3-wild-hunt
https://kaidash.artstation.com/projects/6kP9W

A selection of about 110 HD artworks, resulting in a 18.000 images dataset once tiled & randomized (for a DNN, a flipped/rotated image is a brand new image).

(https://i.imgur.com/ww4szEt.png)


After 3 days of training, I've managed to get a better result out of the box.

(https://pbs.twimg.com/media/Dv7C0C7W0AM7vM6.jpg:large)
(left is the original Manga109 / ESGRAN4x mix), right is mine, after a day of training. It's even better now that it's fully trained)

The next step is to automate the process of upscaling the whole game.
I've started from the PSX images, as the PC version is very poor quality wise, due to the de-dithering filtering applied :

(https://pbs.twimg.com/media/DvrfitkWsAAcLhW.jpg:large)
(left is PC, right is PSX).

All was extracted using Hades Workshop.

I've of course had to apply my own filtering for decreasing the dithering without loosing too much details, through a photoshop batch script.

Then the masking problem.

https://twitter.com/Ze_PilOt/status/1080914390306643968

The masking was quickly solve, but I was still unhappy with the border artifacts from the different layers.
So I rewrite part of Hades Workshop to export me more informations (animations, light/effect layers, ...) so I could recombine layers to have a full image for the GAN generator, and to de-composite it back for Re-inserting it for Hades.

I'm happy to say that all these problem are now solved, and I'm in the process of converting the first CD (222 backgrounds) to do a test mod.


All the work -trained generator, automation scripts (normally no human/manual intervention), Hades modifications- (except the dataset for training due to copyright issues) will be available as open source ASAP.


Title: Re: [FF9PC] Background upscaling mod
Post by: Fraggoso on 2019-01-09 07:52:00
Hi, I'm also working on an Update to my original Waifu HD Backgrounds for FFIX.
What do you mean with rewritting HW to export more information? If you're extracting from PSX Source you just have to untick merge layers and you receive every layer (transparency, tiles and animations).

I've extraced all assets from PSX as well as Steam (as you need the light sources from Steam sadly) and I'm about 66% finished with everything.
I'll also release all the PSX and Steam assets as well as every Folder with their respective Field# so no one have to  worry about dubletts and Field#.

The seam issue is because the Steam Version uses one more pixel on each layer so they overlapp (the psx assets are seamless).
I resolved that with resizing the psx layers with 400% size & lanczos algorithm and get rid of the transperancy around the edges. Afterwards I just cut out the Layers on top of the new ESRGAN Image and voila -> No seam problems.

How did you solve it? Maybe your art/mask is better than mine. :)
Title: Re: [FF9PC] Background upscaling mod
Post by: Ze_PilOt on 2019-01-09 08:22:32
Hi, I'm also working on an Update to my original Waifu HD Backgrounds for FFIX.

Nice ! I've come across your mod searching if it was possible to re-inject the background into the game, seing your mod was a deciding factor :)

What do you mean with rewritting HW to export more information? If you're extracting from PSX Source you just have to untick merge layers and you receive every layer (transparency, tiles and animations).

I need more informations than that in order to generate images that will be handled properly by the GAN. It was trained with full images, and doesn't cope well with uncomplete layers & transparencies (ie. a door on a black background, it will try to find pattern between the black & the door that are not suitable for the final result).

In order to do that, I don't only need the animation layers, I also need to know that a layer (in the tiff file) IS an animation layer (vs. static background), or if the layer has parralax, and if so, what is the depth of that layer.
For lighting effects and other stuff, I also need to know if a layer a different blending mode.

I've extraced all assets from PSX as well as Steam (as you need the light sources from Steam sadly) and I'm about 66% finished with everything.

I would not like to duplicate works here.
I don't know if you do that manually or not, but I wrote scripts that does that automatically (by reading the script field id from both versions in order to match them properly), and that the correct layer from both versions (as your said : blending layers from steam, real background from the PSX version).

I will try to release these (python) scripts ASAP (probably thursday or this week-end). I can't pack the final result due to copyright issues, so I need some time to be sure that they are easy enough to handle.

I'll also release all the PSX and Steam assets as well as every Folder with their respective Field# so no one have to  worry about dubletts and Field#.

Yes, that is a good practice.

The seam issue is because the Steam Version uses one more pixel on each layer so they overlapp (the psx assets are seamless).

It's a classic layering issue. The main problem is that both version doesn't handle semi-transparent images, so the edges of each layer are sharp, instead of having some semi-transparent pixels in order to blend over.
Another issue is that, normally, the background layer shouldn't stop right where a layer on top start. That works on the PSX version because of the very sharp edges & low resolution, but I think they had some issues during the upscaling and used an extra pixel to avoid cover issues.

And on top of all that, their upscaling process had some black edge issues, related to bi-linear interpolation on single layer (that was I'm trying to avoid by recombining all the layers in a meaningful way first).

I resolved that with resizing the psx layers with 400% size & lanczos algorithm and get rid of the transperancy around the edges. Afterwards I just cut out the Layers on top of the new ESRGAN Image and voila -> No seam problems.

Actually, the transparency around the edge is not handle at all by the PC version. What is happening is that all pixels with a alpha < 127 result in a black alpha, and all pixels with an alpha <= 127 result in a white alpha.

I'm still unsure if I will use the PSX layers with a neighbooring algorythm, or something similar to what you did, but from early testing, both solutions seems workable.

How did you solve it? Maybe your art/mask is better than mine. :)

I'm not sure, but I think that the layer recombination is key to have a proper result.

ie .this kind of artefacts on the edge of the wood :

(https://i.imgur.com/SwY6HDs.png)

It's not due to the masking.
I've the same issue in photoshop is a perfect mask that I couldn't in-engine, but that I use for ground truth when testing.
It's due to the fact that the wood is interpolated near black pixels.
I've tried to extend the pixel borders to avoid that : I can reduce or increase the artifact effect depending of how it's done, but I'm only manage to get a perfect result when it's interpolated with a recombined image.
Title: Re: [FF9PC] Background upscaling mod
Post by: satsuki on 2019-01-09 09:13:06
Could you share your trained model ? i'd like to test it with FFVII.
Thanks
Title: Re: [FF9PC] Background upscaling mod
Post by: Ze_PilOt on 2019-01-09 11:30:25
Could you share your trained model ? i'd like to test it with FFVII.
Thanks

Yes, I will share it with the automation scripts.
Title: Re: [FF9PC] Background upscaling mod
Post by: Ze_PilOt on 2019-01-09 20:14:43
Exemple of layer recombination, I'm able to output animated GIF :)

When this project is done, I will enhance the scripts to automate GIF creation of all the background, it's so nice !

(https://i.imgur.com/dNACPAO.gif)
Title: Re: [FF9PC] Background upscaling mod
Post by: Fraggoso on 2019-01-10 11:37:15
Nice ! I've come across your mod searching if it was possible to re-inject the background into the game, seing your mod was a deciding factor :)

Great to hear! :D

I need more informations than that in order to generate images that will be handled properly by the GAN. It was trained with full images, and doesn't cope well with uncomplete layers & transparencies (ie. a door on a black background, it will try to find pattern between the black & the door that are not suitable for the final result).

Quote
In order to do that, I don't only need the animation layers, I also need to know that a layer (in the tiff file) IS an animation layer (vs. static background), or if the layer has parralax, and if so, what is the depth of that layer.
For lighting effects and other stuff, I also need to know if a layer a different blending mode.

That is already handled by Hades Workshop.
Or do you mean you wanted to automate a script to look over all layers instead of going hand by hand to decide if that#s an animation or transparency?
As I did it before hand by hand, I'm really fast on doing that on my own but if your scroipt works without issues, that's great!

Quote
I would not like to duplicate works here.
I don't know if you do that manually or not, but I wrote scripts that does that automatically (by reading the script field id from both versions in order to match them properly), and that the correct layer from both versions (as your said : blending layers from steam, real background from the PSX version).

Is this really working? if yes, I wished I had this beforehand! xD

Quote
It's a classic layering issue. The main problem is that both version doesn't handle semi-transparent images, so the edges of each layer are sharp, instead of having some semi-transparent pixels in order to blend over.

The Steamrelease use half transperancy on some. For example when Zidane and garnet flees from Steiner in the meeting room, the hatch under the table uses haölf transparency. Or the shield/sign that get knocked off in Alexandria.
I tried to replicate it but Tirlititi inserted only opaque as it gave us problems with seams.
I asked him to do another test with HW as I maybe know why that happened beforehand.

Quote
I'm still unsure if I will use the PSX layers with a neighbooring algorythm, or something similar to what you did, but from early testing, both solutions seems workable.
As long as you've one pixel over that should work. I decided against PSX masks as they're blocky as hell and it didn't really looked good with higher res Backgrounds. :/

Quote
ie .this kind of artefacts on the edge of the wood :

(https://i.imgur.com/SwY6HDs.png)

It's not due to the masking.
I've the same issue in photoshop is a perfect mask that I couldn't in-engine, but that I use for ground truth when testing.
It's due to the fact that the wood is interpolated near black pixels.
I've tried to extend the pixel borders to avoid that : I can reduce or increase the artifact effect depending of how it's done, but I'm only manage to get a perfect result when it's interpolated with a recombined image.

As long as you've one pixel more than the PSX masks, that will be completly gone, trust me. That's why i opted for Lanzos upres with 400% and going over some of my scripts to get rid of the half transparency.

If you need more information, let me know. :)

Can you post a comparison between your GAN and manga 109?
Title: Re: [FF9PC] Background upscaling mod
Post by: Ze_PilOt on 2019-01-10 12:25:22
That is already handled by Hades Workshop.

Yes, but I need these informations outside Hades Workshop.

Or do you mean you wanted to automate a script to look over all layers instead of going hand by hand to decide if that#s an animation or transparency?
As I did it before hand by hand, I'm really fast on doing that on my own but if your scroipt works without issues, that's great!

Yes, it's all automated. Even it could be fast to do manually, there are like 800 backgrounds + all the animations, I don't want to spend a week doing what a script can do in 15 min :)


Is this really working? if yes, I wished I had this beforehand! xD

It's really working. The git repository will come soonish.

The Steamrelease use half transperancy on some. For example when Zidane and garnet flees from Steiner in the meeting room, the hatch under the table uses haölf transparency. Or the shield/sign that get knocked off in Alexandria.
I tried to replicate it but Tirlititi inserted only opaque as it gave us problems with seams.
I asked him to do another test with HW as I maybe know why that happened beforehand.

Interesting !

Could you give me the field_id of that scene so I can check that myself ?

Can you post a comparison between your GAN and manga 109?


I will post a first version of the mod around the end of week.
Title: Re: [FF9PC] Background upscaling mod
Post by: Ze_PilOt on 2019-01-10 14:08:26
Here is an exemple with manga109 :

(https://i.imgur.com/sthAAfA.jpg)

And the FF9-trained network :

(https://i.imgur.com/g7jxoHx.png)

Less noise, more details, and it also fix some errors due to the fact that the training set of manga109 doesn't match the kind of pattern we have in FF9

These details are twisted in manga109 :
(https://i.imgur.com/CaIHMk2.png)

Comparaison with the PC & PSX  background :
(https://i.imgur.com/T0gtMK1.gif)

I think the fidelity to the PSX version is a bit better than manga109, while generated as much details as possible.
Title: Re: [FF9PC] Background upscaling mod
Post by: Ze_PilOt on 2019-01-11 10:15:21
I can already give you a list of image correspondance between PSX & PC (assuming that every CD was extracted in its own folder).

This is before layer extraction (as the layer extraction scripts are mixing PC & PSX layers, the match doesn't matter anymore)

It's a Json file, you can easily manipulate it and switch the path to your own.

https://pastebin.com/TKPs8KhQ
Title: Re: [FF9PC] Background upscaling mod
Post by: Fraggoso on 2019-01-11 12:50:40
Oh you've a ton of dublets in your script.
If you want I can upload a dublete free Field ID per Field ID information.
Title: Re: [FF9PC] Background upscaling mod
Post by: Ze_PilOt on 2019-01-11 13:30:12
Oh you've a ton of dublets in your script.
If you want I can upload a dublete free Field ID per Field ID information.

These are all the layers exported by the batch option in Hades. I know that some background are used multiple times by some fields, but this is not the purpose of that file that only tell "for that image from Hades - PC, use that image of Hades - PSX".
Title: Re: [FF9PC] Background upscaling mod
Post by: Ze_PilOt on 2019-01-11 16:41:42
Some backgrounds..

https://twitter.com/Ze_PilOt/status/1083775305423167488
Title: Re: [FF9PC] Background upscaling mod
Post by: Fraggoso on 2019-01-11 19:22:27
Do you really want to release a small slice of the whole mod?
How did you cope with incorrect layering if I may ask? I saw the picture of spindle wheel.
If you load the tiff from HW you may've noticed that the chair (or ta le) is positioned above the spindle wheel or at least some parts.

Another Example is the clock in the field of lindblum church.
One layer is above the clock animation and if you don't at least layer them correctly for the whole image before you use esrgan, you end up without animations at all.
Title: Re: [FF9PC] Background upscaling mod
Post by: Ze_PilOt on 2019-01-11 21:09:25
Do you really want to release a small slice of the whole mod?

Actually I'm planning to release a 0.1 version of the mod : All the backgrounds, but nothing manually checked.
As I'm writing this, The computer is currently recompressing all the bckg in .tex format for re-injection. Unless something went horribly wrong (and I bet it did, as it's the first try :), I should have something tomorrow.

The goal is to detect issues in the automation process (bad layering, artefacts, ...), fix the scripts, and iterate.
I really want to do as few manual operation as possible and concentrate on the important part : the best restoration possible of the images.  I foresee that denoising the images will need more polish (mostly for the skies & others gradients), but I already have some solutions in mind to do that properly.

How did you cope with incorrect layering if I may ask? I saw the picture of spindle wheel.
If you load the tiff from HW you may've noticed that the chair (or ta le) is positioned above the spindle wheel or at least some parts.

Another Example is the clock in the field of lindblum church.
One layer is above the clock animation and if you don't at least layer them correctly for the whole image before you use esrgan, you end up without animations at all.

I don't really know how HW handle the layering for animations, and I don't really care. I'm using HW for convenience, but it's just an extraction tool at that point. I'm exporting all the layers separated + all tiles informations, and I redo the layer compositing myself (automated with python scripting) with these informations.

The process is quite simple : I recompose all the static layers front to back, save that.
Then I isolate each layers of all the animated layers, and for each of of them, I put it in front of the static background.

I'm guess that the problem in HW is that some static layers are in front of the animated layers, so they cover partially the animation.

The only issue I could have is static layers where a front layer cover a back layer that have other pixel information. I don't check that (because there is no parallax or very few), but if it's an issue, it's easy to detect too (detecting a pixel at the same position than an existing one is an easy task)

From what I understand, you export everything (all bckg & animation layer) manually from HW !?
You should really wait for my scripts to avoid that tiresome process :(

It takes about 30 min to extracts all layers properly from the tif from HW, and another 45 min to recompose all the animation frames. Could be quicker (export + compositing + denoise + upscale + de-layering in a single script, and so avoid A TON of costly read/write operations), but as I'm debugging the process, I prefer have all steps well isolated.

in an hour and half, the computer does the work that (I guess) takes days to do manually.
What is very costly for the moment is the photoshop batch denoising. I really want to replace that with a openCV process, but I still need to tweak it to get results as good as photoshop.

I also can detect the duplicate background from different fields (I already know when two files have the exact same layers & dimensions), but I don't really skip these as it's already fast enough for my need as it is. Maybe I will do that if I need to iterate more.
I could also multi-threading everything except the upscaling...
Title: Re: [FF9PC] Background upscaling mod
Post by: Ze_PilOt on 2019-01-12 11:12:36
Version 0.1 of the Mod.

This is for testing purpose, please report any visual glitch :)

LINK REMOVED

Little gitch to solve first...
Title: Re: [FF9PC] Background upscaling mod
Post by: Ze_PilOt on 2019-01-13 20:44:20
We did a stream yesterday with the first version of the mod :

https://www.twitch.tv/videos/362737421

As you can see, some glitches (that's why it wasn't released publicly), but a proof that the whole script pipeline is working.

I'm now multithreading the scripts while solving these glitches, it's takes a couple minutes instead of half an hour on some steps.
Title: Re: [FF9PC] Background upscaling mod
Post by: Fraggoso on 2019-01-13 20:53:52
It looks very promising if you can sort out the glitches and maybe, and that's only if the Vid isn't too compressed: Use ARGB on inserting instead of compressing the tex files.
Title: Re: [FF9PC] Background upscaling mod
Post by: Ze_PilOt on 2019-01-14 09:47:40
It looks very promising if you can sort out the glitches and maybe, and that's only if the Vid isn't too compressed: Use ARGB on inserting instead of compressing the tex files.

I'm pretty sure I can fix the glitches. I wasn't lucky with the test scenes I took, there was no black lines :)

Also I did a mistake with some background matching (altered version as normal version, already solved) and some bad layer composition (as I said previously : static layer covering stuff, I have solution for that).

Title: Re: [FF9PC] Background upscaling mod
Post by: Fraggoso on 2019-01-14 09:51:28
I'm really impressed, great job. :)
If you need help with anything, let me know. I know the Fields very well. =)
Title: Re: [FF9PC] Background upscaling mod
Post by: Ze_PilOt on 2019-01-16 12:37:39
Ok, first version of the mod :

https://drive.google.com/file/d/1Gmy38tVHGF-zJqhuyyBn2SWhrsLc9Qz6/view?usp=sharing

Please report any glitch (bad version of a background / animation, ...) with a screenshot & the location (ie. Alexandria, Lindblum, ...)
Title: Re: [FF9PC] Background upscaling mod
Post by: kennybones on 2019-01-16 13:59:30
This isn't by any chance compatible with Memoria?
Title: Re: [FF9PC] Background upscaling mod
Post by: Fraggoso on 2019-01-16 14:00:17
Congrats to your first Mod.
You should let it transfer into the release section.
Title: Re: [FF9PC] Background upscaling mod
Post by: Ze_PilOt on 2019-01-16 14:03:42
This isn't by any chance compatible with Memoria?

I will look into it as I want to bring back the FF IX font into the game :)
Title: Re: [FF9PC] Background upscaling mod
Post by: kennybones on 2019-01-16 14:06:20
Sweet! There are a few other mods too that isn't compatible with each other. Seems like there's some issues with the csharp.dll file, where each modder compiles their own file to work specific with their mod. While it needs adjustments to be able to work with multiple mods.

We sure could use a decent mod manager for FF9

Edit: Where do I report errors? Just loaded up a save from Cleyra, I see a few issues right away. It's a real troublesome area though, with a lot of masks.
But the backgrounds in general looks pretty darn good
Title: Re: [FF9PC] Background upscaling mod
Post by: Covarr on 2019-01-16 14:31:37
Congrats to your first Mod.
You should let it transfer into the release section.
I'll move it to releases if there's a release in the first post; don't want people having to dig to find the release, also don't know if Ze_PilOt wants it there before more of the kinks are worked out.
Title: Re: [FF9PC] Background upscaling mod
Post by: Ze_PilOt on 2019-01-16 15:11:26
Yes, it's still a beta, please don't move it to finished mod yet.
Title: Re: [FF9PC] Background upscaling mod
Post by: kennybones on 2019-01-16 15:34:48
Maybe create and share a google sheet or something where we can document bugs and errors? I'm gonna start a new playthrough and make notes on the way.
Title: Re: [FF9PC] Background upscaling mod
Post by: Ze_PilOt on 2019-01-17 14:24:56
An general issue has been detected in the denoising process : Some grey lines may appears on a lot of background.
This has been fixed in the scripts, next version will be a lot cleaner !
Title: Re: [FF9PC] Background upscaling mod
Post by: Fraggoso on 2019-01-18 07:28:24
An general issue has been detected in the denoising process : Some grey lines may appears on a lot of background.
This has been fixed in the scripts, next version will be a lot cleaner !

Is this because of the masks?
I had this problem as well but solved it using another upres algorithm.
Title: Re: [FF9PC] Background upscaling mod
Post by: Ze_PilOt on 2019-01-18 10:50:13
Is this because of the masks?
I had this problem as well but solved it using another upres algorithm.

No, it's really because of photoshop trying to denoise colors next to black areas :)
Title: Re: [FF9PC] Background upscaling mod
Post by: Fraggoso on 2019-01-18 11:36:19
So did you use the PSX masks and just used 4x size without any filtering so you get a crisp clean but blocky mask or did you upressed the masks as well?
If yes, how? :)
Title: Re: [FF9PC] Background upscaling mod
Post by: Ze_PilOt on 2019-01-18 11:50:27
So did you use the PSX masks and just used 4x size without any filtering so you get a crisp clean but blocky mask or did you upressed the masks as well?
If yes, how? :)

So far I'm using the PSX masks to combine the layers, then the PC masks (upscaled without refiltering) when I de-compose to the original tiles. Seems to works nicely so far, so I doubt I will change the method.
Title: Re: [FF9PC] Background upscaling mod
Post by: Fraggoso on 2019-01-18 11:58:19
So far I'm using the PSX masks to combine the layers, then the PC masks (upscaled without refiltering) when I de-compose to the original tiles. Seems to works nicely so far, so I doubt I will change the method.

Ah yeah, that works. I, for myself, found the masks too ugly though that's why I used another Methos which works and looks cleaner (at least for myself ^^).
Title: Re: [FF9PC] Background upscaling mod
Post by: satsuki on 2019-01-18 17:14:24
Yes it's an hard work to optimize the layer's cutting but it's realy worth it.
A base layer cutting unoptimised can realy ruin your works.

For exemple with FF7, the base layer cutting (only 4x scale without any edge optimisation):
(http://yatoshicom.free.fr/ff7p/oldmask.jpg)

My optimised version (layer done with potrace and imagemagik):
(http://yatoshicom.free.fr/ff7p/newmask.jpg)

Of course it's your works and you'll done it as you want to but a great layer cutting will make your upscale shine.
Title: Re: [FF9PC] Background upscaling mod
Post by: Ze_PilOt on 2019-01-19 12:48:39
V01 was retired. V03 is up :

https://drive.google.com/open?id=1IA3r5iJDP-Y8CyGwiqQEBmQEp53pa4sA

Changelog :
- Fixed grey lines appearing on several backgrounds,
- Fixed composition errors (text embedded on some background, wrong skies, ...)
- Better compression (should not affect quality).

And a PDF with a selection of background art :
https://drive.google.com/open?id=1IQmNY2j5ZLag8qOiWru1bspURVVRb0tT
Title: Re: [FF9PC] Background upscaling mod
Post by: Ze_PilOt on 2019-01-19 12:49:36
Yes it's an hard work to optimize the layer's cutting but it's realy worth it.
A base layer cutting unoptimised can realy ruin your works.

Well, that's something that can be easily automated. Compositing layers can be tricky and complicated, but it's not dark magic and has basic rules that you can follow (and automate).
Title: Re: [FF9PC] Background upscaling mod
Post by: satsuki on 2019-01-19 13:56:42
I Know
My processus is automated.
But lots of small optimisations need to be done by hand for an optimal result.
Title: Re: [FF9PC] Background upscaling mod
Post by: Ze_PilOt on 2019-01-19 17:36:45
I Know
My processus is automated.
But lots of small optimisations need to be done by hand for an optimal result.

Yes, the layering script I'm currently using takes 400 lines of code to handle all the cases. It was about 20 lines in my first tests.
Title: Re: [FF9PC] Background upscaling mod
Post by: Ze_PilOt on 2019-01-21 11:11:51
A version compatible with Memoria :
https://drive.google.com/open?id=1WxAbradSPoEvo1d1SAnFKVpxL8TNv63x
Title: Re: [FF9PC] Background upscaling mod
Post by: kennybones on 2019-01-21 12:19:51
It works! So now we can actually have HD backgrounds and Memoria working at the same time. Now I just need to figure out how to get the scaled battle UI working. I was under the impression that it was already packaged with Memoria, but it's not turned on by default at least.
Title: Re: [FF9PC] Background upscaling mod
Post by: Ze_PilOt on 2019-01-21 13:46:24
It works! So now we can actually have HD backgrounds and Memoria working at the same time. Now I just need to figure out how to get the scaled battle UI working. I was under the impression that it was already packaged with Memoria, but it's not turned on by default at least.

The widescreen option is broken too. I'm guessing there is an hardcoded offset relative to the original tileset size, but I can't find where it is.
Title: Re: [FF9PC] Background upscaling mod
Post by: kennybones on 2019-01-21 13:56:46
Aha! That makes sense :) I'll just disable widescreen for now.
I've noticed you seem to mean that this mod of yours is more of a guide for other to improve upon right? Every background looks great as far as I've seen. Process seems to work great too, masking looks fine (depends on how much of a perfectionist you are). I do think satsuki's masking looks a tad better. But he's doing some manual work, and you wanna stay away from manual work like the plague :D It's just not part of your philosophy regarding the pipeline you've developed.

I'd think your work could be very much appreciated on FF7 and 8 as well, just giving the other modders a hand at making their lives a bit easier.
Title: Re: [FF9PC] Background upscaling mod
Post by: Ze_PilOt on 2019-01-21 14:06:54
Aha! That makes sense :) I'll just disable widescreen for now.
I've noticed you seem to mean that this mod of yours is more of a guide for other to improve upon right? Every background looks great as far as I've seen. Process seems to work great too, masking looks fine (depends on how much of a perfectionist you are). I do think satsuki's masking looks a tad better. But he's doing some manual work, and you wanna stay away from manual work like the plague :D It's just not part of your philosophy regarding the pipeline you've developed.

I'd think your work could be very much appreciated on FF7 and 8 as well, just giving the other modders a hand at making their lives a bit easier.

A manual mask will always be better. You can do it on dozen of images, not hundred.
I just though of another way to compute a very clean mask, but I have to check if it works :)

I will release the scripts soon, I still need to clean them.
Title: Re: [FF9PC] Background upscaling mod
Post by: Ze_PilOt on 2019-01-22 11:13:26
Aha! That makes sense :) I'll just disable widescreen for now.

You can redownload the package, it was updated with a fix.

I've pushed my change to the memoria repository :
https://github.com/Albeoris/Memoria/pull/78
Title: Re: [FF9PC] Background upscaling mod
Post by: Ze_PilOt on 2019-01-22 14:12:19
Also a font matching the PSX one to use with Memoria. It's the "Zidane" font with an extended set of characters (latin-extended : àéè')

https://drive.google.com/open?id=1pEqqMw-yunNruMr2GRvQo7_AUa8I4Kln

[Font]
Enabled = 1
Names = "Garnet", "Times Bold"
Size = 24

Title: Re: [FF9PC] Background upscaling mod
Post by: DanTsukasa on 2019-01-23 04:31:01
I saw that you mentioned automating exports and turning everything into gifs.

I don't know if you still intend to do it or not, but I thought I'd share when I did it.
I did it back in 2015, but made 2 albums, of every single background in the game, both animated and non-animated, as (largely) seemless gifs.
One or two small hiccups, I just realised I never shared them here in the forums.

Dynamic Backgrounds (Animated)
https://imgur.com/a/ra3OU
 (https://imgur.com/a/ra3OU)

Static Backgrounds
https://imgur.com/a/A09TB (https://imgur.com/a/A09TB)

If you're able to make the same thing at higher resolutions that'll be really cool.

Also, I can't explain how happy I am to finally see someone not call the PSX renders 'downscales from higher resolution renders', which is what half the internet seems to think was how they made the PS1 renders. It's also pretty cool to see someone notice that some of the 'lost art of FFIX' is concept art, and things rendered just for an artists portfolio  and not 'original resolution renders or 'secret files from production'.
Title: Re: [FF9PC] Background upscaling mod
Post by: Ze_PilOt on 2019-01-23 06:51:52
I saw that you mentioned automating exports and turning everything into gifs.

I don't know if you still intend to do it or not, but I thought I'd share when I did it.
I did it back in 2015, but made 2 albums, of every single background in the game, both animated and non-animated, as (largely) seemless gifs.
One or two small hiccups, I just realised I never shared them here in the forums.

Dynamic Backgrounds (Animated)
https://imgur.com/a/ra3OU
 (https://imgur.com/a/ra3OU)

Static Backgrounds
https://imgur.com/a/A09TB (https://imgur.com/a/A09TB)

If you're able to make the same thing at higher resolutions that'll be really cool.

Really nice works ! That's exactly what I had in mind !

I will try to recreate them in HD, but it's already quite good as they are !

Also, I can't explain how happy I am to finally see someone not call the PSX renders 'downscales from higher resolution renders', which is what half the internet seems to think was how they made the PS1 renders. It's also pretty cool to see someone notice that some of the 'lost art of FFIX' is concept art, and things rendered just for an artists portfolio  and not 'original resolution renders or 'secret files from production'.

Thanks :)
Title: Re: [FF9PC] Background upscaling mod
Post by: Ze_PilOt on 2019-01-23 16:43:50
Good news everyone !

V04 of the mod (first official release - Not beta anymore !) :

https://drive.google.com/open?id=1D0CEu3YqFn5zYbiTM3ghzm9IdxQ_2M6h

Changelog :
- Fixed the desaturation of the background
- Crispier graphics !
- Custom launcher
- optional PSX font included

The repository with the scripts & the trained model :

https://github.com/ZePilOt/FFIX_Upscale_Scripts
Title: Re: [FF9PC] Background upscaling mod
Post by: Aavock on 2019-01-23 19:49:18
Awesome job Ze_PilOt, definitely an eye candy ! Merci à toi !

Is it possible to include in the memoria.ini an option for widescreen support during battles/fields only ? Something like:

WidescreenSupportField = 0
WidescreenSupportBattle = 1
Title: Re: [FF9PC] Background upscaling mod
Post by: Ze_PilOt on 2019-01-24 08:29:22
Awesome job Ze_PilOt, definitely an eye candy ! Merci à toi !

Is it possible to include in the memoria.ini an option for widescreen support during battles/fields only ? Something like:

WidescreenSupportField = 0
WidescreenSupportBattle = 1

Should be doable.

It's a bit annoying, but you can use alt-space to switch for now.
Title: Re: [FF9PC] Background upscaling mod
Post by: kennybones on 2019-01-24 11:37:18
This might not be the place to ask but I've tried to reinstall Memoria several times and it seems like its stuck on superspeed mode. Battles are atleast twice as fast, and I suspect everything else is twice as fast as well.

Btw, the backgrounds look even better now! I'm really impressed, it's like a whole new game!
Title: Re: [FF9PC] Background upscaling mod
Post by: Ze_PilOt on 2019-01-24 14:21:27
You can check the memoria.ini to see if something seems wrong. Probably battleFPS or battleSpeed.
Title: Re: [FF9PC] Background upscaling mod
Post by: Fraggoso on 2019-01-24 14:34:45
You should mention in your readme that it's vital to uncheck one specific Field in the extration of the PC Fields -> Opening-For-FMV or else HW will crash everytime. :)
Title: Re: [FF9PC] Background upscaling mod
Post by: Ze_PilOt on 2019-01-24 17:39:14
You should mention in your readme that it's vital to uncheck one specific Field in the extration of the PC Fields -> Opening-For-FMV or else HW will crash everytime. :)

Indeed. It's updated (as well as a better & faster way to handle overlapping layers).
Title: Re: [FF9PC] Background upscaling mod
Post by: kennybones on 2019-01-25 02:05:48
Just had to post this. Look at how beautiful the background are!
I'm also using a super light reshade that does very little, just some slight bloom and slight increase in saturation.

(https://i.ibb.co/sFf7Svr/20190125040143-1.jpg) (https://ibb.co/WchC9W1)
(https://i.ibb.co/yB3274P/20190125035640-1.jpg) (https://ibb.co/yB3274P)
(https://i.ibb.co/1ZM0hvp/20190125040023-1.jpg) (https://ibb.co/PxMr3mb)
(https://i.ibb.co/wc4XjqB/20190125040332-1.jpg) (https://ibb.co/cYQHV9k)



Title: Re: [FF9PC] Background upscaling mod
Post by: Ze_PilOt on 2019-01-25 06:46:48
The reshade is subtle and working very well ! Could you share your settings ?

They are still some artefacts around animated objects, but I have a way to reduce them quite a lot (and it doesn't need any reprocessing)
Title: Re: [FF9PC] Background upscaling mod
Post by: Fraggoso on 2019-01-25 07:45:29
Are you compressing your tex with DX5?
I opted for no compression ARGB and it looks better  but of course your filesize is bigger (packed it's still 2 Gigs or something). maybe you want to give that a shot as well. :)
Title: Re: [FF9PC] Background upscaling mod
Post by: Ze_PilOt on 2019-01-25 08:30:44
Are you compressing your tex with DX5?
I opted for no compression ARGB and it looks better  but of course your filesize is bigger (packed it's still 2 Gigs or something). maybe you want to give that a shot as well. :)

I tried ARGB but it doesn't increased the quality by a factor that justify the size of the files, and more importantly, the in-game loading times.
Title: Re: [FF9PC] Background upscaling mod
Post by: kennybones on 2019-01-25 14:25:11
Here's the preset:
https://drive.google.com/file/d/1yLeVziSwonb43-W4EoXgQ0k_SOHUGM0f/view?usp=sharing

It's posted here too:
https://sfx.thelazy.net/games/preset/5520/
Title: Re: [FF9PC] Background upscaling mod
Post by: Fraggoso on 2019-01-26 11:51:44
Can you adept your script to take the PSX layers instead of the PC layers?
As the PSX Fields are stored in CD_X it's not as easy I guess?

I want to use the PSX Layers, Filter them and use them later on for the final cut.

Hm... If I run 1-match_pc_psx.py I don't receive the - match_PC_PSX.json but Duplicated_PC_Images.json

Error Message is:
(https://i.imgur.com/C4QXnqv.png)

Any clue? ^^
Title: Re: [FF9PC] Background upscaling mod
Post by: Ze_PilOt on 2019-01-26 14:07:52
The scripts are using the PSX layers. But some layers don't exists in the PSX version (ie. field text).
They also takes the PC version in the cases of effect layers (blending,...). You need both.
Title: Re: [FF9PC] Background upscaling mod
Post by: Fraggoso on 2019-01-26 14:57:35
I've both fields in their respective folder. :/
From my understanding is, that your script use the alpha from the pc version for the masking or not?

I want to use the psx masks instead.
Title: Re: [FF9PC] Background upscaling mod
Post by: Ze_PilOt on 2019-01-26 15:53:23
I've both fields in their respective folder. :/
From my understanding is, that your script use the alpha from the pc version for the masking or not?

I want to use the psx masks instead.

It will use the psx masks for compositing, and the PC mask for re-injection into the game. It is commented in the scripts.

As I've said in PM, for every tiff file exported from Hades, you should have a .tiff.info in the exact same folder, coming from my repository. You can easily spot missing tiff files by looking at a .tiff.info not having a .tiff file next to it.
Title: Re: [FF9PC] Background upscaling mod
Post by: Ze_PilOt on 2019-01-26 15:58:22
(https://i.imgur.com/q1sV6uO.png)

If you see a blank icon with no "image" icon next to it, you have a missing file.

It will be the same process for the PSX images.
Title: Re: [FF9PC] Background upscaling mod
Post by: Ze_PilOt on 2019-01-26 19:23:26
New version !

https://twitter.com/Ze_PilOt/status/1089207667304919040

Download link :

https://t.co/LtyUjPvhyC

The repository was also updated with the fixes used in that new version :

https://github.com/ZePilOt/FFIX_Upscale_Scripts/commit/e5d25cbefd7d65d557b7d292d86430995a4c591d
and
https://github.com/ZePilOt/FFIX_Upscale_Scripts/commit/73f2f52dfcab7814057702df68028f2209acdeab
Title: Re: [FF9PC] Background upscaling mod
Post by: Bosola on 2019-01-26 22:15:46
This is truly excellent work. It would never have occurred to me to use ML to upscale the background layers. The results are phenomenal in a game as beautiful as FFIX.

My only concern - and I don't really want to add a sour note to this discussion - what's the position of anyone who distributes artwork that's transformed from SE's original assets? Are ML manipulated backgrounds treated as a transformed work? I wouldn't want any HD pack to be stymied by legal problems.
Title: Re: [FF9PC] Background upscaling mod
Post by: Ze_PilOt on 2019-01-27 08:30:34
This is truly excellent work. It would never have occurred to me to use ML to upscale the background layers. The results are phenomenal in a game as beautiful as FFIX.

My only concern - and I don't really want to add a sour note to this discussion - what's the position of anyone who distributes artwork that's transformed from SE's original assets? Are ML manipulated backgrounds treated as a transformed work? I wouldn't want any HD pack to be stymied by legal problems.

I've asked SE directly, but got the usual answer : We can't endorse it, please follow the user agreement.

It's clearly in breach on that agreement on two points :
- Don't re-verse engineer the game
- Don't (re)use assets

For the first point, I'm living in Europe where such clause is illegal (https://www.worldcadaccess.com/blog/2012/05/eu-court-legalizes-reverse-engineering-of-software.html)
Anything for inter-operability is legal.

For the second point, I believe my work can be considered as fair used :
- It's educational (I'm releasing the whole process in open source), and cultural (restoring lost art).
- I do not sell it and have no commercial advantage.
- It's partial (it's only the background of the game, not the whole game) and derivative (restoration is considered derivative)
- It doesn't hurt their business (on the contrary, they can sell more games because of it)

Of course, if they send me a cease & desist, I won't fight against it, but I don't think they have any interest in doing so.

Title: Re: [FF9PC] Background upscaling mod
Post by: kennybones on 2019-01-27 11:07:30
The waifu background mods have been available for years for all FF games though. The only difference here is that an AI has learned how to paint.
If anything, mods like this will help sell the game! It brings more attention to the games and it's never looked better.

I mean, I bought the steam edition JUST to use this mods.

Edit: Looks like you're getting some attention Ze_PilOt :)
https://www.pcgamer.com/modders-are-using-ai-to-upscale-pre-rendered-ps1-backgrounds-with-phenomenal-results/?ns_campaign=article-feed&amp;ns_mchannel=ref&amp;ns_source=steam&amp;ns_linkname=0&amp;ns_fee=0#comment-jump
Title: Re: [FF9PC] Background upscaling mod
Post by: MattST on 2019-01-31 12:43:44
Hello,

I've been trying to replace the download with the files in the FFIX steam installation folder, following instructions, but because I'm a real ignorant at these matters, something's wrong and I see all messed up crazy lines. I stayed up all night trying to make changes and figure something out. Could you please explain me from scratch how to install this mod? Sorry for this boring question and thank you ^^'
Title: Re: [FF9PC] Background upscaling mod
Post by: PureRok on 2019-02-03 22:44:29
It's been a while since I last played, so I don't know if I had this issue previously, but after installing this mod the game treats all of my button inputs as though I've hit the button several times.

If I press pause it will pause/unpause several times in quick succession. Same thing happens with other buttons and navigating menus. The game is running at 30fps, so I don't think it's running fast.

Edit: After further googling I found a fix for this. Seems to be an issue with the game and just starting the game without the controller plugged in fixes it.
Title: Re: [FF9PC] Background upscaling mod
Post by: Ze_PilOt on 2019-02-05 08:13:45
For my FF IX mod, I end up adding a lot of pre and post processing (de-dithering, sharpening, denoising) that took a long time to process.
So I'm retraining a ESGRAN network with the original background (no process) as input and the current version of my mod as output, and now, I will have a network that can get rid of dithering artefacts and have more details than the original process :

https://twitter.com/Ze_PilOt/status/1092709418276438016

I'm not sure that network will be usuable with any other kind of texture, but as many people use manga109 for stuff that are not drawed, it might work (probably for PSX textures)
Title: Re: [FF9PC] Background upscaling mod
Post by: jasmer on 2019-02-06 02:57:55
I peeked at your twitter link and my jaw literally dropped (I sound like a bad clickbait article now) when I saw one of the before and after sets. Absolutely fantastic.
Title: Re: [FF9PC] Background upscaling mod
Post by: Fraggoso on 2019-02-06 12:17:42
Are you going to add different methods to your mask?
I tried your mod and the only grudge so far is really the horrible PC mask. xD

If you want, I've a better method of processing the masks.
Title: Re: [FF9PC] Background upscaling mod
Post by: Ze_PilOt on 2019-02-06 14:59:00
Are you going to add different methods to your mask?
I tried your mod and the only grudge so far is really the horrible PC mask. xD

If you want, I've a better method of processing the masks.

I still don't get why you are saying that the PSX alpha are better.

The PC alphas are actually the PSX alpha rescaled + potraced.
As the game only support black & white alpha (no AA), it's the best way to manage them.

The main problem is that the PSX alpha are a bit off. At PSX resolution it was okay, but not at PC resolution.


Do you have any example of PC vs PSX ?
Title: Re: [FF9PC] Background upscaling mod
Post by: Fraggoso on 2019-02-06 18:41:26
I'll show you that the pc alpha is way more of because of the resizing they did (weird algorithm or something like that).
You can't take the psx 1:1 of course but with some filtering they look way better and more organic than the pc.

I'll post a comparison later. :)

Edit:
http://www.framecompare.com/image-compare/screenshotcomparison/199MNNNU (Zoom in max and look over the masks)
Ingame it looks better because most of the Fields are also zoomed in.
Round objects have a rounder masks opposed to the tooth saw approach of the PC or blocky approach of the PSX masks.

I also uploaded a psd file. zoom in and you notice the difference.
https://www.dropbox.com/s/q4eflnt2hvhighg/mask%20difference.psd?dl=0

I've all the scripts ready, if you want to give them a shot (Gimp though).
Title: Re: [FF9PC] Background upscaling mod
Post by: snouz on 2019-07-05 23:08:54
Hi everybody,

Just want to let you know I'm currently redoing masks by hand. I developed a technique to be efficient, and after about 2 weeks (while working a full time job), I'm at 140 fields out of 667 (between Dali & Lindblum). I'm confident I will be able to work on this to completion.
Title: Re: [FF9PC] Background upscaling mod
Post by: Fraggoso on 2019-07-11 20:46:50
Bets of luck on that one. Making masks by hand is crazy and that is what we're, right? ;)
Title: Re: [FF9PC] Background upscaling mod
Post by: snouz on 2019-07-20 13:06:24
Bets of luck on that one. Making masks by hand is crazy and that is what we're, right? ;)
Thanks : )

So here's my advancement update: about a week ago, I managed to test everything ingame. I modified zePilot's script to be able to work quicker. He kindly sent me all the finished images to help me.
I realized I had a bunch of black zones due to a fact I did not foresee. I'll be more efficient in the future. I had to manually correct each background to correct that.
Now, everything up to Dali is finished completely.

I also improved most of the animations to avoid the border strangeness on them.

I fixed the images/animations of the windmill's upper floor in Dali, and a few others.

I'll come up with comparison shots soon.
Title: Re: [FF9PC] Background upscaling mod
Post by: snouz on 2019-07-29 08:36:47
I'm 28% done (190/678) (59 to go to finish the first CD, as it's the biggest)
For ref. I'm beginning Lindblum Castle.

I experimented a bit with worldmap texturing as well, walking on Fraggoso's steps.

Waifu2x:
(https://i.imgur.com/EtBn6gf.jpg)
Edit:ESRGAN:
(https://i.imgur.com/5wHNOFH.jpg)
Title: Re: [FF9PC] Background upscaling mod
Post by: snouz on 2019-08-04 20:39:41
Update: I have finished Lindblum. 212/678
Title: Re: [FF9PC] Background upscaling mod
Post by: Aavock on 2019-08-04 23:24:34
Quite excited to see the results snouz, keep it up!
Title: Re: [FF9PC] Background upscaling mod
Post by: snouz on 2019-08-07 10:42:57
Just wanted to share the level of detail I went for with the masks:

(https://i.imgur.com/E8YADyT.gif)

(https://i.imgur.com/10pTbFs.gif)

Ingame, it's MUCH nicer than before.
Title: Re: [FF9PC] Background upscaling mod
Post by: olearyf2525 on 2019-08-07 12:51:12
Looks great Snouz, keep up the good work.
Title: Re: [FF9PC] Background upscaling mod
Post by: jembhus on 2019-09-02 08:43:05
I've been enjoying your works and completed the games on god knows how many times.
your work of art is the closest thing we have on FF IX remaster, and thank you very much for restoring my childhood
but what if i tell you there are archives of from the former FF IX development team spread on the internet.
please check https://www.neogaf.com/threads/the-lost-art-of-final-fantasy-ix-mama-robotnik-research-thread.551612/
it's not complete but combined with your works it should be complements each other
Title: Re: [FF9PC] Background upscaling mod
Post by: snouz on 2019-09-02 19:49:00
I've been enjoying your works and completed the games on god knows how many times.
your work of art is the closest thing we have on FF IX remaster, and thank you very much for restoring my childhood
but what if i tell you there are archives of from the former FF IX development team spread on the internet.
please check https://www.neogaf.com/threads/the-lost-art-of-final-fantasy-ix-mama-robotnik-research-thread.551612/
it's not complete but combined with your works it should be complements each other

Unfortunately, there are at least 3 good reasons no modder is getting close to these artworks, while being aware of them.
1) they're very much copyrighted, they are on the artist's portfolio, and taking material from other sources (like other FF games or original artwork) is pretty much forbidden
2) it's very mych incomplete (there are 816 background images in the game, about 28 here), and does not have the final light applied. This would compromise the visual unity
3) Most fields also have animation frames, surimposed stuff etc. A static image is not enough.
Title: Re: [FF9PC] Background upscaling mod
Post by: hush3r on 2020-05-06 02:16:46
Hello!

I'm so sorry if this is not the proper thread, and for the noob question. FF9 is my favorite game from childhood, and I replay it once every few years. I just discovered this modding, "re-mastering" effort and have not been so excited for something in a long time. As someone who doesn't know much about this stuff: how do I go from not having the game, to installing and modding and playing through with these re-done beautiful backgrounds?

I have a MacBook Pro, so I'm not sure if that affects my ability to be able to download the game. If so, and if the rest of the process is simple enough for a bozo like me to go through, I would be willing to go out and buy a PC just for this purpose.

Would love any insight or suggestions you may have.

Hope everyone is staying safe and sane out there in these strange and challenging times.

Warm regards,

-Justin
Title: Re: [FF9PC] Background upscaling mod
Post by: snouz on 2020-05-12 13:11:23
Hello!

I'm so sorry if this is not the proper thread, and for the noob question. FF9 is my favorite game from childhood, and I replay it once every few years. I just discovered this modding, "re-mastering" effort and have not been so excited for something in a long time. As someone who doesn't know much about this stuff: how do I go from not having the game, to installing and modding and playing through with these re-done beautiful backgrounds?

I have a MacBook Pro, so I'm not sure if that affects my ability to be able to download the game. If so, and if the rest of the process is simple enough for a bozo like me to go through, I would be willing to go out and buy a PC just for this purpose.

Would love any insight or suggestions you may have.

Hope everyone is staying safe and sane out there in these strange and challenging times.

Warm regards,

-Justin
Hi, you can install a double boot with Windows using integrated software bootcamp. Google is your friend for that.