Final Fantasy 9 > FF9 Graphical Releases
[FF9PC] Background upscaling mod
Ze_PilOt:
Hi,
While in vacation, in December, I gave a try with SR-GAN to upscale FF9 background, following a discussion with some friends that the original background where lost, and that DNN could help finding back these lost assets.
So I took all the exported background and gave it a go..
https://imgur.com/gallery/K1R3wJZ
This is with one of the trained generator (ESRGRAN4x).
The main problem here is that it's trained with a standard photoset (usually BSDS100) that is not suitable for artwork.
I think some people gave it a go mixing it with a set trained with the Manga109 dataset, but again not suitable : Photo & drawing are very different from artworks.
A photo from the set :
A drawing from manga109 :
What we are trying to achieve :
Think of the process like this : The DNN is trained to find pattern is small resolution images, and use patterns from the high resolution images it was trained for to fill the gaps.
BSDS or Manga109 dataset will generate too soft, not very detailed images due to the very simple nature of the images (manga109) or over-detailed and artifacts because of wrongly interpreted patterns (BSDS-100).
Also, the final image we want should have a "hand-painted feel" that none of these dataset offer.
This is often the mistake when I see people giving a go with DNN : You have to train your network with the kind of data you want to manipulate. There is sadly no silver bullet.
I've spend a couple of days and trained a GAN with my own dataset : HD artworks from
https://www.neogaf.com/threads/the-lost-art-of-final-fantasy-ix-mama-robotnik-research-thread.551612/
as well as artwork from different games that kind of look like FF9 style, ie :
http://conceptartworld.com/news/dragon-age-inquisition-environment-concept-art/
https://alishermirzoev.artstation.com/
https://www.pinterest.fr/andreamassa/tomb-raider-art/
https://www.videogamesartwork.com/games/witcher-3-wild-hunt
https://kaidash.artstation.com/projects/6kP9W
A selection of about 110 HD artworks, resulting in a 18.000 images dataset once tiled & randomized (for a DNN, a flipped/rotated image is a brand new image).
After 3 days of training, I've managed to get a better result out of the box.
(left is the original Manga109 / ESGRAN4x mix), right is mine, after a day of training. It's even better now that it's fully trained)
The next step is to automate the process of upscaling the whole game.
I've started from the PSX images, as the PC version is very poor quality wise, due to the de-dithering filtering applied :
(left is PC, right is PSX).
All was extracted using Hades Workshop.
I've of course had to apply my own filtering for decreasing the dithering without loosing too much details, through a photoshop batch script.
Then the masking problem.
https://twitter.com/Ze_PilOt/status/1080914390306643968
The masking was quickly solve, but I was still unhappy with the border artifacts from the different layers.
So I rewrite part of Hades Workshop to export me more informations (animations, light/effect layers, ...) so I could recombine layers to have a full image for the GAN generator, and to de-composite it back for Re-inserting it for Hades.
I'm happy to say that all these problem are now solved, and I'm in the process of converting the first CD (222 backgrounds) to do a test mod.
All the work -trained generator, automation scripts (normally no human/manual intervention), Hades modifications- (except the dataset for training due to copyright issues) will be available as open source ASAP.
Fraggoso:
Hi, I'm also working on an Update to my original Waifu HD Backgrounds for FFIX.
What do you mean with rewritting HW to export more information? If you're extracting from PSX Source you just have to untick merge layers and you receive every layer (transparency, tiles and animations).
I've extraced all assets from PSX as well as Steam (as you need the light sources from Steam sadly) and I'm about 66% finished with everything.
I'll also release all the PSX and Steam assets as well as every Folder with their respective Field# so no one have to worry about dubletts and Field#.
The seam issue is because the Steam Version uses one more pixel on each layer so they overlapp (the psx assets are seamless).
I resolved that with resizing the psx layers with 400% size & lanczos algorithm and get rid of the transperancy around the edges. Afterwards I just cut out the Layers on top of the new ESRGAN Image and voila -> No seam problems.
How did you solve it? Maybe your art/mask is better than mine. :)
Ze_PilOt:
--- Quote from: Fraggoso on 2019-01-09 07:52:00 ---Hi, I'm also working on an Update to my original Waifu HD Backgrounds for FFIX.
--- End quote ---
Nice ! I've come across your mod searching if it was possible to re-inject the background into the game, seing your mod was a deciding factor :)
--- Quote from: Fraggoso on 2019-01-09 07:52:00 ---What do you mean with rewritting HW to export more information? If you're extracting from PSX Source you just have to untick merge layers and you receive every layer (transparency, tiles and animations).
--- End quote ---
I need more informations than that in order to generate images that will be handled properly by the GAN. It was trained with full images, and doesn't cope well with uncomplete layers & transparencies (ie. a door on a black background, it will try to find pattern between the black & the door that are not suitable for the final result).
In order to do that, I don't only need the animation layers, I also need to know that a layer (in the tiff file) IS an animation layer (vs. static background), or if the layer has parralax, and if so, what is the depth of that layer.
For lighting effects and other stuff, I also need to know if a layer a different blending mode.
--- Quote from: Fraggoso on 2019-01-09 07:52:00 ---I've extraced all assets from PSX as well as Steam (as you need the light sources from Steam sadly) and I'm about 66% finished with everything.
--- End quote ---
I would not like to duplicate works here.
I don't know if you do that manually or not, but I wrote scripts that does that automatically (by reading the script field id from both versions in order to match them properly), and that the correct layer from both versions (as your said : blending layers from steam, real background from the PSX version).
I will try to release these (python) scripts ASAP (probably thursday or this week-end). I can't pack the final result due to copyright issues, so I need some time to be sure that they are easy enough to handle.
--- Quote from: Fraggoso on 2019-01-09 07:52:00 ---I'll also release all the PSX and Steam assets as well as every Folder with their respective Field# so no one have to worry about dubletts and Field#.
--- End quote ---
Yes, that is a good practice.
--- Quote from: Fraggoso on 2019-01-09 07:52:00 ---The seam issue is because the Steam Version uses one more pixel on each layer so they overlapp (the psx assets are seamless).
--- End quote ---
It's a classic layering issue. The main problem is that both version doesn't handle semi-transparent images, so the edges of each layer are sharp, instead of having some semi-transparent pixels in order to blend over.
Another issue is that, normally, the background layer shouldn't stop right where a layer on top start. That works on the PSX version because of the very sharp edges & low resolution, but I think they had some issues during the upscaling and used an extra pixel to avoid cover issues.
And on top of all that, their upscaling process had some black edge issues, related to bi-linear interpolation on single layer (that was I'm trying to avoid by recombining all the layers in a meaningful way first).
--- Quote from: Fraggoso on 2019-01-09 07:52:00 ---I resolved that with resizing the psx layers with 400% size & lanczos algorithm and get rid of the transperancy around the edges. Afterwards I just cut out the Layers on top of the new ESRGAN Image and voila -> No seam problems.
--- End quote ---
Actually, the transparency around the edge is not handle at all by the PC version. What is happening is that all pixels with a alpha < 127 result in a black alpha, and all pixels with an alpha <= 127 result in a white alpha.
I'm still unsure if I will use the PSX layers with a neighbooring algorythm, or something similar to what you did, but from early testing, both solutions seems workable.
--- Quote from: Fraggoso on 2019-01-09 07:52:00 ---How did you solve it? Maybe your art/mask is better than mine. :)
--- End quote ---
I'm not sure, but I think that the layer recombination is key to have a proper result.
ie .this kind of artefacts on the edge of the wood :
It's not due to the masking.
I've the same issue in photoshop is a perfect mask that I couldn't in-engine, but that I use for ground truth when testing.
It's due to the fact that the wood is interpolated near black pixels.
I've tried to extend the pixel borders to avoid that : I can reduce or increase the artifact effect depending of how it's done, but I'm only manage to get a perfect result when it's interpolated with a recombined image.
satsuki:
Could you share your trained model ? i'd like to test it with FFVII.
Thanks
Ze_PilOt:
--- Quote from: satsuki on 2019-01-09 09:13:06 ---Could you share your trained model ? i'd like to test it with FFVII.
Thanks
--- End quote ---
Yes, I will share it with the automation scripts.
Navigation
[0] Message Index
[#] Next page
Go to full version