Author Topic: [FF7PC] Remako HD Field Mod using AI neural networks (WIP)  (Read 52882 times)

CaptRobau

  • Fast newbie
  • *
  • Posts: 32
  • Karma: 7
    • View Profile
What is it?

The Final Fantasy VII Remako HD Field Mod is a project I’ve been working on for a bit. Major strides have been made in making FF7 look much better. Yet, the field backgrounds have always remained behind. Traditional upscaling such as Waifu2x leaves a lot to be desired and manual remodelling such as Team Avalanche takes a lot of time.

I’ve been looking into machine learning to do the upscaling and landed on a great program called Gigapixel. Initial results were very good and after some more tweaking with the generation algorithm I ended on a good setup for upscaling FF7’s low-res pre-rendered field backgrounds by 400%.

After some in-game testing (boy, can FF7 be a chore to set up for modding) I’m now working on a mod that uses these AI-enhanced upscaled images. Among other things, this means lots of nights of leaving the program on to do its magic. I expect an initial release to be possible within a week or two.

Screenshots

Below is a comparison of the opening scene with vanilla at the top and Remako at the bottom. As you can see it’s a marked improvement and at times it looks like it’s a rerendered image.



Some other examples of the 400% upscaled field backgrounds (these shots are not in-game):







Let me know what you think!

Covarr

  • Covarr-Let
  • Administrator
  • No life
  • *
  • Posts: 4049
  • Karma: 121
  • Just Covarr. No "n".
    • View Profile
Very nice! I've seen efforts made with other NN-based tools, such as Let's Enhance or ESRGAN, but this really seems to blow them out of the water.

DLPB

  • No life
  • *
  • Posts: 9630
  • Karma: 320
  • For I realized that God's a young man, too.
    • View Profile
Yup. This is a winner.  Normal upscales suck in comparison. This is a game changer.
« Last Edit: 2019-01-03 23:50:28 by DLPB »

Tsunamix

  • Freak
  • *
  • Posts: 767
  • Karma: 16
  • Check out my work - https://goo.gl/m4NFlm
    • View Profile
    • The home of Tsunamods
Got my vote, this is brilliant

satsuki

  • Insane poster
  • *
  • Posts: 299
  • Karma: 47
    • View Profile
    • My softs (french only)
Seems like a manga109 esrgan tweaked... again someone ripped others works (found about 4 "AI upscale neurotal... uspcale" and all was a ripp of srgan or esrgan)
Here's my current method's results (use only ersgan + waifu for some prefilter).









Hard to tell witch is better, could you post full 4x uscaled and not smaller preview ?

FatedCourage

  • Insane poster
  • *
  • Posts: 334
  • Karma: 54
  • I'll never regret knowing you.
    • View Profile
    • Youtube Channel
I'm very impressed by this as well. It does an excellent job with FF7's fields. At least from what you posted, CaptRobau. I even tried this method on some of the FF8 fields and it seems to give some pretty great results. Great find and I hope to see this completed. :)

CaptRobau

  • Fast newbie
  • *
  • Posts: 32
  • Karma: 7
    • View Profile
Thank you for the kind words, everyone.

@satsuki

Yeah, machine learning aided upscaling has exploded within the last year or so. Before you had to do with something like Waifu2x, but then you got SRGAN, ESRGAN, Let's Enhance, Gigapixel. This stuff is evolving fast. Anything we do this year, will probably be blown out of the water by the methods of 2020.

Anyway, here are some of them at full resolution.

https://imgur.com/a/Xfyk8gs

Comparing the 7th Heaven pics, I notice that your version has more noise in the lighting glares. However, your Texas sign to the right looks crisper. Both look great.

While I have you here, maybe you (or anyone) else could explain why I'm getting a bugged out battle HUD:



I feel like there's a setting/installation problemen in 7th Heaven/the Game Converter. It hasn't got anything to do with my mod, as it happens even if I remove all the new field textures. Any help would be much appreciated.



DLPB

  • No life
  • *
  • Posts: 9630
  • Karma: 320
  • For I realized that God's a young man, too.
    • View Profile
One other thing - may take a long time, but it's possible to extract each frame of fmvs from psx game to images and run this upscale on them. Though probably would need to be specific to video AI or inconsistent from frame to frame.  That would fix the FMV good also. Possible?

Covarr

  • Covarr-Let
  • Administrator
  • No life
  • *
  • Posts: 4049
  • Karma: 121
  • Just Covarr. No "n".
    • View Profile
My concern with using this kind of NN-based upscaling on FMVs is how it would handle compression artifacts. The PS1 videos are better than the PC videos, but even they are pretty heavily and visibly compressed. If Gigapixel were to misinterpret those artifacts as intentional detail, the end result could well be worse, not better.

It could make a good plate for compositing, though, for those static FMVs such as junair_u, to upscale only the parts of the screen that don't move/change, and then a different scaling method (such as that already used in the Steam release) for the moving bits. Could make the transition between background and FMV less jarring, at least, if only the moving part of the screen gets all blurry and smudgy rather than the whole thing.

DLPB

  • No life
  • *
  • Posts: 9630
  • Karma: 320
  • For I realized that God's a young man, too.
    • View Profile
At 320 * 224 the compression artifacts are low...  It's not too bad at all.  It depends on how this "AI" reconstructs it.  But it would likely need to be taking into account multiple frames rather than one to be any use.

satsuki

  • Insane poster
  • *
  • Posts: 299
  • Karma: 47
    • View Profile
    • My softs (french only)
Re: [FF7PC] Remako HD Field Mod using AI neural networks (WIP)
« Reply #10 on: 2019-01-04 17:52:23 »
One other thing - may take a long time, but it's possible to extract each frame of fmvs from psx game to images and run this upscale on them. Though probably would need to be specific to video AI or inconsistent from frame to frame.  That would fix the FMV good also. Possible?
Alredy tried that but the psx's ones have too much temporal noise modulation so the video look "jumpy".
The best resul i got was to take the steam videos, downscale 4x (bicubic) then esrgan them, no more aliasing as in the steam original version, good stability and more details than the psx ones.

DLPB

  • No life
  • *
  • Posts: 9630
  • Karma: 320
  • For I realized that God's a young man, too.
    • View Profile
Re: [FF7PC] Remako HD Field Mod using AI neural networks (WIP)
« Reply #11 on: 2019-01-05 03:58:31 »
Did you try psx extracted using jpsxdec?  That one has highest decode quality of all.

The steam ones are upscales of psx.

satsuki

  • Insane poster
  • *
  • Posts: 299
  • Karma: 47
    • View Profile
    • My softs (french only)
Re: [FF7PC] Remako HD Field Mod using AI neural networks (WIP)
« Reply #12 on: 2019-01-05 07:27:53 »
i'll try again, just need to found my olds psx disc ^^

satsuki

  • Insane poster
  • *
  • Posts: 299
  • Karma: 47
    • View Profile
    • My softs (french only)
Re: [FF7PC] Remako HD Field Mod using AI neural networks (WIP)
« Reply #13 on: 2019-01-05 09:27:32 »
Here's an esrgan opening from psx (low compression level for testing) : http://wowsatsuki.free.fr/opentest.mkv
As you can see when aerith appears it's more detailled than the steam version, but you'll see also that there's lots of temporal noise when midgar show up (as in le original psx one) witch needs to be heavily filtered.

CaptRobau

  • Fast newbie
  • *
  • Posts: 32
  • Karma: 7
    • View Profile
Re: [FF7PC] Remako HD Field Mod using AI neural networks (WIP)
« Reply #14 on: 2019-01-05 12:08:04 »
Looks good satsuki.

Anyway, found out the reason behind the bug I had. In 7th Heaven I had the game's exe set as a ff7_bc.exe. It should have been ff7.exe.



About Remako, the mod's coming along well. I'm about half way with converting everything to game readable files. Palmer crapped out on me half way, otherwise I would've been much closer to completion.

Working on this mod, I've started to notice the varying levels of quality of the original's design. The Mako Reactor levels look really good upscaled. Fort Condor's interior is much less pretty. But if you look at the original files for those, you notice that the Mako Reactor had a good design even in it's low-res form, while Fort Condor's interior was a muddled, ugly mess to start with.
« Last Edit: 2019-01-05 12:12:24 by CaptRobau »

kennybones

  • Fast newbie
  • *
  • Posts: 30
  • Karma: 2
    • View Profile
Re: [FF7PC] Remako HD Field Mod using AI neural networks (WIP)
« Reply #15 on: 2019-01-05 13:08:09 »
Have you tried denoising the frames using waifu, and then upscale the thing with ESRGAN?

satsuki

  • Insane poster
  • *
  • Posts: 299
  • Karma: 47
    • View Profile
    • My softs (french only)
Re: [FF7PC] Remako HD Field Mod using AI neural networks (WIP)
« Reply #16 on: 2019-01-05 13:14:42 »
Yes but waifu is a noise denoiser (don't do a great job in this case) , to stabilize the video you need to use a temporal noise remover/stabiliser and it's killing lots of details.

DLPB

  • No life
  • *
  • Posts: 9630
  • Karma: 320
  • For I realized that God's a young man, too.
    • View Profile
Re: [FF7PC] Remako HD Field Mod using AI neural networks (WIP)
« Reply #17 on: 2019-01-05 17:52:25 »
I still have a project to upscale all FMV - as well as using the higher res 640 * 448 renders that were released (but they have a lot of noise and colour issues from conversion to dvd mpeg2 - which I've fixed). 

Because of this, I already have some fully extracted highest quality rips.  Here is one to try:
https://drive.google.com/file/d/11lGFyIspe9mXXZPJGy2nIJr7GyjM4_eu/view?usp=sharing

It also has my render (from project Goblins Bar).


Let's see how you guys go with that.
« Last Edit: 2019-01-05 17:55:19 by DLPB »

satsuki

  • Insane poster
  • *
  • Posts: 299
  • Karma: 47
    • View Profile
    • My softs (french only)
Re: [FF7PC] Remako HD Field Mod using AI neural networks (WIP)
« Reply #18 on: 2019-01-05 20:14:04 »
Here's my best (quick and dirty) result : http://wowsatsuki.free.fr/test.w2.25bcb.mg02.mkv
To my mind it's looking better than your version but there's not a huge gap (mine does better with aliasing and don't have the strange "sharp" effect your's have)

DLPB

  • No life
  • *
  • Posts: 9630
  • Karma: 320
  • For I realized that God's a young man, too.
    • View Profile
Re: [FF7PC] Remako HD Field Mod using AI neural networks (WIP)
« Reply #19 on: 2019-01-05 20:29:35 »
I think you're right.  My sharpness is a little more than needed. And that's led to one or two areas looking better - but some serious distortion on straight lines.

What did you use for this?

satsuki

  • Insane poster
  • *
  • Posts: 299
  • Karma: 47
    • View Profile
    • My softs (french only)
Re: [FF7PC] Remako HD Field Mod using AI neural networks (WIP)
« Reply #20 on: 2019-01-05 20:48:05 »
I use almost the same than for the esrgan background pack of ff7 i'm making:
1)Use waifu for noise reduction only (for this video i use noise level 2 with an old waifu build : "--scale_ratio 1 -m noise_scale --noise_level 2", i can upload you the specif exe if you can't get this result with modern waifu builds)
2)make a 25% downscale with imagemagik ("-filter box -resize 25%")
3)Upscale with esrgan (model used is an interpolation of Manga109 and RRDB_ESRGAN, ratio 80%Manga/20%ESRGAN)

I know that the "downscale then upscale again" process is a mathematic aberation but if you read the esrgan paper, they train the models with a set of high quality picture they donwscale with the bicubic algorythm (about the same as imagemagik box filter).
So the best way to get the full power of esrgan is to provide bicubic downscaled picture ^^
I tried looooots of methods with ff7 background and videos, the "noise reduction, downscale, upscale again" was always the best (off cource for the ff7 background, as a high resolution doesn't exist, i use waifu for noise reduction + upscale, before downscaling with imagemagic, and it's working great)
« Last Edit: 2019-01-05 20:49:51 by satsuki »

DLPB

  • No life
  • *
  • Posts: 9630
  • Karma: 320
  • For I realized that God's a young man, too.
    • View Profile
Re: [FF7PC] Remako HD Field Mod using AI neural networks (WIP)
« Reply #21 on: 2019-01-05 22:50:44 »
Might be worth looking at NeatVideo plugin for noise, too.
https://www.neatvideo.com/

Granfalloon

  • Fast newbie
  • *
  • Posts: 17
  • Karma: 0
    • View Profile
Re: [FF7PC] Remako HD Field Mod using AI neural networks (WIP)
« Reply #22 on: 2019-01-06 14:06:05 »
Super excited for this, the results are at the level where it's an actual genuine upgrade. Kinda mind blowing.
Can't wait to try it out!

Covarr

  • Covarr-Let
  • Administrator
  • No life
  • *
  • Posts: 4049
  • Karma: 121
  • Just Covarr. No "n".
    • View Profile
Re: [FF7PC] Remako HD Field Mod using AI neural networks (WIP)
« Reply #23 on: 2019-01-07 15:47:09 »
So, this has gotten some attention from PC Gamer.

Just a heads up to anybody who was linked here from PC Gamer:

You have not missed a download link anywhere; this mod is not completed or released yet.
« Last Edit: 2019-01-07 15:59:16 by Covarr »

satsuki

  • Insane poster
  • *
  • Posts: 299
  • Karma: 47
    • View Profile
    • My softs (french only)
Re: [FF7PC] Remako HD Field Mod using AI neural networks (WIP)
« Reply #24 on: 2019-01-07 18:12:17 »
Working on this mod, I've started to notice the varying levels of quality of the original's design. The Mako Reactor levels look really good upscaled. Fort Condor's interior is much less pretty. But if you look at the original files for those, you notice that the Mako Reactor had a good design even in it's low-res form, while Fort Condor's interior was a muddled, ugly mess to start with.
It's mainly because of the palleted colors use, if it's needed lots of colors the original render is realy bad ingame (most of the blin fields are not easy to process too because of alisased lines , mainly blin1/2/3)
That's why i use 2 main methods for each field then choose the best one ingame, and if it's realy not good looking a try to found specifics settings (right now i used a total of 5 differents setup).
By the way if you use my 4xcut software to autocut the resized fields, test thems ingame because since the release i found somes bugs and lots of optimisation to do on doors or lights and some fieds must be corrected by hand like the blin1, game1, and game2 because of a very specific layering.

here's my render of 2 field hard to properly upcale, colne2 and blin1: