Qhimm.com Forums
Final Fantasy 7 => Graphics => WIP => Topic started by: CaptRobau on 2019-01-03 21:46:44
-
What is it?
The Final Fantasy VII Remako HD Field Mod is a project I’ve been working on for a bit. Major strides have been made in making FF7 look much better. Yet, the field backgrounds have always remained behind. Traditional upscaling such as Waifu2x leaves a lot to be desired and manual remodelling such as Team Avalanche takes a lot of time.
I’ve been looking into machine learning to do the upscaling and landed on a great program called Gigapixel. Initial results were very good and after some more tweaking with the generation algorithm I ended on a good setup for upscaling FF7’s low-res pre-rendered field backgrounds by 400%.
After some in-game testing (boy, can FF7 be a chore to set up for modding) I’m now working on a mod that uses these AI-enhanced upscaled images. Among other things, this means lots of nights of leaving the program on to do its magic. I expect an initial release to be possible within a week or two.
Screenshots
Below is a comparison of the opening scene with vanilla at the top and Remako at the bottom. As you can see it’s a marked improvement and at times it looks like it’s a rerendered image.
(https://i.imgur.com/MmxxkAk.jpg)
Some other examples of the 400% upscaled field backgrounds (these shots are not in-game):
(https://i.imgur.com/MytO3g1.jpg)
(https://i.imgur.com/R7Q5cSd.jpg)
(https://i.imgur.com/B6BcBpk.jpg)
Let me know what you think!
-
Very nice! I've seen efforts made with other NN-based tools, such as Let's Enhance or ESRGAN, but this really seems to blow them out of the water.
-
Got my vote, this is brilliant
-
Seems like a manga109 esrgan tweaked... again someone ripped others works (found about 4 "AI upscale neurotal... uspcale" and all was a ripp of srgan or esrgan)
Here's my current method's results (use only ersgan + waifu for some prefilter).
(http://yatoshicom.free.fr/ff7p/md1stin_0_base_0000516.jpg)
(http://yatoshicom.free.fr/ff7p/md8_2_0_base_0000516.jpg)
(http://yatoshicom.free.fr/ff7p/mds7pb_1_0_base_0000516.jpg)
(http://yatoshicom.free.fr/ff7p/uutai1_0_base_0000130.jpg)
Hard to tell witch is better, could you post full 4x uscaled and not smaller preview ?
-
I'm very impressed by this as well. It does an excellent job with FF7's fields. At least from what you posted, CaptRobau. I even tried this method on some of the FF8 fields and it seems to give some pretty great results. Great find and I hope to see this completed. :)
-
Thank you for the kind words, everyone.
@satsuki
Yeah, machine learning aided upscaling has exploded within the last year or so. Before you had to do with something like Waifu2x, but then you got SRGAN, ESRGAN, Let's Enhance, Gigapixel. This stuff is evolving fast. Anything we do this year, will probably be blown out of the water by the methods of 2020.
Anyway, here are some of them at full resolution.
https://imgur.com/a/Xfyk8gs
Comparing the 7th Heaven pics, I notice that your version has more noise in the lighting glares. However, your Texas sign to the right looks crisper. Both look great.
While I have you here, maybe you (or anyone) else could explain why I'm getting a bugged out battle HUD:
(https://i.imgur.com/iQ1qAcs.png)
I feel like there's a setting/installation problemen in 7th Heaven/the Game Converter. It hasn't got anything to do with my mod, as it happens even if I remove all the new field textures. Any help would be much appreciated.
-
My concern with using this kind of NN-based upscaling on FMVs is how it would handle compression artifacts. The PS1 videos are better than the PC videos, but even they are pretty heavily and visibly compressed. If Gigapixel were to misinterpret those artifacts as intentional detail, the end result could well be worse, not better.
It could make a good plate for compositing, though, for those static FMVs such as junair_u, to upscale only the parts of the screen that don't move/change, and then a different scaling method (such as that already used in the Steam release) for the moving bits. Could make the transition between background and FMV less jarring, at least, if only the moving part of the screen gets all blurry and smudgy rather than the whole thing.
-
One other thing - may take a long time, but it's possible to extract each frame of fmvs from psx game to images and run this upscale on them. Though probably would need to be specific to video AI or inconsistent from frame to frame. That would fix the FMV good also. Possible?
Alredy tried that but the psx's ones have too much temporal noise modulation so the video look "jumpy".
The best resul i got was to take the steam videos, downscale 4x (bicubic) then esrgan them, no more aliasing as in the steam original version, good stability and more details than the psx ones.
-
i'll try again, just need to found my olds psx disc ^^
-
Here's an esrgan opening from psx (low compression level for testing) : http://wowsatsuki.free.fr/opentest.mkv
As you can see when aerith appears it's more detailled than the steam version, but you'll see also that there's lots of temporal noise when midgar show up (as in le original psx one) witch needs to be heavily filtered.
-
Looks good satsuki.
Anyway, found out the reason behind the bug I had. In 7th Heaven I had the game's exe set as a ff7_bc.exe. It should have been ff7.exe.
(https://i.imgur.com/iQ1qAcs.png)
About Remako, the mod's coming along well. I'm about half way with converting everything to game readable files. Palmer crapped out on me half way, otherwise I would've been much closer to completion.
Working on this mod, I've started to notice the varying levels of quality of the original's design. The Mako Reactor levels look really good upscaled. Fort Condor's interior is much less pretty. But if you look at the original files for those, you notice that the Mako Reactor had a good design even in it's low-res form, while Fort Condor's interior was a muddled, ugly mess to start with.
-
Have you tried denoising the frames using waifu, and then upscale the thing with ESRGAN?
-
Yes but waifu is a noise denoiser (don't do a great job in this case) , to stabilize the video you need to use a temporal noise remover/stabiliser and it's killing lots of details.
-
Here's my best (quick and dirty) result : http://wowsatsuki.free.fr/test.w2.25bcb.mg02.mkv
To my mind it's looking better than your version but there's not a huge gap (mine does better with aliasing and don't have the strange "sharp" effect your's have)
-
I use almost the same than for the esrgan background pack of ff7 i'm making:
1)Use waifu for noise reduction only (for this video i use noise level 2 with an old waifu build : "--scale_ratio 1 -m noise_scale --noise_level 2", i can upload you the specif exe if you can't get this result with modern waifu builds)
2)make a 25% downscale with imagemagik ("-filter box -resize 25%")
3)Upscale with esrgan (model used is an interpolation of Manga109 and RRDB_ESRGAN, ratio 80%Manga/20%ESRGAN)
I know that the "downscale then upscale again" process is a mathematic aberation but if you read the esrgan paper, they train the models with a set of high quality picture they donwscale with the bicubic algorythm (about the same as imagemagik box filter).
So the best way to get the full power of esrgan is to provide bicubic downscaled picture ^^
I tried looooots of methods with ff7 background and videos, the "noise reduction, downscale, upscale again" was always the best (off cource for the ff7 background, as a high resolution doesn't exist, i use waifu for noise reduction + upscale, before downscaling with imagemagic, and it's working great)
-
Super excited for this, the results are at the level where it's an actual genuine upgrade. Kinda mind blowing.
Can't wait to try it out!
-
So, this has gotten some attention from PC Gamer. (https://www.pcgamer.com/ai-neural-networks-are-giving-final-fantasy-7-a-makeover/)
Just a heads up to anybody who was linked here from PC Gamer:
You have not missed a download link anywhere; this mod is not completed or released yet.
-
Working on this mod, I've started to notice the varying levels of quality of the original's design. The Mako Reactor levels look really good upscaled. Fort Condor's interior is much less pretty. But if you look at the original files for those, you notice that the Mako Reactor had a good design even in it's low-res form, while Fort Condor's interior was a muddled, ugly mess to start with.
It's mainly because of the palleted colors use, if it's needed lots of colors the original render is realy bad ingame (most of the blin fields are not easy to process too because of alisased lines , mainly blin1/2/3)
That's why i use 2 main methods for each field then choose the best one ingame, and if it's realy not good looking a try to found specifics settings (right now i used a total of 5 differents setup).
By the way if you use my 4xcut software to autocut the resized fields, test thems ingame because since the release i found somes bugs and lots of optimisation to do on doors or lights and some fieds must be corrected by hand like the blin1, game1, and game2 because of a very specific layering.
here's my render of 2 field hard to properly upcale, colne2 and blin1:
(http://yatoshicom.free.fr/ff7p/v3c.jpg)
(http://yatoshicom.free.fr/ff7p/v31c.jpg)
-
I use almost the same than for the esrgan background pack of ff7 i'm making:
1)Use waifu for noise reduction only (for this video i use noise level 2 with an old waifu build : "--scale_ratio 1 -m noise_scale --noise_level 2", i can upload you the specif exe if you can't get this result with modern waifu builds)
2)make a 25% downscale with imagemagik ("-filter box -resize 25%")
3)Upscale with esrgan (model used is an interpolation of Manga109 and RRDB_ESRGAN, ratio 80%Manga/20%ESRGAN)
I know that the "downscale then upscale again" process is a mathematic aberation but if you read the esrgan paper, they train the models with a set of high quality picture they donwscale with the bicubic algorythm (about the same as imagemagik box filter).
So the best way to get the full power of esrgan is to provide bicubic downscaled picture ^^
I tried looooots of methods with ff7 background and videos, the "noise reduction, downscale, upscale again" was always the best (off cource for the ff7 background, as a high resolution doesn't exist, i use waifu for noise reduction + upscale, before downscaling with imagemagic, and it's working great)
Waifu only looks at individual frames because it was programmed as an image upscaler. What you need a high quality temporal denoiser, right? I highly recommend V-BM3D or KNLMeansCL, as both are way better than waifu at temporal noise/grain like that.
You can actually do everything (artifact removal, denoise, upscaling, resize back down) in a single step with vapoursynth, and do intermediate steps like 2x or 3x... But that's a deeep rabbit hole to go down.
-
I kown but i'm stuck with old avisynth, never take the time to go to vapoursynth.
I've done lots of video filtering long time ago (started with the first release of virtualdub to end with avisynth with its great features).
But honestly the hard part of this type of upscale is that esrgan (and other ia of the same type) is not designed to upscale cleaned up video picture but low resolution still picture.
With the original psx videos of ff7 you have to deal with low resolution (320*xxx) + temporal noise + low colors palette compression... not realy easy to work with ^^'
Right now i don't have much time to spend in it because i'm already do a test/correction off the fields of the game.. maybe when my pack will be done ^^
-
I kown but i'm stuck with old avisynth, never take the time to go to vapoursynth.
I've done lots of video filtering long time ago (started with the first release of virtualdub to end with avisynth with its great features).
But honestly the hard part of this type of upscale is that esrgan (and other ia of the same type) is not designed to upscale cleaned up video picture but low resolution still picture.
With the original psx videos of ff7 you have to deal with low resolution (320*xxx) + temporal noise + low colors palette compression... not realy easy to work with ^^'
Right now i don't have much time to spend in it because i'm already do a test/correction off the fields of the game.. maybe when my pack will be done ^^
Yeah. I've tested a few algos, and ESGRAN DOES NOT like being fed dirty frames. It seems to amplify halos, noise and such even more than other AI upscalers.
As for the difficulty of Vapoursynth... This week, I'm working on packaging the Vapoursynth Fatpack and this amazing repo in one easy package: https://github.com/WolframRhodium/Super-Resolution-Zoo
So, theoretically, you'd download a big zip file, unzip some separate Nvidia (or Intel) files I'm not allowed to redistribute, run a .bat file to install everything, then open a custom vapoursynth script in VSEDIT, and you'd be able try dozens of different AI upscalers, temporal GPU denoisers, artifact removers and such and such by just changing a number in a text file. The setup could batch process basically any image or video file under the sun, no conversion needed, and previewing/tweaking would be fast.
-
This is amazing.
OP, do you have Patreon or something to donate to for your time?
-
Yeah. I've tested a few algos, and ESGRAN DOES NOT like being fed dirty frames. It seems to amplify halos, noise and such even more than other AI upscalers.
As for the difficulty of Vapoursynth... This week, I'm working on packaging the Vapoursynth Fatpack and this amazing repo in one easy package: https://github.com/WolframRhodium/Super-Resolution-Zoo
So, theoretically, you'd download a big zip file, unzip some separate Nvidia (or Intel) files I'm not allowed to redistribute, run a .bat file to install everything, then open a custom vapoursynth script in VSEDIT, and you'd be able try dozens of different AI upscalers, temporal GPU denoisers, artifact removers and such and such by just changing a number in a text file. The setup could batch process basically any image or video file under the sun, no conversion needed, and previewing/tweaking would be fast.
Cool !
i'll wait for it ^^
-
Cool !
i'll wait for it ^^
Do you happen to have an Nvidia GPU?
The upscalers will work on a CPU, but they're slow. AFAIK, only Waifu2X has a version that's accelerated by AMD GPUs atm.
-
Yes gtx1060
-
Iam working on a Crono Kross remake using the same metods, but the game only exist for Ps1
Anyone could help me extractintg the raw textures of the game and put again in the iso file.
I will post what i already Made
-
Iam working on a Crono Kross remake using the same metods, but the game only exist for Ps1
Anyone could help me extractintg the raw textures of the game and put again in the iso file.
I will post what i already Made
Like wise as per psx titles.
Id like to do this for Legend of Dragoon.
-
Hey,
I just want to say that this is legit amazing. I was planning on playing FFVII with the release of R06, but I think I'm going to wait for this. Any estimation on when it would be complete?
Keep up the great work and Happy New Year!
-
Mine is complete right now (fields, combats, worldmap), but i'im doing a full game trough to find bug and optimise filtering (because ingame rendering is not alway the same than picture rendering depending how much the game zoom the picture).
It's a long process to do but it'll ensure the best result a can do.
So maybe one or two months.
I'll also wait the AlphaAtlas tool to check i can do a better job on videos, can take some time too.
-
A dutch webpage send be to this topic (long time member here) first time I have see them metioning a mod! It more a page for hardware and other technical related newbits
Well done.
guess also tnx is in order to you Satsuki. Been useing your work for quite some time. Can't play without it anymore.
-
A dutch webpage send be to this topic (long time member here) first time I have see them metioning a mod! It more a page for hardware and other technical related newbits
Well done.
Yeah, someone linked this thread on /r/pcgaming, which PC Gamer is notorious for watching, then other tech media saw it on PC Gamer and reported on it.
-
Mine is complete right now (fields, combats, worldmap), but i'im doing a full game trough to find bug and optimise filtering (because ingame rendering is not alway the same than picture rendering depending how much the game zoom the picture).
It's a long process to do but it'll ensure the best result a can do.
So maybe one or two months.
I'll also wait the AlphaAtlas tool to check i can do a better job on videos, can take some time too.
Pleaase post some screenshots of your play through in that time if you can, I'm dying to see more backgrounds in-game.
-
Can you start doing more frequent posts about this project? I think more people would like to get involved with it. It's been in the news. Do you need any type of help?
-
Can you start doing more frequent posts about this project? I think more people would like to get involved with it. It's been in the news. Do you need any type of help?
I made my own topic here : http://forums.qhimm.com/index.php?topic=18556.0
-
this is REALLY good news.. i hope this ends well and completed.. MORE POWER TO MODDERS AND TESTERS! :)
-
i can olny say one about that. Perfect!
-
This is amazing.
OP, do you have Patreon or something to donate to for your time?
i can olny say one about that. Perfect!
this is REALLY good news.. i hope this ends well and completed.. MORE POWER TO MODDERS AND TESTERS! :)
I'm glad you all like it.
Yeah, someone linked this thread on /r/pcgaming, which PC Gamer is notorious for watching, then other tech media saw it on PC Gamer and reported on it.
I hadn't seen that. That's awesome! Being featured like that really helps push me past the finish line.
Can you start doing more frequent posts about this project? I think more people would like to get involved with it. It's been in the news. Do you need any type of help?
There is a lot of work involved. Doing updates all the time isn't helpful.
Sorry for not responding in a while. I've been working on it in the background and went on a short trip this past week. I'm putting the finishing touches on it. I expect the beta release to arrive soon. I can understand you want to see more stuff, but so close to finish line I'm just going to focus on getting it out there. I'm sure you'd rather play it an evening sooner than later.
-
I hadn't seen that. That's awesome! Being featured like that really helps push me past the finish line.
And it is also a thing on Steam now.
(https://i.imgur.com/7Z3COm3.png)
-
You've misread my post... I am saying doing updates ISN'T helpful - to YOU. I was responding to the other person.
I know. Just wanted to respond to both, that what you suggest is what I'm doing. Sorry for the confusion.
And it is also a thing on Steam now.
(https://i.imgur.com/7Z3COm3.png)
Very cool! :mrgreen:
-
Aside from rounding up the development work, I've also been working on my own little Blogger site to showcase the mod when it's released. Qhimm is great and everything, but forums just aren't a very casual-friendly way to show off a mod. I want to use the release thread (once it's been made) more for bug reports, technical discussions, updates, etc.
So while I said I wasn't going to do a big update here, as I wanted to work on the mod (and the accompanying webpage) I just realized that I needed to do a bunch of screenshots anyway for my Blogger site. So if you go to the link below and click on screenshots, you will be able to see some more in-game screenshots and renders. Aside from these, there will be no new media until release. It's just actual development work now.
https://captrobau.blogspot.com/
-
This mod now has its first release. Here is the release thread: http://forums.qhimm.com/index.php?topic=18573.0
I think it's smart if all further discussion of this mod is done there. Thanks for all the good words so far. I hope you like it too when you actually get to play it.
-
amazing work.
are you going to make a version for bootleg?
-
Locking thread since release thread is open.