Author Topic: [FF7PC]FF7 SYW V3, Battle, world map and fields with ERSGAN (WIP)  (Read 38144 times)

satsuki

  • *
  • Posts: 782
    • View Profile
    • My softs (french only)
Hello.
As you may have seen, i started some time ago an upscaling of FF7 using a mix of method around ESRGAN.
I posted in others post for helping and sharing but as i dont want to pollute theirs threads i start my own.

So far the filtering is complete but i'm doing a full game play to found bugs and optimise layering cut (doors, animations...) and try to find better upscaling/filtering option for screen witch are not good enouth.

Here's some ingame screens (some render better than others, as ff7 as not a consistant quality on all screen, but i tryed to find the best settings):























kennybones

  • *
  • Posts: 29
    • View Profile
Jesus Christ!! Amazing work! Holy shit, we're gonna have a renaissance with background upscaling mods for FF7, 8 and 9! My body is ready!

BahamutSIN

  • *
  • Posts: 64
    • View Profile
Awesome job. This makes the world of FFVII look alive again. :) And I can see details that I din't even notice before which is great.

Granfalloon

  • *
  • Posts: 24
    • View Profile
There is some things that could benefit from a touch up here and there but overall it is seriously stunning. This is looking like an actual remaster which is blowing my mind, super exciting stuff!

satsuki

  • *
  • Posts: 782
    • View Profile
    • My softs (french only)
There is some things that could benefit from a touch up here and there but overall it is seriously stunning. This is looking like an actual remaster which is blowing my mind, super exciting stuff!
Of course but the process is mainly automatic.
It already took me lots of time to make propers cut of layers, find the good filterings.... if i need to correct each screen to optimse it with gimp, it's take me about 15/30minutes per screen so about 200 hours to 300 hours again, + ingame tests... would be to long for the real game gain, on still picture default are clearly visible, not realy while playing game.

oraclex

  • *
  • Posts: 28
    • View Profile
very nice! :)

kennybones

  • *
  • Posts: 29
    • View Profile
I really hope you guys who are working on upscaling projects for FF7-8 and 9 are cooperating with your techniques and findings. The user who's upscaling FF9 backgrounds is using his own data training model, trained specifically for FF9 and artwork which is similar. And the results are muuuch better than ESRGAN with Manga dataset.

AuthenticM

  • *
  • Posts: 148
    • View Profile
This technology is revolutionary. Good lord.

The future is here.

satsuki

  • *
  • Posts: 782
    • View Profile
    • My softs (french only)
I really hope you guys who are working on upscaling projects for FF7-8 and 9 are cooperating with your techniques and findings. The user who's upscaling FF9 backgrounds is using his own data training model, trained specifically for FF9 and artwork which is similar. And the results are muuuch better than ESRGAN with Manga dataset.
I'll try is model as soon as he's release it.
But ff7 is realy different from ff9 because of the color palette is realy restrained in ff7, so ff7 quality base is lower than ff9 (and the quality is not the same on all ff7 bakground too...).
I can't train my own model because there's not realy much artwork of ff7 to train a model with.
I don't use manga109 only, i use it with a mix of esrgan, and some waifu prefilter, and layering up to 4 different filters method on a single background to alway use the best render possible.
The filter is a big part of the works but the layer cutting is another big part of the job, that's why i take long time (started this project about 3 month ago)

Kaldarasha

  • *
  • Posts: 2449
  • Prince of Model Editing
    • View Profile
Can't you use the current remade high Res art from team avalanche jusete and jmp to train the tool? Btw the PSX has a different colour output. As far as I know the colour variation is higher in the middle while it's lower in the black and white colour space. We have a post shader (ported by sunwalker) which does correct this and the colours do blend much better together.

satsuki

  • *
  • Posts: 782
    • View Profile
    • My softs (french only)
Can't you use the current remade high Res art from team avalanche jusete and jmp to train the tool? Btw the PSX has a different colour output. As far as I know the colour variation is higher in the middle while it's lower in the black and white colour space. We have a post shader (ported by sunwalker) which does correct this and the colours do blend much better together.
that's the first thing i tried but the results where realy bad T_T

kennybones

  • *
  • Posts: 29
    • View Profile
But ff7 is realy different from ff9 because of the color palette is realy restrained in ff7, so ff7 quality base is lower than ff9 (and the quality is not the same on all ff7 bakground too...).
Sounds like you're really meticulous in your work, I like it! I've noticed FF8 has these color palette issues as well, my guess is that both FF7 and 8 have now bit color palettes, compared to FF9. But these are still looking as good as it gets!

satsuki

  • *
  • Posts: 782
    • View Profile
    • My softs (french only)
Can't you use the current remade high Res art from team avalanche jusete and jmp to train the tool? Btw the PSX has a different colour output. As far as I know the colour variation is higher in the middle while it's lower in the black and white colour space. We have a post shader (ported by sunwalker) which does correct this and the colours do blend much better together.
Maybe i can try another method.
I'll try to train a model HR art as high quality picture and instead of simply do a 25% reduction for the low res picture, i'll try ot use a color reducted + 25% reduction, maybe more logic as ff7 pictures are resolution and colors restrained.
Got some issue to train the model, but i'll try as soon as possible.

Covarr

  • Covarr-Let
  • Administrator
  • *
  • Posts: 3941
  • Just Covarr. No "n".
    • View Profile
Can't you use the current remade high Res art from team avalanche jusete and jmp to train the tool?
Good training data for a tool like this is data that is quite similar to the data you'll be scaling. The reason the FF9 stuff works so well is because the high-res images are literally the source images for the backgrounds. As such, ESRGAN can understand exactly how those images were scaled, complete with dithering, limited palettes, etc., and thus better understand how to undo that.

Team Avalanche backgrounds, though gorgeous, are often fairly heavily reimagined. Things that might seem innocuous artistically make a big difference on the technical side of things. A different wood texture here, a different cloth material there, reworked lighting... If you downscale those, you don't get the original in-game backgrounds. And ESRGAN doesn't truly understand concepts like "wood", "cloth", etc. It is looking at frequent patterns, color variations, etc. And those would be substantially different on the TA stuff.

So you could use TA backgrounds for both the high- and low-res training images and create potentially okay but not really helpful training data with similar problems to the Manga109 training (albeit in the opposite direction), or you could use the TA backgrounds for high-res and the original in-game backgrounds for low-res training images, and get awful useless training materials that aren't grounded in how scaling actually works. Neither is really desirable.

satsuki

  • *
  • Posts: 782
    • View Profile
    • My softs (french only)
Good training data for a tool like this is data that is quite similar to the data you'll be scaling. The reason the FF9 stuff works so well is because the high-res images are literally the source images for the backgrounds. As such, ESRGAN can understand exactly how those images were scaled, complete with dithering, limited palettes, etc., and thus better understand how to undo that.

Team Avalanche backgrounds, though gorgeous, are often fairly heavily reimagined. Things that might seem innocuous artistically make a big difference on the technical side of things. A different wood texture here, a different cloth material there, reworked lighting... If you downscale those, you don't get the original in-game backgrounds. And ESRGAN doesn't truly understand concepts like "wood", "cloth", etc. It is looking at frequent patterns, color variations, etc. And those would be substantially different on the TA stuff.

So you could use TA backgrounds for both the high- and low-res training images and create potentially okay but not really helpful training data with similar problems to the Manga109 training (albeit in the opposite direction), or you could use the TA backgrounds for high-res and the original in-game backgrounds for low-res training images, and get awful useless training materials that aren't grounded in how scaling actually works. Neither is really desirable.
Of course this is exact, that's why my first trained model was no use, but i'll try with a reducted color pallette model training as most of the upscale problems with ff7 is due to reducted colors in pictures.
Don't think it would be good.... but you know, why not test it ^^

AlphaAtlas

  • *
  • Posts: 9
    • View Profile
Figured I'd post an update on that vapoursynth video/batch image tool here.

Though I'm not doing much original work in the first place, I'm almost done. I've made picture tutorials for installing CUDA and cuDNN from Nvidia's website, I've made a .bat file that installs and checks MXNet, I've made a .bat that downloads or updates the Super Resolution Zoo repo from Github, and I've made a python file that grabs all the neural net info files.

I just need to write some code in that python file that parses the NN info files and automatically inserts the correct syntax into a Vapoursynth script. Basically, without the script, you'd have to manually enter the path and correct arguments for the super resolution model you want, like:
Code: [Select]
sr_args = dict(model_filename=r'D:\CustomPrograms\VapourSynth64Portable\Models\Super-Resolution-Zoo\MSRN\MSRN_4x', device_id=0, up_scale=4, is_rgb_model=True, pad=None, crop=None, pre_upscale=False)Which is tedious if you're trying to decide between 30+ different (pretrained) models. Anyway, I have plenty of time to finish that this weekend.
« Last Edit: 2019-01-12 11:01:45 by AlphaAtlas »

satsuki

  • *
  • Posts: 782
    • View Profile
    • My softs (french only)
Model training on its way (thanks to Ze_PilOt who help me understand why my computer wasn't processed the model).
Will take about 5 to 10 days to train it (as i'll stop it from time to time to process ff7 fields with my current method).
Let's wait and see ...

blippyp

  • *
  • Posts: 100
    • View Profile
Model training on its way (thanks to Ze_PilOt who help me understand why my computer wasn't processed the model).
Will take about 5 to 10 days to train it (as i'll stop it from time to time to process ff7 fields with my current method).
Let's wait and see ...

neato - good luck! :)

How do you go about doing something like that anyway? Do you have any good resources to link me to? I've been digging into neural networking ever since I heard about ESRGAN myself, and actually just finished coding my very own extremely basic neural network for the first time in python within the last hour (the very basics, not even using any libraries, but I understand most of it for the most part at least). But I have no idea how to even begin training a model for specific images like you're doing now and would love to know how. I'm more interested in how this can be used in different ways tbh, like adding this kind of code to a game bot (npc, whatever) but would love to play with the graphics end as well.
« Last Edit: 2019-01-12 16:23:13 by blippyp »

Fraggoso

  • *
  • Posts: 278
    • View Profile
It's really great to see so much sharing and carrying in the community.

blippyp

  • *
  • Posts: 100
    • View Profile
neato - good luck! :)

How do you go about doing something like that anyway?

I think I found it, do you use BasicSR?

satsuki

  • *
  • Posts: 782
    • View Profile
    • My softs (french only)
Yes it's with basicSR.
Some usefull informations
1)don't install nor use lmdb, it's a pain to install with python (on my windows at least) and it takes lots of ressources (and even in my classic no ssd drive, there's no benefits)
2)edit and use the extract_subimgs_single.py script for HQ set generation from your HQ sources
3)generate the LQ set with the imagemagik command : "convert.exe -filter box -resize 25%% #input# #output#"
4)Be sure all your HQ and LQ picture use the same color depth and setting (for that i use the imagemagik command :  "convert.exe #inptut# -define png:format=png24 #inpt#" , it ensure that all the png are compressed the same way and it's auto overwriting the source).
5)Make a set of verification of about 10% of the total pictures (for LQ anb HQ)
6)edit  the train_ESRGAN.json with your folders and to avoid a out of memory error :
      , "n_workers": 4
      , "batch_size": 8
I also set a "save_checkpoint_freq": 5000 to be able to stop/resume the training easily

for reference here's my edited python files (all the other files are as in the git without modification and i use python 3.6 and not anaconda):

extract_subimgs_single.py
Code: [Select]
import os
import os.path
import sys
from multiprocessing import Pool
import numpy as np
import cv2
sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from utils.progress_bar import ProgressBar


def main():
    """A multi-thread tool to crop sub imags."""
    input_folder = "E:/ESRGANModel/ff7sources/"
    save_folder = "E:/ESRGANModel/dataset/ff7hq/"
    n_thread = 20
    crop_sz = 480
    step = 240
    thres_sz = 48
    compression_level = 0  # 3 is the default value in cv2
    # CV_IMWRITE_PNG_COMPRESSION from 0 to 9. A higher value means a smaller size and longer
    # compression time. If read raw images during training, use 0 for faster IO speed.

    if not os.path.exists(save_folder):
        os.makedirs(save_folder)
        print('mkdir [{:s}] ...'.format(save_folder))
    else:
        print('Folder [{:s}] already exists. Exit...'.format(save_folder))
        sys.exit(1)

    img_list = []
    for root, _, file_list in sorted(os.walk(input_folder)):
        path = [os.path.join(root, x) for x in file_list]  # assume only images in the input_folder
        img_list.extend(path)

    def update(arg):
        pbar.update(arg)

    pbar = ProgressBar(len(img_list))

    pool = Pool(n_thread)
    for path in img_list:
        pool.apply_async(worker,
            args=(path, save_folder, crop_sz, step, thres_sz, compression_level),
            callback=update)
    pool.close()
    pool.join()
    print('All subprocesses done.')


def worker(path, save_folder, crop_sz, step, thres_sz, compression_level):
    img_name = os.path.basename(path)
    img = cv2.imread(path, cv2.IMREAD_UNCHANGED)

    n_channels = len(img.shape)
    if n_channels == 2:
        h, w = img.shape
    elif n_channels == 3:
        h, w, c = img.shape
    else:
        raise ValueError('Wrong image shape - {}'.format(n_channels))

    h_space = np.arange(0, h - crop_sz + 1, step)
    if h - (h_space[-1] + crop_sz) > thres_sz:
        h_space = np.append(h_space, h - crop_sz)
    w_space = np.arange(0, w - crop_sz + 1, step)
    if w - (w_space[-1] + crop_sz) > thres_sz:
        w_space = np.append(w_space, w - crop_sz)

    index = 0
    for x in h_space:
        for y in w_space:
            index += 1
            if n_channels == 2:
                crop_img = img[x:x + crop_sz, y:y + crop_sz]
            else:
                crop_img = img[x:x + crop_sz, y:y + crop_sz, :]
            crop_img = np.ascontiguousarray(crop_img)
            # var = np.var(crop_img / 255)
            # if var > 0.008:
            #     print(img_name, index_str, var)
            cv2.imwrite(
                os.path.join(save_folder, img_name.replace('.png', '_s{:03d}.png'.format(index))),
                crop_img, [cv2.IMWRITE_PNG_COMPRESSION, compression_level])
    return 'Processing {:s} ...'.format(img_name)


if __name__ == '__main__':
    main()

train_ESRGAN.json
Code: [Select]
{
  "name": "RRDB_ESRGAN_x4_FF7" //  please remove "debug_" during training
  , "use_tb_logger": true
  , "model":"srragan"
  , "scale": 4
  , "gpu_ids": [0]

  , "datasets": {
    "train": {
      "name": "FF7"
      , "mode": "LRHR"
      , "dataroot_HR": "E:/ESRGANModel/Dataset/ff7hq"
      , "dataroot_LR": "E:/ESRGANModel/Dataset/ff7lq"
      , "subset_file": null
      , "use_shuffle": true
      , "n_workers": 4
      , "batch_size": 8
      , "HR_size": 128
      , "use_flip": true
      , "use_rot": true
    }
   , "val": {
      "name": "val_set14_part"
      , "mode": "LRHR"
      , "dataroot_HR": "E:/ESRGANModel/Dataset/ff7hqval"
      , "dataroot_LR": "E:/ESRGANModel/Dataset/ff7lqval"
   }
  }

  , "path": {
    "root": "E:/ESRGANModel/InWorks"
   // , "resume_state": "E:/ESRGANModel/InWorks/experiments/RRDB_ESRGAN_x4_FF7/training_state/15000.state"  //<=edit and uncomment to resume training
    , "pretrain_model_G": "../experiments/pretrained_models/RRDB_PSNR_x4.pth"
  }

  , "network_G": {
    "which_model_G": "RRDB_net" // RRDB_net | sr_resnet
    , "norm_type": null
    , "mode": "CNA"
    , "nf": 64
    , "nb": 23
    , "in_nc": 3
    , "out_nc": 3
    , "gc": 32
    , "group": 1
  }
  , "network_D": {
    "which_model_D": "discriminator_vgg_128"
    , "norm_type": "batch"
    , "act_type": "leakyrelu"
    , "mode": "CNA"
    , "nf": 64
    , "in_nc": 3
  }

  , "train": {
    "lr_G": 1e-4
    , "weight_decay_G": 0
    , "beta1_G": 0.9
    , "lr_D": 1e-4
    , "weight_decay_D": 0
    , "beta1_D": 0.9
    , "lr_scheme": "MultiStepLR"
    , "lr_steps": [50000, 100000, 200000, 300000]
    , "lr_gamma": 0.5

    , "pixel_criterion": "l1"
    , "pixel_weight": 1e-2
    , "feature_criterion": "l1"
    , "feature_weight": 1
    , "gan_type": "vanilla"
    , "gan_weight": 5e-3

    //for wgan-gp
    // , "D_update_ratio": 1
    // , "D_init_iters": 0
    // , "gp_weigth": 10

    , "manual_seed": 0
    , "niter": 5e5
    , "val_freq": 5e3
  }

  , "logger": {
    "print_freq": 200
    , "save_checkpoint_freq": 5000 //5e3
  }
}

satsuki

  • *
  • Posts: 782
    • View Profile
    • My softs (french only)
Figured I'd post an update on that vapoursynth video/batch image tool here.

Though I'm not doing much original work in the first place, I'm almost done. I've made picture tutorials for installing CUDA and cuDNN from Nvidia's website, I've made a .bat file that installs and checks MXNet, I've made a .bat that downloads or updates the Super Resolution Zoo repo from Github, and I've made a python file that grabs all the neural net info files.

I just need to write some code in that python file that parses the NN info files and automatically inserts the correct syntax into a Vapoursynth script. Basically, without the script, you'd have to manually enter the path and correct arguments for the super resolution model you want, like:
Code: [Select]
sr_args = dict(model_filename=r'D:\CustomPrograms\VapourSynth64Portable\Models\Super-Resolution-Zoo\MSRN\MSRN_4x', device_id=0, up_scale=4, is_rgb_model=True, pad=None, crop=None, pre_upscale=False)Which is tedious if you're trying to decide between 30+ different (pretrained) models. Anyway, I have plenty of time to finish that this weekend.

Thanks for your works, i think i'll have some fun with this ^^

blippyp

  • *
  • Posts: 100
    • View Profile
Yes it's with basicSR.

Awesome, thx for the tips, I'm sure they'll come in handy when I get a chance to try it out :)

blippyp

  • *
  • Posts: 100
    • View Profile
Model training on its way...

Alright, so I installed BasicSR and ran it thru a test, despite your urging and considering that I do have a SSD drive, I still used the lmdb however ;-P

Also, I couldn't use your recommended (or the default) n_workers or batch_size settings. It bugged out on me, according to the posted issues on the github, this is a known issue on windows with torch, someone recommended that I set them to 0 and +1 total images respectively, and then it just ran thru them without hassle. I also had some weird issue with the progress_bar.py files as well. Where it checks for fps (line 41'ish) about it trying to divide by zero, so I simply altered the code there to check if the elapsed variable was 0 and if it was to skip the next line and instead set the fps to 0 by default.

I swiped thru just over 1.3K images in like 10 minutes, it was super fast actually (of course my 'high res' images were very small). I basically used the original backgrounds from FFVIII as my high-res images, so they were weren't exactly high res to begin with. But even still, if it only takes four times that long to swipe thru images four times bigger, that's still really fast. I'm thinking I did something wrong, but it spit out a model for me and I used it and it worked fine (although it sucked).

Was just curious what your strategy was for creating your own model. Are you using your current 'processes' upscaled images you created for FFVII as your high res images? Or are you using an entirely different set of images to generate your model?  Or did you do like I did with my first test, and use the original files as the high res?

I'm curious if applying this to a bunch of masks would also be helpful...??? What do you think?

This was neat though, not sure how I'll use this, but glad I have it to play with now :)
« Last Edit: 2019-01-13 00:30:19 by blippyp »

satsuki

  • *
  • Posts: 782
    • View Profile
    • My softs (french only)
-progress_bar.py
>I removed it from the code, as the lmdb woring fast, no progresse needed for my tests ^^ (and i don't use it at end so...)

-I swiped thru just over 1.3K images in like 10 minutes
>I think you do something wrong, i own a gtx1060, the process uses about 90% of gpu (gpu is the bottleneck in basicSR) and it take about 3 minutes for 200 iter (and the learning process need to do 500 000 iter, so about 5.2days 24h/24h)

-the original backgrounds from FFVIII as my high-res images
>I used the team avalanche/jmp/jusete 3d art background as HQ picture, than downscale them and make a color reduction to get LQ picture (color reduction is a test as ff7 use paletted color pictures)

-I'm curious if applying this to a bunch of masks would also be helpful
>I don't think it'll helping anything, the best bet is that you found and use HQ ff8 art as close as possible as game picture (not sure my approach with 3d art will be usefull, it'a just an idea)