AI Neural Networks being used to generate HQ textures for older games

[OmniStrife]

Hey guys, I've stumbled upon a thread in Resetera where a new method of improving old textures is discussed,
The results are amazing!



Apparently anyone can use the method.

Here's the thread title:

Credit to Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, Wenzhe Shi/Arxiv on their paper about Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. I suggest you give it a read if you're interested in this kind of thing.

https://arxiv.org/pdf/1609.04802.pdf

Credit to Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Chen Change Loy, Yu Qiao, Xiaoou Tang for their paper on ESRGAN and for providing the ESRGAN repo with pretrained models

https://arxiv.org/pdf/1809.00219.pdf
https://github.com/xinntao/ESRGAN

Guide if ya wanna try it yourself:
Credit to kingdomakrillic their amazing work!

https://kingdomakrillic.tumblr.com/post/178254875891/i-figured-out-how-to-get-esrgan-and-sftgan

(also you don't need an NVIDIA card for this just go into test.py and change “device = torch.device(‘cuda’)“ to “ device = torch.device(‘cpu’)”. No AMD/Intel though)
(if you want to try this, make sure you have the image at its native resolution!)
(for the best results, don't use compressed strawberries off the internet)

NVIDIA has their own Generative Adverserial Network but you have to sign up to use it as it is still in beta: https://developer.nvidia.com/gwmt

Please read through the whole thread! I won't be updating this OP with newer images and instead'll be posting newer stuff within the thread! There's some cool stuff below!

Explanation:

Long story short, Enhanced Super Resolution Generative Adverserial Network, or ESRGAN, is an upscaling method that is capable of generating realistic textures during single image super-resolution. Basically it's a machine learning technique that uses a generative adverserial network to upres smaller images. By doing it over several passes, it will usually produce an image with more fidelity than methods such as SRCNN and SRGAN. In fact, ESRGAN is based off SRGAN. The difference between the two is that ESRGAN improves on SRGAN's network architecture, adversarial loss and perceptual loss. Furthermore ESRGAN

source (with more examples): https://www.resetera.com/threads/ai-neural-networks-being-used-to-generate-hq-textures-for-older-games-you-can-do-it-yourself.88272/

[Fraggoso]

I'm working on something for FFIX with esrgan.

[Ausar]

Farg I want to thank you for all of your hard work. I can’t wait to see what you have in store for FFIX. It’s a game that holds a special place in my heart and helped my though a very rough time in my adolescence.

I’m sure you are ask frequently, but any idea when we might see your ESRGAN work come to fruition?

[satsuki]

I'm working on something for FFIX with esrgan.

Same here for FFVII (filters done, ingame optimisation and layer bug solving WIP)

[Fraggoso]

Farg I want to thank you for all of your hard work. I can’t wait to see what you have in store for FFIX. It’s a game that holds a special place in my heart and helped my though a very rough time in my adolescence.

I’m sure you are ask frequently, but any idea when we might see your ESRGAN work come to fruition?

I'm doing a bit more than just a simple mod.
I'm trying to get all the assets bundled with a nice how to so everyone can just jump into making their own Background mod for FFIX. That way no one needs to deal with exporting them and sorting them properly.
In between I'm making the mod as well so it'll take some time but not as long as my first mod as I now know all the details to all scene and were I had problems before.

@Satsuki Do you know if there's a way to configure ESRGAN so that subfolders are read as well as added to the results folder?
I've added an issue to the repo https://github.com/DmitryUlyanov/deep-image-prior/issues/43 but maybe I just didn't read and it's already possible?

That way mods would be more easy to incorporate mods with their 100 or even 1.000 subfolders.

[satsuki]

Don't know if there's a way.
Just do a batch maker witch create and modify the "test_img_folder" and the "'results/{:s}_rlt.png'" in test.py:
1)Generate one "test00x.py" for each subfolder
2)make a .bat file to launch each "test00x.py" one after one
That's what i've done

[Fraggoso]

I'll take this as option B.

Thanks a ton!

[blippyp]

@Satsuki Do you know if there's a way to configure ESRGAN so that subfolders are read as well as added to the results folder?
I've added an issue to the repo https://github.com/DmitryUlyanov/deep-image-prior/issues/43 but maybe I just didn't read and it's already possible?

I altered mine to do that, I modified this a bit, so take it with a grain of salt, but the beef and potatoes are there if you want to take a look:

Code: [Select]

import sys
import os.path
import glob
import cv2
import numpy as np
import torch
import architecture as arch
from subprocess import check_call
import fnmatch

model_path = sys.argv[1]
src_path= sys.argv[2]

if model_path == "models/RRDB_ESRGAN_x4.pth":
modelType = "ESRGAN"
elif model_path == "models/RRDB_PSNR_x4.pth":
modelType = "PSNR"
elif model_path == "models/Manga109Attempt.pth":
modelType = "MANGA"
elif model_path == "models/Interp_08.pth":
modelType = "INTERP"
else:
modelType = "UNKNOWN"

startingPath="results/"+modelType+"/"
workingPath="results/"+modelType+"/"

device = torch.device('cuda')  # if you want to run on CPU, change 'cuda' -> cpu
#device = torch.device('cpu')

test_img_folder = "LR/*"
test_img_folder=src_path

model = arch.RRDB_Net(3, 3, 64, 23, gc=32, upscale=4, norm_type=None, act_type='leakyrelu', \
                        mode='CNA', res_scale=1, upsample_mode='upconv')
model.load_state_dict(torch.load(model_path), strict=True)
model.eval()
for k, v in model.named_parameters():
    v.requires_grad = False
model = model.to(device)

print('Model path {:s}. \n'.format(model_path))

idx = 0

for root, dirnames, filenames in os.walk(test_img_folder):
for filename in fnmatch.filter(filenames, '*.png'):

dstPath="H:\\GUM\\esrgan\\results\\"+modelType+"\\"
dstDir=root.replace(src_path, dstPath)

path=root+"\\"+filename
dstFile=dstDir+"\\"+filename

if not os.path.isfile(dstFile):
if not os.path.exists(dstDir):
os.makedirs(dstDir)

idx += 1
print(root+"\\"+filename)
base = os.path.basename(filename)
img = cv2.imread(path, cv2.IMREAD_COLOR)
img = img * 1.0 / 255
img = torch.from_numpy(np.transpose(img[:, :, [2, 1, 0]], (2, 0, 1))).float()
img_LR = img.unsqueeze(0)
img_LR = img_LR.to(device)

output = model(img_LR).data.squeeze().float().cpu().clamp_(0, 1).numpy()
output = np.transpose(output[[2, 1, 0], :, :], (1, 2, 0))
output = (output * 255.0).round()
cv2.imwrite(dstDir+"\\"+filename, output)


[Fraggoso]

First of all, thanks.

If I use your code I receive this error message:

Code: [Select]

Traceback (most recent call last):
  File "test.py", line 12, in <module>
    src_path= sys.argv[2]
IndexError: list index out of range

[blippyp]

your new script wants two arguments, the parameters for it is:
1) which model you want to use
2) the folder you want to start recursively start making images from

so for example:

mybatch.bat models/RRDB_ESRGAN_x4.pth H:\GUM\fieldsXCF

with this command it would use the rrdb_esrgan model and begin generating from images in anything in the H:\GUM\fieldsxcf folder (recursively), so if there's a folder in H:\GUM\fieldsxcf\testImages, then it will also generate images from that folder as well.

basically, if using the original files, just modify your batch file and put the path for your images in it after the model name, the line will look something similar to this:

python test.py models/RRDB_ESRGAN_x4.pth H:\GUM\fieldsXCF

[Fraggoso]

Hm... it doesn't work on my end. I can see that you've a mybatch.bat beforehand.
Sorry, I'm really not a coder and need more guidance *lol*.

I've altered the test.py with your code and run it in command line with this code:

models/Manga109Attempt.pth D:\ESRGAN\LR
If I run it, it wants to open the Manga109Attempt.pth

I've two subfolders for test in D:\ESRGAN\LR.

[blippyp]

post your batch file in code blocks and your py file in code blocks so I can see them

[blippyp]

oh sorry, I'm in a bit of a hurry so kind of quickly read that (getting daughter ready for school also, so if I disappear, just give me a 30-40 mins and I'll be back).

edit your batch file - put that path at the end of the python line that calls the model

[Fraggoso]

I don't have any batch files that's why I asked for a solution, sorry. ^_^

[blippyp]

ESRGAN comes with a batch file that looks something like this:

Code: [Select]

@echo off
echo Upscaling images in H:\GUM\fieldsXCF folder
python test.py models/RRDB_ESRGAN_x4.pth H:\GUM\fieldsXCF
echo Images can be located in Results folder
pause

so instead (from the dos prompt) you can just type something like:

Code: [Select]

python test.py models/RRDB_ESRGAN_x4.pth H:\GUM\fieldsXCF

just replace the H:\GUM\fieldsXCF with whatever folder you want to generate files from

I have to go now and take my daughter to school, if you have any further issues, just post your test.py file and your batch file if you can find it and I'll reply when I get back.

[Fraggoso]

I've send you a PM.

[blippyp]

Hello,

I apologize, I would of written back sooner, but when I got back my internet had dropped and it only just came back ;-P

I tried to send you a PM back, but it says you've blocked PM's, whatever that means. I guess it's an option in you settings for the forum.

I already gave you the information for the batch file tbh, all it does is activate that python command.

Send me your test.py script, and then I can tell you exactly what to type to activate it. if you copied what I wrote exactly, then all you should have to type is:

Code: [Select]

python test.py models/RRDB_ESRGAN_x4.pth H:\GUM\fieldsXCF

replace models/RRDB_ESRGAN_x4.pth with whichever ESRGAN model you wish to use.
replace H:\GUM\fieldsXCF with whatever folder you wish to have ESRGAN generate files from.

But send me your python script in your next message and then we'll know for sure.
Also, what operating system are you using? Windows/Linux/Mac??
If you're not on windows then make sure that all \ slashes are changed to / slashes.

[Fraggoso]

Wahooo that worked!
I had to modify your code a bit in regards to your results folder but otherwise it's awesome!

Can you try sending me another message? I didn't really block anything but I unchecked "allow user to send me mails" I can't find anything to blocking PM's. :/

[blippyp]

Wahooo that worked!
I had to modify your code a bit in regards to your results folder but otherwise it's awesome!

Can you try sending me another message? I didn't really block anything but I unchecked "allow user to send me mails" I can't find anything to blocking PM's. :/

Glad to hear it's working for you now
Unfortunately I'm still blocked for some reason. Error is:

User 'Fraggoso' has blocked your personal message.

So whatever causes that I have no idea, I've sent PM's to people on here before without any issues, so I'm guessing it's something in your settings.

[Fraggoso]

Now it'l work.
I've seen that I had admin and buddys only.

Thanks again for your help. That'll help extremely!