Qhimm.com Forums
Final Fantasy 7 => Graphics => WIP => Topic started by: satsuki on 2019-01-10 09:18:46
-
Hello.
As you may have seen, i started some time ago an upscaling of FF7 using a mix of method around ESRGAN.
I posted in others post for helping and sharing but as i dont want to pollute theirs threads i start my own.
So far the filtering is complete but i'm doing a full game play to found bugs and optimise layering cut (doors, animations...) and try to find better upscaling/filtering option for screen witch are not good enouth.
Here's some ingame screens (some render better than others, as ff7 as not a consistant quality on all screen, but i tryed to find the best settings):
(http://yatoshicom.free.fr/v3/ff7sywv3001.jpg)
(http://yatoshicom.free.fr/v3/ff7sywv3002.jpg)
(http://yatoshicom.free.fr/v3/ff7sywv3003.jpg)
(http://yatoshicom.free.fr/v3/ff7sywv3004.jpg)
(http://yatoshicom.free.fr/v3/ff7sywv3005.jpg)
(http://yatoshicom.free.fr/v3/ff7sywv3006.jpg)
(http://yatoshicom.free.fr/v3/ff7sywv3007.jpg)
(http://yatoshicom.free.fr/v3/ff7sywv3008.jpg)
(http://yatoshicom.free.fr/v3/ff7sywv3009.jpg)
(http://yatoshicom.free.fr/v3/ff7sywv3010.jpg)
(http://yatoshicom.free.fr/v3/ff7sywv3011.jpg)
(http://yatoshicom.free.fr/v3/ff7sywv3012.jpg)
-
Jesus Christ!! Amazing work! Holy shit, we're gonna have a renaissance with background upscaling mods for FF7, 8 and 9! My body is ready!
-
Awesome job. This makes the world of FFVII look alive again. :) And I can see details that I din't even notice before which is great.
-
There is some things that could benefit from a touch up here and there but overall it is seriously stunning. This is looking like an actual remaster which is blowing my mind, super exciting stuff!
-
There is some things that could benefit from a touch up here and there but overall it is seriously stunning. This is looking like an actual remaster which is blowing my mind, super exciting stuff!
Of course but the process is mainly automatic.
It already took me lots of time to make propers cut of layers, find the good filterings.... if i need to correct each screen to optimse it with gimp, it's take me about 15/30minutes per screen so about 200 hours to 300 hours again, + ingame tests... would be to long for the real game gain, on still picture default are clearly visible, not realy while playing game.
-
very nice! :)
-
I really hope you guys who are working on upscaling projects for FF7-8 and 9 are cooperating with your techniques and findings. The user who's upscaling FF9 backgrounds is using his own data training model, trained specifically for FF9 and artwork which is similar. And the results are muuuch better than ESRGAN with Manga dataset.
-
This technology is revolutionary. Good lord.
The future is here.
-
I really hope you guys who are working on upscaling projects for FF7-8 and 9 are cooperating with your techniques and findings. The user who's upscaling FF9 backgrounds is using his own data training model, trained specifically for FF9 and artwork which is similar. And the results are muuuch better than ESRGAN with Manga dataset.
I'll try is model as soon as he's release it.
But ff7 is realy different from ff9 because of the color palette is realy restrained in ff7, so ff7 quality base is lower than ff9 (and the quality is not the same on all ff7 bakground too...).
I can't train my own model because there's not realy much artwork of ff7 to train a model with.
I don't use manga109 only, i use it with a mix of esrgan, and some waifu prefilter, and layering up to 4 different filters method on a single background to alway use the best render possible.
The filter is a big part of the works but the layer cutting is another big part of the job, that's why i take long time (started this project about 3 month ago)
-
Can't you use the current remade high Res art from team avalanche jusete and jmp to train the tool? Btw the PSX has a different colour output. As far as I know the colour variation is higher in the middle while it's lower in the black and white colour space. We have a post shader (ported by sunwalker) which does correct this and the colours do blend much better together.
-
Can't you use the current remade high Res art from team avalanche jusete and jmp to train the tool? Btw the PSX has a different colour output. As far as I know the colour variation is higher in the middle while it's lower in the black and white colour space. We have a post shader (ported by sunwalker) which does correct this and the colours do blend much better together.
that's the first thing i tried but the results where realy bad T_T
-
But ff7 is realy different from ff9 because of the color palette is realy restrained in ff7, so ff7 quality base is lower than ff9 (and the quality is not the same on all ff7 bakground too...).
Sounds like you're really meticulous in your work, I like it! I've noticed FF8 has these color palette issues as well, my guess is that both FF7 and 8 have now bit color palettes, compared to FF9. But these are still looking as good as it gets!
-
Can't you use the current remade high Res art from team avalanche jusete and jmp to train the tool? Btw the PSX has a different colour output. As far as I know the colour variation is higher in the middle while it's lower in the black and white colour space. We have a post shader (ported by sunwalker) which does correct this and the colours do blend much better together.
Maybe i can try another method.
I'll try to train a model HR art as high quality picture and instead of simply do a 25% reduction for the low res picture, i'll try ot use a color reducted + 25% reduction, maybe more logic as ff7 pictures are resolution and colors restrained.
Got some issue to train the model, but i'll try as soon as possible.
-
Can't you use the current remade high Res art from team avalanche jusete and jmp to train the tool?
Good training data for a tool like this is data that is quite similar to the data you'll be scaling. The reason the FF9 stuff works so well is because the high-res images are literally the source images for the backgrounds. As such, ESRGAN can understand exactly how those images were scaled, complete with dithering, limited palettes, etc., and thus better understand how to undo that.
Team Avalanche backgrounds, though gorgeous, are often fairly heavily reimagined. Things that might seem innocuous artistically make a big difference on the technical side of things. A different wood texture here, a different cloth material there, reworked lighting... If you downscale those, you don't get the original in-game backgrounds. And ESRGAN doesn't truly understand concepts like "wood", "cloth", etc. It is looking at frequent patterns, color variations, etc. And those would be substantially different on the TA stuff.
So you could use TA backgrounds for both the high- and low-res training images and create potentially okay but not really helpful training data with similar problems to the Manga109 training (albeit in the opposite direction), or you could use the TA backgrounds for high-res and the original in-game backgrounds for low-res training images, and get awful useless training materials that aren't grounded in how scaling actually works. Neither is really desirable.
-
Good training data for a tool like this is data that is quite similar to the data you'll be scaling. The reason the FF9 stuff works so well is because the high-res images are literally the source images for the backgrounds. As such, ESRGAN can understand exactly how those images were scaled, complete with dithering, limited palettes, etc., and thus better understand how to undo that.
Team Avalanche backgrounds, though gorgeous, are often fairly heavily reimagined. Things that might seem innocuous artistically make a big difference on the technical side of things. A different wood texture here, a different cloth material there, reworked lighting... If you downscale those, you don't get the original in-game backgrounds. And ESRGAN doesn't truly understand concepts like "wood", "cloth", etc. It is looking at frequent patterns, color variations, etc. And those would be substantially different on the TA stuff.
So you could use TA backgrounds for both the high- and low-res training images and create potentially okay but not really helpful training data with similar problems to the Manga109 training (albeit in the opposite direction), or you could use the TA backgrounds for high-res and the original in-game backgrounds for low-res training images, and get awful useless training materials that aren't grounded in how scaling actually works. Neither is really desirable.
Of course this is exact, that's why my first trained model was no use, but i'll try with a reducted color pallette model training as most of the upscale problems with ff7 is due to reducted colors in pictures.
Don't think it would be good.... but you know, why not test it ^^
-
Figured I'd post an update on that vapoursynth video/batch image tool here.
Though I'm not doing much original work in the first place, I'm almost done. I've made picture tutorials for installing CUDA and cuDNN from Nvidia's website, I've made a .bat file that installs and checks MXNet, I've made a .bat that downloads or updates the Super Resolution Zoo repo from Github, and I've made a python file that grabs all the neural net info files.
I just need to write some code in that python file that parses the NN info files and automatically inserts the correct syntax into a Vapoursynth script. Basically, without the script, you'd have to manually enter the path and correct arguments for the super resolution model you want, like:
sr_args = dict(model_filename=r'D:\CustomPrograms\VapourSynth64Portable\Models\Super-Resolution-Zoo\MSRN\MSRN_4x', device_id=0, up_scale=4, is_rgb_model=True, pad=None, crop=None, pre_upscale=False)Which is tedious if you're trying to decide between 30+ different (pretrained) models. Anyway, I have plenty of time to finish that this weekend.
-
Model training on its way (thanks to Ze_PilOt who help me understand why my computer wasn't processed the model).
Will take about 5 to 10 days to train it (as i'll stop it from time to time to process ff7 fields with my current method).
Let's wait and see ...
-
Model training on its way (thanks to Ze_PilOt who help me understand why my computer wasn't processed the model).
Will take about 5 to 10 days to train it (as i'll stop it from time to time to process ff7 fields with my current method).
Let's wait and see ...
neato - good luck! :)
How do you go about doing something like that anyway? Do you have any good resources to link me to? I've been digging into neural networking ever since I heard about ESRGAN myself, and actually just finished coding my very own extremely basic neural network for the first time in python within the last hour (the very basics, not even using any libraries, but I understand most of it for the most part at least). But I have no idea how to even begin training a model for specific images like you're doing now and would love to know how. I'm more interested in how this can be used in different ways tbh, like adding this kind of code to a game bot (npc, whatever) but would love to play with the graphics end as well.
-
It's really great to see so much sharing and carrying in the community.
-
neato - good luck! :)
How do you go about doing something like that anyway?
I think I found it, do you use BasicSR?
-
Yes it's with basicSR.
Some usefull informations
1)don't install nor use lmdb, it's a pain to install with python (on my windows at least) and it takes lots of ressources (and even in my classic no ssd drive, there's no benefits)
2)edit and use the extract_subimgs_single.py script for HQ set generation from your HQ sources
3)generate the LQ set with the imagemagik command : "convert.exe -filter box -resize 25%% #input# #output#"
4)Be sure all your HQ and LQ picture use the same color depth and setting (for that i use the imagemagik command : "convert.exe #inptut# -define png:format=png24 #inpt#" , it ensure that all the png are compressed the same way and it's auto overwriting the source).
5)Make a set of verification of about 10% of the total pictures (for LQ anb HQ)
6)edit the train_ESRGAN.json with your folders and to avoid a out of memory error :
, "n_workers": 4
, "batch_size": 8
I also set a "save_checkpoint_freq": 5000 to be able to stop/resume the training easily
for reference here's my edited python files (all the other files are as in the git without modification and i use python 3.6 and not anaconda):
extract_subimgs_single.py
import os
import os.path
import sys
from multiprocessing import Pool
import numpy as np
import cv2
sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from utils.progress_bar import ProgressBar
def main():
"""A multi-thread tool to crop sub imags."""
input_folder = "E:/ESRGANModel/ff7sources/"
save_folder = "E:/ESRGANModel/dataset/ff7hq/"
n_thread = 20
crop_sz = 480
step = 240
thres_sz = 48
compression_level = 0 # 3 is the default value in cv2
# CV_IMWRITE_PNG_COMPRESSION from 0 to 9. A higher value means a smaller size and longer
# compression time. If read raw images during training, use 0 for faster IO speed.
if not os.path.exists(save_folder):
os.makedirs(save_folder)
print('mkdir [{:s}] ...'.format(save_folder))
else:
print('Folder [{:s}] already exists. Exit...'.format(save_folder))
sys.exit(1)
img_list = []
for root, _, file_list in sorted(os.walk(input_folder)):
path = [os.path.join(root, x) for x in file_list] # assume only images in the input_folder
img_list.extend(path)
def update(arg):
pbar.update(arg)
pbar = ProgressBar(len(img_list))
pool = Pool(n_thread)
for path in img_list:
pool.apply_async(worker,
args=(path, save_folder, crop_sz, step, thres_sz, compression_level),
callback=update)
pool.close()
pool.join()
print('All subprocesses done.')
def worker(path, save_folder, crop_sz, step, thres_sz, compression_level):
img_name = os.path.basename(path)
img = cv2.imread(path, cv2.IMREAD_UNCHANGED)
n_channels = len(img.shape)
if n_channels == 2:
h, w = img.shape
elif n_channels == 3:
h, w, c = img.shape
else:
raise ValueError('Wrong image shape - {}'.format(n_channels))
h_space = np.arange(0, h - crop_sz + 1, step)
if h - (h_space[-1] + crop_sz) > thres_sz:
h_space = np.append(h_space, h - crop_sz)
w_space = np.arange(0, w - crop_sz + 1, step)
if w - (w_space[-1] + crop_sz) > thres_sz:
w_space = np.append(w_space, w - crop_sz)
index = 0
for x in h_space:
for y in w_space:
index += 1
if n_channels == 2:
crop_img = img[x:x + crop_sz, y:y + crop_sz]
else:
crop_img = img[x:x + crop_sz, y:y + crop_sz, :]
crop_img = np.ascontiguousarray(crop_img)
# var = np.var(crop_img / 255)
# if var > 0.008:
# print(img_name, index_str, var)
cv2.imwrite(
os.path.join(save_folder, img_name.replace('.png', '_s{:03d}.png'.format(index))),
crop_img, [cv2.IMWRITE_PNG_COMPRESSION, compression_level])
return 'Processing {:s} ...'.format(img_name)
if __name__ == '__main__':
main()
train_ESRGAN.json
{
"name": "RRDB_ESRGAN_x4_FF7" // please remove "debug_" during training
, "use_tb_logger": true
, "model":"srragan"
, "scale": 4
, "gpu_ids": [0]
, "datasets": {
"train": {
"name": "FF7"
, "mode": "LRHR"
, "dataroot_HR": "E:/ESRGANModel/Dataset/ff7hq"
, "dataroot_LR": "E:/ESRGANModel/Dataset/ff7lq"
, "subset_file": null
, "use_shuffle": true
, "n_workers": 4
, "batch_size": 8
, "HR_size": 128
, "use_flip": true
, "use_rot": true
}
, "val": {
"name": "val_set14_part"
, "mode": "LRHR"
, "dataroot_HR": "E:/ESRGANModel/Dataset/ff7hqval"
, "dataroot_LR": "E:/ESRGANModel/Dataset/ff7lqval"
}
}
, "path": {
"root": "E:/ESRGANModel/InWorks"
// , "resume_state": "E:/ESRGANModel/InWorks/experiments/RRDB_ESRGAN_x4_FF7/training_state/15000.state" //<=edit and uncomment to resume training
, "pretrain_model_G": "../experiments/pretrained_models/RRDB_PSNR_x4.pth"
}
, "network_G": {
"which_model_G": "RRDB_net" // RRDB_net | sr_resnet
, "norm_type": null
, "mode": "CNA"
, "nf": 64
, "nb": 23
, "in_nc": 3
, "out_nc": 3
, "gc": 32
, "group": 1
}
, "network_D": {
"which_model_D": "discriminator_vgg_128"
, "norm_type": "batch"
, "act_type": "leakyrelu"
, "mode": "CNA"
, "nf": 64
, "in_nc": 3
}
, "train": {
"lr_G": 1e-4
, "weight_decay_G": 0
, "beta1_G": 0.9
, "lr_D": 1e-4
, "weight_decay_D": 0
, "beta1_D": 0.9
, "lr_scheme": "MultiStepLR"
, "lr_steps": [50000, 100000, 200000, 300000]
, "lr_gamma": 0.5
, "pixel_criterion": "l1"
, "pixel_weight": 1e-2
, "feature_criterion": "l1"
, "feature_weight": 1
, "gan_type": "vanilla"
, "gan_weight": 5e-3
//for wgan-gp
// , "D_update_ratio": 1
// , "D_init_iters": 0
// , "gp_weigth": 10
, "manual_seed": 0
, "niter": 5e5
, "val_freq": 5e3
}
, "logger": {
"print_freq": 200
, "save_checkpoint_freq": 5000 //5e3
}
}
-
Figured I'd post an update on that vapoursynth video/batch image tool here.
Though I'm not doing much original work in the first place, I'm almost done. I've made picture tutorials for installing CUDA and cuDNN from Nvidia's website, I've made a .bat file that installs and checks MXNet, I've made a .bat that downloads or updates the Super Resolution Zoo repo from Github, and I've made a python file that grabs all the neural net info files.
I just need to write some code in that python file that parses the NN info files and automatically inserts the correct syntax into a Vapoursynth script. Basically, without the script, you'd have to manually enter the path and correct arguments for the super resolution model you want, like:
sr_args = dict(model_filename=r'D:\CustomPrograms\VapourSynth64Portable\Models\Super-Resolution-Zoo\MSRN\MSRN_4x', device_id=0, up_scale=4, is_rgb_model=True, pad=None, crop=None, pre_upscale=False)Which is tedious if you're trying to decide between 30+ different (pretrained) models. Anyway, I have plenty of time to finish that this weekend.
Thanks for your works, i think i'll have some fun with this ^^
-
Yes it's with basicSR.
Awesome, thx for the tips, I'm sure they'll come in handy when I get a chance to try it out :)
-
Model training on its way...
Alright, so I installed BasicSR and ran it thru a test, despite your urging and considering that I do have a SSD drive, I still used the lmdb however ;-P
Also, I couldn't use your recommended (or the default) n_workers or batch_size settings. It bugged out on me, according to the posted issues on the github, this is a known issue on windows with torch, someone recommended that I set them to 0 and +1 total images respectively, and then it just ran thru them without hassle. I also had some weird issue with the progress_bar.py files as well. Where it checks for fps (line 41'ish) about it trying to divide by zero, so I simply altered the code there to check if the elapsed variable was 0 and if it was to skip the next line and instead set the fps to 0 by default.
I swiped thru just over 1.3K images in like 10 minutes, it was super fast actually (of course my 'high res' images were very small). I basically used the original backgrounds from FFVIII as my high-res images, so they were weren't exactly high res to begin with. But even still, if it only takes four times that long to swipe thru images four times bigger, that's still really fast. I'm thinking I did something wrong, but it spit out a model for me and I used it and it worked fine (although it sucked).
Was just curious what your strategy was for creating your own model. Are you using your current 'processes' upscaled images you created for FFVII as your high res images? Or are you using an entirely different set of images to generate your model? Or did you do like I did with my first test, and use the original files as the high res?
I'm curious if applying this to a bunch of masks would also be helpful...??? What do you think?
This was neat though, not sure how I'll use this, but glad I have it to play with now :)
-
-progress_bar.py
>I removed it from the code, as the lmdb woring fast, no progresse needed for my tests ^^ (and i don't use it at end so...)
-I swiped thru just over 1.3K images in like 10 minutes
>I think you do something wrong, i own a gtx1060, the process uses about 90% of gpu (gpu is the bottleneck in basicSR) and it take about 3 minutes for 200 iter (and the learning process need to do 500 000 iter, so about 5.2days 24h/24h)
-the original backgrounds from FFVIII as my high-res images
>I used the team avalanche/jmp/jusete 3d art background as HQ picture, than downscale them and make a color reduction to get LQ picture (color reduction is a test as ff7 use paletted color pictures)
-I'm curious if applying this to a bunch of masks would also be helpful
>I don't think it'll helping anything, the best bet is that you found and use HQ ff8 art as close as possible as game picture (not sure my approach with 3d art will be usefull, it'a just an idea)
-
-I swiped thru just over 1.3K images in like 10 minutes
>I think you do something wrong, i own a gtx1060, the process uses about 90% of gpu (gpu is the bottleneck in basicSR) and it take about 3 minutes for 200 iter (and the learning process need to do 500 000 iter, so about 5.2days 24h/24h)
ya, I have the same card and I just compared the PSNR model with the one it produced for me and the results were identical actually, so I'm guessing it wasn't doing anything. I think this is related to the worker/batch arguments I modified to get it to 'work'. I'm guessing I'll have to try a linux install due to this torch issue in windows I suppose. What OS are you using?
-
-I swiped thru just over 1.3K images in like 10 minutes
>I think you do something wrong, i own a gtx1060, the process uses about 90% of gpu (gpu is the bottleneck in basicSR) and it take about 3 minutes for 200 iter (and the learning process need to do 500 000 iter, so about 5.2days 24h/24h)
As a side note, just for confirmation on how I did it. I put the 'original hi-res' images in the hqval folder, and my original downsized versions in the lqval folder. I then ran the extract_subimgs_single.py script on my hqval folder putting those files into the hr folder, and then downsized those and put them into my lr folder if that makes sense. I also ran that command you gave me on all the mages to makes sure they were all png 24 format.
-
That's not exactly the way it needed to be
1)Took your HQ pictures and run extract_subimgs_single.py on that folder
>Results in a ff8hq folder
2)Took the picture from ff8hq folder and make a 25% downsize
>Results in a ff8lq folder
3)Copy about 10%/15% of pictures from ff8hq folder
>Results in ff8hqval folder
4)Copy about 10%/15% of (the same than hq) pictures from ff8lq folder
>Results in ff8lqval folder
5)"Normalize" with imagemagik
edit your train_esrgan.json file with your folders:
, "datasets": {
"train": {
"name": "FF8"
, "mode": "LRHR"
, "dataroot_HR": "E:/ESRGANModel/Dataset/ff8hq"
, "dataroot_LR": "E:/ESRGANModel/Dataset/ff8lq"
, "subset_file": null
, "use_shuffle": true
, "n_workers": 4
, "batch_size": 8
, "HR_size": 128
, "use_flip": true
, "use_rot": true
}
, "val": {
"name": "val_set14_part"
, "mode": "LRHR"
, "dataroot_HR": "E:/ESRGANModel/Dataset/ff8hqval"
, "dataroot_LR": "E:/ESRGANModel/Dataset/ff8lqval"
}
launch the training ^^
As workers don't seems to work, try the values 1-2 or 2-4 instead of the original 8-16
My computer using windows 10 pro 1809 64 bits (corei7 8700k / 8go ram / gtx1060-6go)
here's my python config (default value without any mods, from ersgan use to basicSR):
1)install python 3.6.2 64bits
2)pip install scipy
3)pip install pillow
4)pip install numpy opencv-python
5)pip3 install http://download.pytorch.org/whl/cu90/torch-0.4.1-cp36-cp36m-win_amd64.whl
6)pip3 install torchvision
7)pip install numpy opencv-python
8)pip install tensorboardX
As i dont use lmdb i haven't put it here, but i installed it with a wheel file because of the well know vc++ 14 error (even if i installed the right buildtools)
-
okay, so if I'm understanding you correctly, I did exactly the same thing, except that I put all my files in the val folders (which I'm guessing would only make the process take longer in the end) and I didn't normalize my files, which again, I'm guessing would only have an effect on how my model ends up.
aside from that I didn't have scipy installed, which I just installed.
so I'm pretty confused as to why this isn't working for me, as was suggested in the notes on the github, I modified the worker/batch arguments and now the process seems to complete, but the model doesn't actually 'change', instead it just makes a copy of it from what I can tell and it completed the training super fast.
oh the joys of all of this... the best part is that if I figure this out, I can look forward to days of my computer doing nothing but crunching numbers just to end up with an even worse model than what I'm already using haha ;-P
-
The imagemagic normalization is only if you get a picture import error while runing training.
Have you tried the worker-batch to 1-2 or 2-4, i think the "0" worker you used make it bugs
"... the best part is that if I figure this out, I can look forward to days of my computer doing nothing but crunching numbers just to end up with an even worse model than what I'm already using"
That will probably be the same for my model training, but with a lot of luck maybe ....
-
Have you tried the worker-batch to 1-2 or 2-4, i think the "0" worker you used make it bugs
ya, anything above 0 makes it crash for some reason, we seem to have nearly identical systems, except that I'm using an i5, not that it should matter much though considering we're using our GPU's, which are identical. However I'm on Windows 7 and using Python 3.7.1. I'm pretty sure I ran into this issue before I even started using lmdb, so don't think it's related, but I'm not aware of the buildtools bug you were talking about which might be part of the problem I suppose. My guess is that the issue has more to do with either the OS differences or the Python version differences. I also just realized that I'm using PyTorch v1.0 not 0.4.1, but every attempt so far to downgrade has failed with a 'not supported wheel on this platform' message :'(
-
Okay, I'm a big dummy and should of listened to you to begin with! :P
I disabled lmdb, and off it went without a hitch using the variables as they should be. (so much wasted time, I even installed python 3.6.7, 3.6.2, Anaconda...) haha
definitely taking a lot longer now...
-
Great you made it works ^^
-
Thanks for your works, i think i'll have some fun with this ^^
I lied about having plenty of time to finish it this weekend ;D. This is a WIP, but a functional WIP:
https://github.com/AlphaAtlas/vs_mxnet_helper/
You can format the imagemagik input to take (and write) a sequence of images easily enough, but among many other things, I need to figure out a script that can work through a whole directory recursively.
-
You can format the imagemagik input to take (and write) a sequence of images easily enough, but among many other things, I need to figure out a script that can work through a whole directory recursively.
I didn't really look at it much, but noticed you were using python and batch scripting, recursion is pretty simple tbh:
BATCH FILE RECURSION
@echo off
for /r %%f in (*.*) do (
echo The File: %%f
echo The Directory: %%~pf
echo The Filename: %%~nf
echo The File Extension: %%~xf
)
PYTHON RECURSION
src="C:/Windows"
for root, dirnames, filenames in os.walk(src+"/"):
for filename in fnmatch.filter(filenames, '*.*'):
infile=root+filename
-
I didn't really look at it much, but noticed you were using python and batch scripting, recursion is pretty simple tbh:
BATCH FILE RECURSION
@echo off
for /r %%f in (*.*) do (
echo The File: %%f
echo The Directory: %%~pf
echo The Filename: %%~nf
echo The File Extension: %%~xf
)
PYTHON RECURSION
src="C:/Windows"
for root, dirnames, filenames in os.walk(src+"/"):
for filename in fnmatch.filter(filenames, '*.*'):
infile=root+filename
Thanks, I may use that. The real trick is feeding that data to the ImageMagick plugin, which I can't just stick in another for loop IIRC.
http://www.vapoursynth.com/doc/plugins/imwri.html
I think I can physically move/rename all the images to a single directory with that script, let Vapoursynth do its thing on the nicely formatted images, then move the processed images back using the saved directory data once the script is done.
-
it's been a while, but imagemagick has it's own way of dealing with multiple files (I believe they are sequential, ie: image000.png, image001.png, image002.png) so if your files are number as such they are easy to manipulate. If you're moving files around that don't have names like that, you'll have to generate the names for the files and then likely keep a record as well (image000.png=somefile.png, image001.png=anotherfile.png, etc...) then when you're done manipulating the files in sequence, rename them again.
-
Finally some breakthrough for the pre-rendered backgrounds. The other method would have been to remake em in a 3D software which can take lot of work.
-
Game checked to the end.
To do before release:
-Need to find better filtering for 12 fields.
-Need to do some world map and battle fields optimisation.
-Need to wait the final alpha test from azertyuiop2 (ff7.fr) to spot bug or optimisation i havent seen.
After the release:
-Check if i can do a good videos upscale method
-
This looks so good. Just wonder exactly what pc version of FFVII do I need to make this mod work?
-
Downgraded steam version or the 98 release with Aali's driver.
-
Downgraded steam version or the 98 release with Aali's driver.
Thank you.
-
Hi, I want to say, amazing job! Really wanted to play FFVII again before the remaster, it's my childhood favorite game!
Just got a few questions:
- How much time does it need for the first release?
- Do you need any beta tester?
Again thanks for your work, cheers! ;D
-
The first release will take some time (i found that on some TV or mis-calibrated computer screens, the black offset isn't right (#020301 or so instead of #000000)), i'm generating the whole background again then do the game again for test.
I also need to found better filtering for some screens.
Maybe one or two month, i don't want to speed up the process, i want to do a bug free release.
I don't need more beta tester right now but i'll tell you if i need.
-
I'm a little rusty on Avisynth filtering, but I'd wager the FMVs could look alright with some MVDegrain3, Gradfunkmirror and NNEDI2. Maybe with a light AddGrain to fake extra detail.
-
Sorry for being kinda afk. I'll clean up the vapoursynth assistant more soon, but it more or less works now.
I'm a little rusty on Avisynth filtering, but I'd wager the FMVs could look alright with some MVDegrain3, Gradfunkmirror and NNEDI2. Maybe with a light AddGrain to fake extra detail.
I've tried NNEDI3 and fast denoiser with a test video recently, and I can tell you that neural network upscaling + KNLMeansCL or V-BM3D for temporal denoising looks WAY better. And it shouldn't be too slow with a good GPU.
Also, when you say NNEDI2... are the source clips interlaced, or did you just mean NNEDI2 image doubling?
-
Thanks AlphaAtlas, take your time, i'm doing the game again to correct the black offset bug i have on some screens, so i won't process the videos ATM
-
Is it possible to get just a few more gameplay screenshots? :P
-
here's some more ingame :
(http://yatoshicom.free.fr/ff7p/ScreenShot001.jpg)
(http://yatoshicom.free.fr/ff7p/ScreenShot002.jpg)
(http://yatoshicom.free.fr/ff7p/ScreenShot003.jpg)
(http://yatoshicom.free.fr/ff7p/ScreenShot004.jpg)
(http://yatoshicom.free.fr/ff7p/ScreenShot005.jpg)
(http://yatoshicom.free.fr/ff7p/ScreenShot006.jpg)
(http://yatoshicom.free.fr/ff7p/ScreenShot007.jpg)
-
Thank you. Well ... I have not seen FFVII ever look better than that thats for sure. I am hoping for a release soon :)
-
As soon as possible.
As i said i don't want to release a bugged or insconstant upscale from screen to screen.
As ff7 background quality isn't constant from one screen to another in original game, the upscaled version need to be tweaked as much as possible to get a great render.
So i've done the game twice to try to find the better filtering option for each screen, i tried lots of waifu/esrgan/imagmegick combo filtering (from no filtrering at all, just esrgan, to a combinaison of about 5 filtering renders layered).
There's no more big bug ATM, and the background pack is well optimised.
But i have a great alpha tester how's doing the game right know and have found little fix and optimisation i correct while is testing (some black pixel (not missed a lot myself but he found some)) a mis-animated door ...
I'll take the time needed, but when a release will be avaible, it won't be a beta.
-
Thats correct of you. Not wanting to release a glitchy version.
Its a shame they did not save the original files back in the days. Such a shame.
-
Its a shame they did not save the original files back in the days. Such a shame.
That's i saying to myself each time i've upscaled any graphic of the game
-
Outstanding progress.
-
Hi Satsuki, maybe you can try this out?
https://github.com/kornelski/undither
It seems very promising on dithered color palates.
Dithered:
(https://cloud.githubusercontent.com/assets/72159/2559943/e076175e-b796-11e3-8006-95b16b9563f8.png)
Filtered:
(https://cloud.githubusercontent.com/assets/72159/2559942/e0724f98-b796-11e3-8ba3-0347b852fbef.png)
-
Thanks but waifu do a better cleaning job.
I'll try on complexe screen to see if it can boost the final quality
-
Sorry for being kinda afk. I'll clean up the vapoursynth assistant more soon, but it more or less works now.
I've tried NNEDI3 and fast denoiser with a test video recently, and I can tell you that neural network upscaling + KNLMeansCL or V-BM3D for temporal denoising looks WAY better. And it shouldn't be too slow with a good GPU.
Also, when you say NNEDI2... are the source clips interlaced, or did you just mean NNEDI2 image doubling?
Yeah, NNEDI3 (I meant that version) would be meant here for image doubling. At its time, it was the best scaler by far, and it yielded some very light denoising and great antialiasing as a side bonus. And it was developed with neural networks.
As for M(V)Degrain3, it's just the latest motion compensated denoiser I know of. It is quite slow (much more than, say, dfttest) and at its time struck a great balance between denoising and actual detail removal. Where can I read about KNLMeansCL and V-BM3D?
-
hy, i have just seen huge possibilities how your "IA" can recover information of images and, while i have no idea of programming im a amateur photographer and pixel peeper and i have an idea, for example get this chart https://www.dpreview.com/articles/3692631027/newreschart scan it at at least 600dpi and then with a DNG image of a camera to train the IA, this chart have much information and you can deal with big problem of photography industry, moire, nyquist frequencys, diagonal lines that seem stairs etc
-
This chart is well know (i'm doing digital reflex photo for about 15 years) but the IA need to train from real cases, this absctract chart won't help the IA rendering to my mind.
But if you provide an HQ version of it, i could try to train a model to see what it's going to do ^^
-
satsuki, do you know if is possible to make a portable version out of Steam's FF7 and mods like yours and Final Heaven's?
-
A portable version can't be easily made.
Can't be with steam version without heavy hacking.
With a converted 98 version where's may be a way:
FF7 needs registry entry and select the right audio with it's own config exe to works.
So you'll need to do a launcher witch take care of it for you them clean the registry once game over.
You'll need to found a way too for the cd verification needed at launch.
I don't have to do it right now but maybe someone'll have the time, try to post it in the game tool forum
-
here's a video test of a part of the opening, filtered from psx version then esrgan : http://wowsatsuki.free.fr/optest.mkv
don't know if i prefer the steam version or this one, mine don't have aliasing and is sharper , but steam seems to bring out more details
-
The game tool forum is for releases only. But excuse me for being short not related to the topic.
The game can be made portable on an usb stick, some one had made a tool for it. But since it needs the CD check removed from the exe, it was removed from Quinn (at last the bundle with the hacked exe).
-
here's a video test of a part of the opening, filtered from psx version then esrgan : http://wowsatsuki.free.fr/optest.mkv
don't know if i prefer the steam version or this one, mine don't have aliasing and is sharper , but steam seems to bring out more details
Thats actually a much better result then what I got during testing!!! You should use the videos from http://forums.qhimm.com/index.php?topic=8414.0. I personally use Toshiba DVD<No Chibi<AC,ACC
Higher quality base, and then replace any FMV that has horrible chibis, and then finally replace anything I can with AC/ACC. This gives me the best results until some of the FMV's that had chibis are just remade... I know Echo-S is already working on a few for testing now.
-
Yeah, NNEDI3 (I meant that version) would be meant here for image doubling. At its time, it was the best scaler by far, and it yielded some very light denoising and great antialiasing as a side bonus. And it was developed with neural networks.
As for M(V)Degrain3, it's just the latest motion compensated denoiser I know of. It is quite slow (much more than, say, dfttest) and at its time struck a great balance between denoising and actual detail removal. Where can I read about KNLMeansCL and V-BM3D?
https://en.wikipedia.org/wiki/Block-matching_and_3D_filtering
https://en.wikipedia.org/wiki/Non-local_means
https://github.com/Khanattila/KNLMeansCL/wiki
https://github.com/HomeOfVapourSynthEvolution/VapourSynth-BM3D
This is a really good (but sloooooow) artifact remover too: https://forum.doom9.org/showthread.php?t=173470
Doom9 has some good threads on all of em, and the vapoursynth fatpack supports all those filters right out of the box. You can also enhance those filters by using cutting edge masks to further preserve detail, like retinex.
Someones working on a GPU BM3D, but it doesn't support temporal denoising yet, which means it really doesn't have an advantage over the ML-based image denoisers that are out there.
https://github.com/JeffOwOSun/gpu-bm3d
Also, if you think MVDegrain3 is slow, just wait until you try BM3D or Oyster on the CPU :D
-
I'm trying to filter as much elements of the game i can but i got 2 battle elements i can't found/extract from the original game.
the logo of the truck witch bump palmer in the rocket village battle:
(http://yatoshicom.free.fr/ff7p/logo.jpg)
the crackle in the stalactite at the end of the battle in gaia mountain:
(http://yatoshicom.free.fr/ff7p/ice.jpg)
Any gess ?
Thanks
-
Wait, if you can't find them in the game, how do you know you need them?
-
The problem is that he can find them in the game but not in the game files.
-
exactly ^^
-
Those are going to be most likely contained within their respective model. GKAC will be the truck, IJAC will be the icicle. You'll need to rename them to gkac.tex and ijac.tex then download image2tex to convert them to bmp files
-
thanks for the information.I'll try as soon as possible
-
I managed to extract and convert them but in witch subfolder of my mod do i need to place them ?
thanks again
-
I don't know your folder structure nor your method of injecting these textures into the game.
-
I don't know your folder structure nor your method of injecting these textures into the game.
I only use the "mods" folder set up in "ff7_opengl.cfg"
So here's my mods folder:
(http://wowsatsuki.free.fr/fld.png)
Thanks again
-
The textures from battle have to be placed into the battle folder. If I remember right the name structure was as following: 'texture name'_00.png. So the texture rtac would be replaced by rtac_00.png from the mod folder.
-
Already tryed without luck this folder and this name :lol:
-
The folder is right but I think the name may be wrong. you could try rtac_t00_00.png. I can't remember how you find out the correct names. You could instead use your direct folder for the time being, or just convert it back to a .tex file and put it back in the battle.lgp
-
Spotted with the app.log and show missing texture option
ijac need to be named ENEMY217_T00_00
gkac need to be named ENEMY166_T00_00
Thanks for helping
-
Sorry I forgot that Aali's driver has a hidden option: save_textures = on
This way you can dump all textures the game loads in memory. But keep in mind it will be written into the current active mod folder.
-
Good to know
-
Hey, would you be so kind and release the non Palmer converted up-scaled fields, too. I would like to tweak some of them manually. Nothing special, just working out some letters or adding details like stars in the sky (the scene when you leave Midgar).
-
Hey, would you be so kind and release the non Palmer converted up-scaled fields, too. I would like to tweak some of them manually. Nothing special, just working out some letters or adding details like stars in the sky (the scene when you leave Midgar).
The archive would be realy big, if you tell me wich one you need i can upload them to you one the pack will be released
-
Thread can be closed
Release thread : http://forums.qhimm.com/index.php?topic=18661.0