Author Topic: Could we use ML to upscale FFVII's FMVs?  (Read 1277 times)

Bosola

  • Fire hazard!
  • No life
  • *
  • Posts: 1769
  • Karma: 15
    • View Profile
    • My YouTube Channel
Could we use ML to upscale FFVII's FMVs?
« on: 2019-02-01 23:58:06 »
Inspired by this thread, I wonder - could machine learning help us upscale the old 320x224 FF7 FMVs?

The videos were downscaled from original sources which seem to have been lost, and the nearest thing we have to them are a handful of 720p renders on a very rare, very old Toshiba demo DVD.

Still, I wonder if perhaps tiles from them could be used to create a training set for upscaling other videos. If we sliced the videos into squares of 32x32 we would get 70 reference sets per frame, which multiplied by 30fps should equal 2100 pairs per second of video (though there will be a lot of duplication in the data).

Is this approach feasible? How many images would we need? Are there other training sets we could augment for our own purposes? What's the incident of artefacts on this kind of processing?
« Last Edit: 2019-02-02 00:04:12 by Bosola »

opengrip

  • Fast newbie
  • *
  • Posts: 30
  • Karma: 2
    • View Profile
Re: Could we use ML to upscale FFVII's FMVs?
« Reply #1 on: 2019-02-02 01:09:17 »
I have done this but the results i got were only ok. The video file I tested on was only 2 seconds long and ran at 15 fps so i needed 30 images. longer videos would take quite a bit longer to process. You can use VLC to split the video into images and then blender for image sequences to put it back together.

Covarr

  • Covarr-Let
  • Administrator
  • No life
  • *
  • Posts: 4037
  • Karma: 122
  • Just Covarr. No "n".
    • View Profile
Re: Could we use ML to upscale FFVII's FMVs?
« Reply #2 on: 2019-02-02 06:57:25 »
a handful of 720p renders on a very rare, very old Toshiba demo DVD.
They are 480p.

The thing is, any given frame of the currently best-available videos is going to be far, far worse quality than the backgrounds. Crappy '90s video compression is crappy, and what we have is loaded with artifacts. And as the saying goes, garbage in, garbage out. Quite frankly, I'd be very, very surprised if upscaling frame-by-frame could get worthwhile results.

I would think the best scaling results we could see would come from something temporal-based, something that fills in details that may have been compressed to death in one frame by examining neighboring frames, but unless the videos found in the PS1 version are loaded up with I-frames, I don't know how useful this'll actually be, since other frames already only record motion, what's changed, and not a full image.

As for using the clips from the DVD as training material... that could help, but I don't know how much, as this DVD is rife with its own problems. Among other things, it's all interlaced, and the aspect ratio is rather different. It could probably be done, and would probably help a lot (and might even help as training material for background scaling), but it would need significant cleaning up before it would be useful.

MysticLord

  • Fast newbie
  • *
  • Posts: 26
  • Karma: 6
    • View Profile
Re: Could we use ML to upscale FFVII's FMVs?
« Reply #3 on: 2019-02-14 18:41:13 »
Someone on /vr/ tried upscaling Chrono Cross backgrounds using models derived from other types of images, but iirc the results were so-so at best.  You'd need to train it on a data set specific to old-school games. Beyond that I know nothing of ML, just repeating hearsay from those threads on /vr/. Maybe try the existing Chrono Cross community to see if you can find some advice? And be prepared for a massive pile of data, like multiple terabytes worth of images.

edit

I think the issue was that the models used to upscale an image were based on those used to upscale low resolution jpegs of real world stuff, or stuff from games like Skyrim, and the images found in most PS1 games - especially those with prerendered backgrounds - are not upscaled correctly by those models. The TC stated that he tried training them on images ripped from CC and he got some good results, but his hard drive ran out of space after the 2nd iteration or something.
« Last Edit: 2019-02-14 18:50:31 by MysticLord »

AlphaAtlas

  • Fast newbie
  • *
  • Posts: 10
  • Karma: 1
    • View Profile
Re: Could we use ML to upscale FFVII's FMVs?
« Reply #4 on: 2019-02-14 20:09:00 »
Still, I wonder if perhaps tiles from them could be used to create a training set for upscaling other videos. If we sliced the videos into squares of 32x32 we would get 70 reference sets per frame, which multiplied by 30fps should equal 2100 pairs per second of video (though there will be a lot of duplication in the data).

Upscaling is possible! But you might be better off using thoroughly pre-trained "generic" models rather than working from such a limited data set, though I'm just diving into all this myself.

You can use VLC to split the video into images and then blender for image sequences to put it back together.

That's a... inefficient way to do it, especially with longer videos. It's better to upscale, filter, and encode the video all at once so you dont have all those images sitting on your hard drive. I started a project (based on other projects) just for this kind of thing: https://github.com/AlphaAtlas/vs_mxnet_helper



EDIT: Here's a quick and dirty encode:

https://drive.google.com/open?id=1kQ37rQ3RaLBVvtaNVtHT3z4pIdkdrsjg

It's got some noise and is still 15 FPS, but that could be solved with a few more minutes of tweaking.

« Last Edit: 2019-02-15 03:11:56 by AlphaAtlas »

satsuki

  • Crazy poster
  • *
  • Posts: 232
  • Karma: 37
    • View Profile
    • My softs (french only)
Re: Could we use ML to upscale FFVII's FMVs?
« Reply #5 on: 2019-02-16 18:36:00 »
here's my current method (as seen in my upscale post)
(some custom noise and tmpclean with avisynth then esrgan)
http://wowsatsuki.free.fr/optest.mkv

Bosola

  • Fire hazard!
  • No life
  • *
  • Posts: 1769
  • Karma: 15
    • View Profile
    • My YouTube Channel
Re: Could we use ML to upscale FFVII's FMVs?
« Reply #6 on: 2019-02-18 08:52:32 »
That's really good work, satsuki. The method definitely shows promise.

DLPB

  • No life
  • *
  • Posts: 9451
  • Karma: 323
  • For I realized that God's a young man, too.
    • View Profile
Re: Could we use ML to upscale FFVII's FMVs?
« Reply #7 on: 2019-02-18 19:31:54 »
Actually they aren't 480p either those Toshiba ones.  They are 640*448 when you remove the black bars.

I recently bought NeatVideo, which is prob the best denoiser around.  I think any solution needs something that clears up frames by using preceding ones.

And using Toshiba where possible, as I did here (remember to set to 720p) for my project Goblins Bar:

https://drive.google.com/file/d/1MOgxeI8j6xqzCUZxmt_IkZj2I1kGB6Jv/view?usp=sharing
« Last Edit: 2019-02-18 19:37:07 by DLPB »

seam

  • Newbie
  • *
  • Posts: 4
  • Karma: 0
    • View Profile
Re: Could we use ML to upscale FFVII's FMVs?
« Reply #8 on: 2019-03-01 01:54:24 »
That's really good work, satsuki. The method definitely shows promise.

This looks awesome! I hope you intend to do all the fmvs this way :D

Brokenxwing

  • Fast newbie
  • *
  • Posts: 17
  • Karma: 0
    • View Profile
Re: Could we use ML to upscale FFVII's FMVs?
« Reply #9 on: 2019-04-05 09:46:24 »
Wow! These are all a pretty big improvement over even the Steam upscaled FMV's. I took screenshots of all 3 of those as well as the steam version, showing the opening shot of Aerith's face. They all look quite a bit better. Though to be honest, I think Satsuki's is the sharpest and has the nicest color! Are any of these available with all the FMV's done this way?

satsuki

  • Crazy poster
  • *
  • Posts: 232
  • Karma: 37
    • View Profile
    • My softs (french only)
Re: Could we use ML to upscale FFVII's FMVs?
« Reply #10 on: 2019-04-06 12:42:37 »
Wow! These are all a pretty big improvement over even the Steam upscaled FMV's. I took screenshots of all 3 of those as well as the steam version, showing the opening shot of Aerith's face. They all look quite a bit better. Though to be honest, I think Satsuki's is the sharpest and has the nicest color! Are any of these available with all the FMV's done this way?
here : http://forums.qhimm.com/index.php?topic=18661.0 (from french version so 2 of then have french texts)

Brokenxwing

  • Fast newbie
  • *
  • Posts: 17
  • Karma: 0
    • View Profile
Re: Could we use ML to upscale FFVII's FMVs?
« Reply #11 on: 2019-04-08 03:22:17 »
Out of curiosity, how do your 4x background upscales compare to CaptRobau's Remako HD Field mod? Would you say yours are better? Or about the same? I'm currently using his. And I found them to be a BIG improvement over older upscales and even more so compared to the blurry stock ones.

satsuki

  • Crazy poster
  • *
  • Posts: 232
  • Karma: 37
    • View Profile
    • My softs (french only)
Re: Could we use ML to upscale FFVII's FMVs?
« Reply #12 on: 2019-04-09 10:04:43 »
I found my own better:
The pack is final, not beta
I adapted filtering for most of screen, it's not a quick setting for all
I used esrgan, it's upscalling without adding false texture (as gigapixel does)

It's just my opinion, test and you'll see ^^