Author Topic: Could we use ML to upscale FFVII's FMVs?  (Read 342 times)

Bosola

  • Fire hazard!
  • No life
  • *
  • Posts: 1759
  • Karma: 15
    • View Profile
    • My YouTube Channel
Could we use ML to upscale FFVII's FMVs?
« on: 2019-02-01 23:58:06 »
Inspired by this thread, I wonder - could machine learning help us upscale the old 320x224 FF7 FMVs?

The videos were downscaled from original sources which seem to have been lost, and the nearest thing we have to them are a handful of 720p renders on a very rare, very old Toshiba demo DVD.

Still, I wonder if perhaps tiles from them could be used to create a training set for upscaling other videos. If we sliced the videos into squares of 32x32 we would get 70 reference sets per frame, which multiplied by 30fps should equal 2100 pairs per second of video (though there will be a lot of duplication in the data).

Is this approach feasible? How many images would we need? Are there other training sets we could augment for our own purposes? What's the incident of artefacts on this kind of processing?
« Last Edit: 2019-02-02 00:04:12 by Bosola »

opengrip

  • Fast newbie
  • *
  • Posts: 23
  • Karma: 1
    • View Profile
Re: Could we use ML to upscale FFVII's FMVs?
« Reply #1 on: 2019-02-02 01:09:17 »
I have done this but the results i got were only ok. The video file I tested on was only 2 seconds long and ran at 15 fps so i needed 30 images. longer videos would take quite a bit longer to process. You can use VLC to split the video into images and then blender for image sequences to put it back together.

Covarr

  • Covarr-Let
  • Administrator
  • No life
  • *
  • Posts: 4017
  • Karma: 122
  • Just Covarr. No "n".
    • View Profile
Re: Could we use ML to upscale FFVII's FMVs?
« Reply #2 on: 2019-02-02 06:57:25 »
a handful of 720p renders on a very rare, very old Toshiba demo DVD.
They are 480p.

The thing is, any given frame of the currently best-available videos is going to be far, far worse quality than the backgrounds. Crappy '90s video compression is crappy, and what we have is loaded with artifacts. And as the saying goes, garbage in, garbage out. Quite frankly, I'd be very, very surprised if upscaling frame-by-frame could get worthwhile results.

I would think the best scaling results we could see would come from something temporal-based, something that fills in details that may have been compressed to death in one frame by examining neighboring frames, but unless the videos found in the PS1 version are loaded up with I-frames, I don't know how useful this'll actually be, since other frames already only record motion, what's changed, and not a full image.

As for using the clips from the DVD as training material... that could help, but I don't know how much, as this DVD is rife with its own problems. Among other things, it's all interlaced, and the aspect ratio is rather different. It could probably be done, and would probably help a lot (and might even help as training material for background scaling), but it would need significant cleaning up before it would be useful.

MysticLord

  • Fast newbie
  • *
  • Posts: 22
  • Karma: 5
    • View Profile
    • SaGa Frontier Research Board
Re: Could we use ML to upscale FFVII's FMVs?
« Reply #3 on: 2019-02-14 18:41:13 »
Someone on /vr/ tried upscaling Chrono Cross backgrounds using models derived from other types of images, but iirc the results were so-so at best.  You'd need to train it on a data set specific to old-school games. Beyond that I know nothing of ML, just repeating hearsay from those threads on /vr/. Maybe try the existing Chrono Cross community to see if you can find some advice? And be prepared for a massive pile of data, like multiple terabytes worth of images.

edit

I think the issue was that the models used to upscale an image were based on those used to upscale low resolution jpegs of real world stuff, or stuff from games like Skyrim, and the images found in most PS1 games - especially those with prerendered backgrounds - are not upscaled correctly by those models. The TC stated that he tried training them on images ripped from CC and he got some good results, but his hard drive ran out of space after the 2nd iteration or something.
« Last Edit: 2019-02-14 18:50:31 by MysticLord »

AlphaAtlas

  • Fast newbie
  • *
  • Posts: 10
  • Karma: 1
    • View Profile
Re: Could we use ML to upscale FFVII's FMVs?
« Reply #4 on: 2019-02-14 20:09:00 »
Still, I wonder if perhaps tiles from them could be used to create a training set for upscaling other videos. If we sliced the videos into squares of 32x32 we would get 70 reference sets per frame, which multiplied by 30fps should equal 2100 pairs per second of video (though there will be a lot of duplication in the data).

Upscaling is possible! But you might be better off using thoroughly pre-trained "generic" models rather than working from such a limited data set, though I'm just diving into all this myself.

You can use VLC to split the video into images and then blender for image sequences to put it back together.

That's a... inefficient way to do it, especially with longer videos. It's better to upscale, filter, and encode the video all at once so you dont have all those images sitting on your hard drive. I started a project (based on other projects) just for this kind of thing: https://github.com/AlphaAtlas/vs_mxnet_helper



EDIT: Here's a quick and dirty encode:

https://drive.google.com/open?id=1kQ37rQ3RaLBVvtaNVtHT3z4pIdkdrsjg

It's got some noise and is still 15 FPS, but that could be solved with a few more minutes of tweaking.

« Last Edit: Yesterday at 03:11:56 by AlphaAtlas »