[FF9 PC] FMV upscaling & 30fps project

[snouz]

I think it's better to make another thread for the discussion I'm having with @Lykon (Steam chat is not permanent, main thread is full of other discussions and other people might want to contribute)

There are 2 video types:
-MBG, used as background to 3d objects
-FMV, simple videos

MASTER PLAN
1) test upscaled 60fps in the game (check for speed in the game, vanilla and Memoria)
2a) try to get source videos
2b) or use downscaled images to 320x224 nearest
3) If there's artifact like dithering, there's an ESRGAN model to undither.
4) Upscaling possibilities: srmd-ncnn-vulkan / ESRGAN / other?
5) if possible, blend the background in the video and color correct
6) Use DAIN to add frame interpolation
7) TEST


OBSERVATIONS
-MBGs above 15fps make 3D models glitch, but FMVs look fine. It might be fixable if we find the right code. Lykon found a relevant part in HW in CIL Code > Raw Code > MBG
-vids are tretched by the game, so we can keep original PSX x4 resolution (320x224 x4 = 1280x896) instead of 1280x960 in the steam version
-videos raised to 60fps look weird (too much guessing by DAIN, and contrasts too much with the original game)

INVENTORY
FMV000
FMV001
FMV002
FMV003
FMV004
FMV005
FMV006A
FMV006B
FMV008
FMV0011
FMV0012
FMV0013
FMV0014
FMV0016
FMV0017
FMV0019
FMV0021
FMV0023
FMV0024
FMV0027
FMV0029
FMV0031
FMV0032
FMV0033
FMV0034
FMV0035
FMV0036
FMV0038
FMV0039
FMV0040
FMV0042
FMV0045
FMV0046
FMV0052
FMV0053
FMV0055A
FMV0055B
FMV0055C
FMV0055D
FMV0056
FMV0059
FMV0060

mbg101 | Field100 (end)
mbg102 | Field209 (middle)
mbg103 | Field254 (start)
mbg105 | (none)
mbg106 | Field1001 (end)
mbg107 | (none)
mbg108 | Field1464 (start & end)
mbg109 | Field1554 (start & end)
mbg110 | (none)
mbg111 | Field2007 (end)
mbg112 | Field2255 (start)
mbg113 | Field2715 (start)
mbg114 | Field2752 (middle)
mbg115 | Field2929 ()
mbg116 | (none)
mbg117 | (none)
mbg118 | Field2905 (start & end)
mbg119 | (none)

[Lykon]

from very initial testing:

- As stated by snouz, every single video is 15 fps by default.

- The algorithm used for image upscaling (ESRGAN if I'm not wrong) doesn't work well on videos because of "guesses" that vary from frame to frame. We are using video algorithms for super resolution (waifu2x vulkan for now, MMSR as soon as possible). That doesn't mean that ESRGAN can't be used at all, it's probably perfect for the final part of FMV000 (when the logo appears) since there is not much room for "guesses".

- I'd avoid upscaling FMV videos (except FMV060) because they're already good (and except for the final part of FMV000).
- Character animations on MGB videos are linked to frames, so script editing is required. The duration is fine, but the double or quadruple framerate makes the character move 2 times or 4 times faster to keep the same fps as the new background video.
- 60 fps for MGB videos is too much since the game is locked to 30 fps (and even less in world map and battle). 30 fps blends in a nicer way (but still requires script editing as mentioned above, at least for some of them).
- FMV videos are fine even at higher fps. I'm interpolating the first menu video to 60 fps right now, so more on that later (going from 60 to 30 is trivial).

- Our biggest problem is still the blending of GAN-upscaled static backgrounds with upscaled video backgrounds. The new tests are promising from a pixel perspective. The issue now is more on the colors. E.g. mbg111 ends in totally different colors compared to the static background, even the default one.
This can be probably achieved in a second pass just using color correction.

- I'd personally avoid overlapping video and images to avoid a "powerpoint"-like transition.

Reposting here some of the videos we used for testing:
https://mega.nz/file/EQFHQCqT#wHkao7ZTtRKLEQBmyZTYlmYznPaYuvgM09eqTFMdOLg
https://mega.nz/file/UB0hxQQZ#5xDMZYGMa87GwC4CkqqEMio_jYEgmei3YfpclzPBBzQ
The password for the archives is "lykon"
EDIT: a 30 fps file was instead 60 fps, I've fixed it

[Lykon]

Tools I'm using:

- Waifu2x-Extension-GUI ( https://github.com/AaronFeng753/Waifu2x-Extension-GUI/releases ) for super resolution.
- ffmpeg for video downscaling (our pipeline requires nearest neighbor)
- Free_OGV_Video_Converter_64bit_PORTABLE for mp4 to ogv (bytes) conversion
- DAIN for frame interpolation and fps increase

I'm also keeping an eye on Video2x ( https://github.com/k4yt3x/video2x ) because they're planning to add MMSR  to their gui.

[Lykon]

Did frame interpolation on the main menu video. This one looks very promising and it's easy to test:
https://mega.nz/file/dJV2xSYb#bvenyorNvktCUdVyZ5btU5P1WO7rYKaBzj713zgSZTo
(password is "lykon")

NB: this video is 720p instead of 1280x960. This is done to also test something else and it's not final.

A couple of considerations:
1. Scene changes are bad, but there is an option in the algorithm for automated scene change detection that I want to test next (on a shorter video at first)
2. This one is maybe too good. The game later runs at 30 FPS, so starting with a 60 FPS video may "train" the eye to spot the difference. However it's not that obvious and we'd need feedback. It's true that this video is very particular, with many moving parts mounted on one another, so it may be the only one that's fine leaving at 60 fps (if we don't decide to do the same on all of them). 

Watching the 60 FPS version makes the 30 FPS one look like crap an the 15 FPS one look like trash, expecially the very first part with the zoom inside the text.

30 FPS is probably the way to go for 99% of the videos in that folder, with (maybe) the exception of 000 and 060, because they have sliding effects and zoom-in effects, that are usually crap at rates lower than 60.

[thebrokepope]

Don't have much useful to add but I think this is really important work and I'm glad someone is tackling it. Maintaining a convincing transition between the FMVs and gameplay is imperative to preserve the original intention of the games on PSX. When it comes to FPS, and all matters really, my preference is to keep the experience as consistent as possible and only providing upgrades when they don't violate this consistency. I believe IX updates at 30fps in the field on both PSX and PC? In that case I would definitely prefer FMVs be kept to 30fps at most to maintain consistency with gameplay. It's also highly unusual to see CGI like the intro video at 60fps so the sample Lykon posted looks very strange to my eyes. Personally though I don't care about boosting the FPS, I just want the transitions to be done right. Thank you so much.

[thebrokepope]

P.S the 30fps sample certainly does not look "crap" to my eyes! It looks just fine. I actually can't think of any precedence for pre-rendered CGI being done at 60fps.

[snouz]

I agree with Brokenpope on 30fps. Looks definitely better than the original 15, but 60fps looks uncanny.

[Lykon]

Yeah, I'm also in for 30 fps. It fits the game better and it's acceptable if you don't watch it right after you've watched the 60 fps one.
By "crap" I was referring to the zoom in effect in particular. The worst however was the scene change. More on that below.


New test on DAIN scene change detection
Here you have the same video in 4 files:
- upscaled from 320x224 (nearest),
- upscaled from 640x448 (nearest),
- upscaled from 640x448 with DAIN applied without scene change detection,
- upscaled from 640x448 with DAIN applied with scene change detection -> new

The best one is by far the last one (of course, upscaled from 640x448, it's by far the best version).

https://mega.nz/file/BVdA1IAL#Q-iHqM5QdwBx6dXuZnARDK_qIHbPBOtOZR-xVDfhVaY
password is my name lowercase as usual.
EDIT: forgot to convert to ogv, but it doesn't really matter since the purpose of this test is not to check what happens in the game.

I've also devised a strategy for background blending.

for MBG:

- color correction to match the final (or initial) static scene
- downscale nearest to 640x480 (mbg101 is the only one that needs to be 320x224 because of the character models)
- zoom waifu2x and zoom esrgan (2 different video outputs) to 1280x896
- blending of the 2 videos (before or after depending on the scene) -----> this is yet to be tested, should produce fine results in theory
- DAIN x2 (if we manage to fix the script)

For FMV:

- DAIN x2

[Caledor]

Syncing issue with 30 or 60 FPS videos:
Increasing the frame rate of videos makes the field models start moving earlier, as one can easily see for themselves with the Alexandria videos uploaded here.

What prompts the actual field models to start moving is a check within the field script. For example, with the Alexandria video, field is "Alexandria/Main Street"

Function Vivi_Loop, we can see the following:

        if ( VAR_GlobBool_146 == 0 ) {
            Wait( 1 )
        } else {
            while ( ( GetCinematicFrame < 100) && ( ( SyncCinematic & 127 ) != 1 ) ) {
                Wait( 1 )
            }
        }

That GetCinematicFrame is the threshold. Double it for 30 FPS videos or x4 with 60 FPS video.

[Lykon]

Syncing issue with 30 or 60 FPS videos:
Increasing the frame rate of videos makes the field models start moving earlier, as one can easily see for themselves with the Alexandria videos uploaded here.

What prompts the actual field models to start moving is a check within the field script. For example, with the Alexandria video, field is "Alexandria/Main Street"

Function Vivi_Loop, we can see the following:

That GetCinematicFrame is the threshold. Double it for 30 FPS videos or x4 with 60 FPS video.

Nice one. Can't we simply do

Code: [Select]

GetCinematicFrame / 2
?

[snouz]

Nice one. Can't we simply do

Code: [Select]

GetCinematicFrame / 2

Lykon, could you do a simple resampling in 30fps of all MBG so that I can try each of them and make the list of those that'll be able to up-frame and those that'll have to stay 15fps.

[Lykon]

Lykon, could you do a simple resampling in 30fps of all MBG so that I can try each of them and make the list of those that'll be able to up-frame and those that'll have to stay 15fps.

Consider it done. Should take some minutes.
I'm going to use 30 fps for mbg instead of 29.94 fps. Even if that's the output of dain, I'll always resample manually to 30 fps for MBG videos (while leaving FMV to 29.94 to avoid quality loss).
Also keep in mind that we can mix true 30 fps with fake 30 fps (when I say fake 30 fps I mean 15 resampled to 30). This could help in scenes when the character is overlapped with the video and needs to move at the same framerate to avoid strange things happening.

In the meantime I've created a folder for finished videos. I've uploaded FMV001.bytes that should be finished and used the previously mentioned technique of mixing true 30 and fake 30. This is because DAIN was unable to interpolate the rain scene (too much noise!!)

https://mega.nz/folder/cVchlAbL
this is the password: A5qU-wvZiy3wc2Mm_bGeag
UPDATE: this is now outdated, check posts below

The folder contains FMV001 in 30fps (mostly native using DAIN, but I couldn't use 100% of that).
I'm also trying a 20 and a 24 fps version to avoid obvious framedrops.

I'm also doing DAIN on FMV003 right now and it's turning out great.
21% examined frame by frame manually and it's perfect so far, so the issues may be only related to noise and overlay text, meaning we have to struggle only for FMV001 and FMV002.

I've also completed waifu2x upscaling on every MGB video

[Lykon]

So, I've finally managed to have FMV001 fully 30fps mixing DAIN (for the most part) with SMV and MotionFlow where DAIN failed.

You can find it in the folder
https://mega.nz/folder/cVchlAbL
this is the password: A5qU-wvZiy3wc2Mm_bGeag

Together with FMV003 (that was way easier, 100% DAIN with 5 frames fixed manually).

[Neodoris]

Did frame interpolation on the main menu video. This one looks very promising and it's easy to test:
https://mega.nz/file/dJV2xSYb#bvenyorNvktCUdVyZ5btU5P1WO7rYKaBzj713zgSZTo
(password is "lykon")

NB: this video is 720p instead of 1280x960. This is done to also test something else and it's not final.

The 60 fps version is truly a dream come true, especially for the zooming Squaresoft logo and Odin attack on Iifa Tree. Wonderful. If you do not put it in the final game, I will keep this file and put it myself in the right folder 

[Lykon]

The 60 fps version is truly a dream come true, especially for the zooming Squaresoft logo and Odin attack on Iifa Tree. Wonderful. If you do not put it in the final game, I will keep this file and put it myself in the right folder 

I'm actually redoing that one with the new stuff we've learned. It's "just" 30 fps however (almost impossible to get a good result on 60 fps since the source material is 15 fps).

However the problem now is that the game still reads them as 15fps, skipping frames...


Also a suggestion if you want to use DAIN APP: Never use "perform all steps".

Do:

- step 1 (split video source blabla)
- check scene change sensitivity (won't get all scene changes or will get too many), requires step 1
- step 2 (dain)
- manyally check frames (REPLACE !!! the ones that have errors with good frames, DO NOT delete them, replace)
- do step 3 (convert dain to videos)


Hypotetically the order of things would be:

1) we fix the MBGs (color correction, upscaling, blending)
2) we can release the first version in mog rework
3) we work on the 30fps issue for movies (that are currently read at 15fps by the game engine, despite being 30fps)
4) we bring the movies to 30 fps (new release?)
5) we work on 30fps for mbgs6) we bring mbgs to 30 fps
7) new release

there is no need to do 1 big release, we should go in an iterative way (cascade is bad)

[Lykon]

So for now (and just for now) I'll stop running DAIN on videos to bring them to 30 fps (except when I've nothing better to do ahah). Until we figure out if it's possible and convenient to use them in game.
Running DAIN on a video takes usually from 3 to 6 hours and there is also a lot of work to do manually to fix broken frames and artifacts.

However, since SVP is NOT a free software and I can't share the license, I'll run SVP on all of them so that there is the backup track (I'm basically using the crappy SVP track to fix the broken frames from DAIN that is usually much better).

I'll share every FMV interpolated using SVP as soon as I'm done so that you're not forced to buy the license. DAIN is free so anyone can take over and split the work with me once we're ready.

SVP interpolated videos will be here: https://mega.nz/folder/kRERCKSR
password: yEP59xdOzzjT7opPpoRS1w

[Lykon]

Fun fact

https://imgur.com/a/Rq6O1NP

Scene change always happen at frames that are multiple of 5 in the original videos. This maybe means that their original video editing software couldn't handle precise selections and only worked in chunks of 5 frames. Or it can mean that their original codec only supported 1 keyframe every 5.

[Lykon]

So I finished converting all the FMV videos (mounting the last 2 tomorrow).

After that we'll just check for sound sync in game, minor errors and probably try and remove some compression noise by running waifu2x on all the videos.

Hope you all like what you can see so far. Every video except maybe 5 has been a frame by frame reconstruction from multiple interpolation algorithms (depending on where the main one failed).

[m_artn]

Awesome work with the Videos!
Can't wait to have it all in one pack with snouz rework an memoria.
This will definitely be the best way to play ff9

[Neodoris]

So I finished converting all the FMV videos (mounting the last 2 tomorrow).

After that we'll just check for sound sync in game, minor errors and probably try and remove some compression noise by running waifu2x on all the videos.

Hope you all like what you can see so far. Every video except maybe 5 has been a frame by frame reconstruction from multiple interpolation algorithms (depending on where the main one failed).

Can you show us one of the video so that we may have a look at it? Thanks!

[Lykon]

Can you show us one of the video so that we may have a look at it? Thanks!

I've just finished converting all the videos in high bitrate after a waifu2x pass. I just miss the very last one (the credits) but it's going to be done tonight.

In the meantime, this is the link to the folder: https://mega.nz/folder/RJEFnarS
password: phkQlOS8goP72x71v2fM3w

Snouz is working on some redo (from scratch) of the parts of fmv000 and fmv060 where logos are involved. We also managed to extract the map of the first scene of fmv000, so that he can recreate the logo transition from scratch.

After that we just need in-game testing.

Please keep in mind that the game runs videos at 15fps unless you have the last version of Memoria. (regardless of the original framerate)

Also keep in mind that videos are converted to OGV using an outrageous bitrate to avoid quality loss (ogv sucks). Hope they run well on your machines.
Talking about ~3 GB of videos so, despite all, that not so far from the original file size that is around 2.5 GB

[decency_breakin3]

so ready to try this out.

[Lykon]

so ready to try this out.

Yes, it has been included today in the new mog rework and will also be in the installer version (tomorrow maybe)