Final Fantasy 9 > FF9 Graphical Releases

[FF9 PC] FMV upscaling & 30fps project

(1/5) > >>

snouz:
I think it's better to make another thread for the discussion I'm having with @Lykon (Steam chat is not permanent, main thread is full of other discussions and other people might want to contribute)

There are 2 video types:
-MBG, used as background to 3d objects
-FMV, simple videos

MASTER PLAN
1) test upscaled 60fps in the game (check for speed in the game, vanilla and Memoria)
2a) try to get source videos
2b) or use downscaled images to 320x224 nearest
3) If there's artifact like dithering, there's an ESRGAN model to undither.
4) Upscaling possibilities: srmd-ncnn-vulkan / ESRGAN / other?
5) if possible, blend the background in the video and color correct
6) Use DAIN to add frame interpolation
7) TEST


OBSERVATIONS
-MBGs above 15fps make 3D models glitch, but FMVs look fine. It might be fixable if we find the right code. Lykon found a relevant part in HW in CIL Code > Raw Code > MBG
-vids are tretched by the game, so we can keep original PSX x4 resolution (320x224 x4 = 1280x896) instead of 1280x960 in the steam version
-videos raised to 60fps look weird (too much guessing by DAIN, and contrasts too much with the original game)

INVENTORY
FMV000
FMV001
FMV002
FMV003
FMV004
FMV005
FMV006A
FMV006B
FMV008
FMV0011
FMV0012
FMV0013
FMV0014
FMV0016
FMV0017
FMV0019
FMV0021
FMV0023
FMV0024
FMV0027
FMV0029
FMV0031
FMV0032
FMV0033
FMV0034
FMV0035
FMV0036
FMV0038
FMV0039
FMV0040
FMV0042
FMV0045
FMV0046
FMV0052
FMV0053
FMV0055A
FMV0055B
FMV0055C
FMV0055D
FMV0056
FMV0059
FMV0060

mbg101 | Field100 (end)
mbg102 | Field209 (middle)
mbg103 | Field254 (start)
mbg105 | (none)
mbg106 | Field1001 (end)
mbg107 | (none)
mbg108 | Field1464 (start & end)
mbg109 | Field1554 (start & end)
mbg110 | (none)
mbg111 | Field2007 (end)
mbg112 | Field2255 (start)
mbg113 | Field2715 (start)
mbg114 | Field2752 (middle)
mbg115 | Field2929 ()
mbg116 | (none)
mbg117 | (none)
mbg118 | Field2905 (start & end)
mbg119 | (none)

Lykon:
from very initial testing:

- As stated by snouz, every single video is 15 fps by default.

- The algorithm used for image upscaling (ESRGAN if I'm not wrong) doesn't work well on videos because of "guesses" that vary from frame to frame. We are using video algorithms for super resolution (waifu2x vulkan for now, MMSR as soon as possible). That doesn't mean that ESRGAN can't be used at all, it's probably perfect for the final part of FMV000 (when the logo appears) since there is not much room for "guesses".

- I'd avoid upscaling FMV videos (except FMV060) because they're already good (and except for the final part of FMV000).
- Character animations on MGB videos are linked to frames, so script editing is required. The duration is fine, but the double or quadruple framerate makes the character move 2 times or 4 times faster to keep the same fps as the new background video.
- 60 fps for MGB videos is too much since the game is locked to 30 fps (and even less in world map and battle). 30 fps blends in a nicer way (but still requires script editing as mentioned above, at least for some of them).
- FMV videos are fine even at higher fps. I'm interpolating the first menu video to 60 fps right now, so more on that later (going from 60 to 30 is trivial).

- Our biggest problem is still the blending of GAN-upscaled static backgrounds with upscaled video backgrounds. The new tests are promising from a pixel perspective. The issue now is more on the colors. E.g. mbg111 ends in totally different colors compared to the static background, even the default one.
This can be probably achieved in a second pass just using color correction.

- I'd personally avoid overlapping video and images to avoid a "powerpoint"-like transition.

Reposting here some of the videos we used for testing:
https://mega.nz/file/EQFHQCqT#wHkao7ZTtRKLEQBmyZTYlmYznPaYuvgM09eqTFMdOLg
https://mega.nz/file/UB0hxQQZ#5xDMZYGMa87GwC4CkqqEMio_jYEgmei3YfpclzPBBzQ
The password for the archives is "lykon"
EDIT: a 30 fps file was instead 60 fps, I've fixed it

Lykon:
Tools I'm using:

- Waifu2x-Extension-GUI ( https://github.com/AaronFeng753/Waifu2x-Extension-GUI/releases ) for super resolution.
- ffmpeg for video downscaling (our pipeline requires nearest neighbor)
- Free_OGV_Video_Converter_64bit_PORTABLE for mp4 to ogv (bytes) conversion
- DAIN for frame interpolation and fps increase

I'm also keeping an eye on Video2x ( https://github.com/k4yt3x/video2x ) because they're planning to add MMSR  to their gui.

Lykon:
Did frame interpolation on the main menu video. This one looks very promising and it's easy to test:
https://mega.nz/file/dJV2xSYb#bvenyorNvktCUdVyZ5btU5P1WO7rYKaBzj713zgSZTo
(password is "lykon")

NB: this video is 720p instead of 1280x960. This is done to also test something else and it's not final.

A couple of considerations:
1. Scene changes are bad, but there is an option in the algorithm for automated scene change detection that I want to test next (on a shorter video at first)
2. This one is maybe too good. The game later runs at 30 FPS, so starting with a 60 FPS video may "train" the eye to spot the difference. However it's not that obvious and we'd need feedback. It's true that this video is very particular, with many moving parts mounted on one another, so it may be the only one that's fine leaving at 60 fps (if we don't decide to do the same on all of them). 

Watching the 60 FPS version makes the 30 FPS one look like crap an the 15 FPS one look like trash, expecially the very first part with the zoom inside the text.

30 FPS is probably the way to go for 99% of the videos in that folder, with (maybe) the exception of 000 and 060, because they have sliding effects and zoom-in effects, that are usually crap at rates lower than 60.

thebrokepope:
Don't have much useful to add but I think this is really important work and I'm glad someone is tackling it. Maintaining a convincing transition between the FMVs and gameplay is imperative to preserve the original intention of the games on PSX. When it comes to FPS, and all matters really, my preference is to keep the experience as consistent as possible and only providing upgrades when they don't violate this consistency. I believe IX updates at 30fps in the field on both PSX and PC? In that case I would definitely prefer FMVs be kept to 30fps at most to maintain consistency with gameplay. It's also highly unusual to see CGI like the intro video at 60fps so the sample Lykon posted looks very strange to my eyes. Personally though I don't care about boosting the FPS, I just want the transitions to be done right. Thank you so much.

Navigation

[0] Message Index

[#] Next page