Author Topic: [FF8PC-Steam] Batch extraction of backgrounds  (Read 19320 times)

SaintBahamut

  • *
  • Posts: 5
    • View Profile
[FF8PC-Steam] Batch extraction of backgrounds
« on: 2018-11-30 18:38:14 »
Hello

Lately wanted wanted to replay ff8, first time on PC (steam) but low res backgrounds bit bother me and current mods dont really fit my taste, so i want tackle enhancing them myself. I know i need use tonbery and pupu but i cant find a good reliableway of batch extraction of backgrounds.

Extracting asaide, what i want to do is actualy put funds on some commercial app that does some AI processing. Some free examples i could get, shows promise:





Bit noisy but i think its a solid base for further improvements

Thanks


Maki

  • 0xBAADF00D
  • *
  • Posts: 624
  • 0xCCCCCCCC
    • View Profile
Re: [FF8PC-Steam] Batch extraction of backgrounds
« Reply #1 on: 2018-12-02 19:13:48 »
That upscale is extremely precise! I love it

Fraggoso

  • *
  • Posts: 278
    • View Profile
Re: [FF8PC-Steam] Batch extraction of backgrounds
« Reply #2 on: 2018-12-02 19:43:56 »
I can't help with the batch extract but one question from my side: Are you using letsenhance.io?
If yes, ask for a batch as filtering thousands of images will be a pain in the ass if you've to do it manually.

Maki

  • 0xBAADF00D
  • *
  • Posts: 624
  • 0xCCCCCCCC
    • View Profile
Re: [FF8PC-Steam] Batch extraction of backgrounds
« Reply #3 on: 2018-12-03 12:53:42 »
Damn, that's really great. There's no way to export all in one click, yet I'll try to do my best to do it asap.
New processed images:




Mcindus

  • *
  • Posts: 929
  • Artist, Modder, Musician.
    • View Profile
    • Lunatic Pandora
Re: [FF8PC-Steam] Batch extraction of backgrounds
« Reply #4 on: 2018-12-03 21:56:42 »
WOW.

So, If you find some of this too noisey, you can always mask with 'project angelwing' and use Angelwing as a color base, with possible gaussian blurs manually done for skies and other gradients.  That would be the 'ultimate' Field Background mod.

LeonhartGR

  • *
  • Posts: 2577
  • ~Whatever...~ Enjoy life!
    • View Profile
    • LeonhartGR Productions
Re: [FF8PC-Steam] Batch extraction of backgrounds
« Reply #5 on: 2018-12-05 17:24:54 »
This level of sharpness is exactly the one that matches my preferences! Congratulations mate and you just received your first "like" by me.

Maki

  • 0xBAADF00D
  • *
  • Posts: 624
  • 0xCCCCCCCC
    • View Profile
Re: [FF8PC-Steam] Batch extraction of backgrounds
« Reply #6 on: 2018-12-12 15:48:52 »
If you're still interested in batch background extraction, then I'm installing Qt now and will fire unofficial build of Deling with mass export

SaintBahamut

  • *
  • Posts: 5
    • View Profile
Re: [FF8PC-Steam] Batch extraction of backgrounds
« Reply #7 on: 2018-12-12 19:16:45 »
Yea im still definitely interested

Any idea how much effort would be to make them work with tonberry?

atm trying to approach problem from other side and asemble my own upscaller as there are crapton of examples in python, if i get subpar results i will probably go with letsenhance

Maki

  • 0xBAADF00D
  • *
  • Posts: 624
  • 0xCCCCCCCC
    • View Profile
Re: [FF8PC-Steam] Batch extraction of backgrounds
« Reply #8 on: 2018-12-13 10:06:36 »
Here it is:
https://www.dropbox.com/s/cpuqot7hno0jise/deling-modded.7z?dl=0

It's modified Deling by Myst6re. That Qt is weird, it's like C++, but not c++ O_o
Anyway, open field.fs and then click here:

it will ask you for a folder and then it will start exporting images (software may not response, but it's working):

SaintBahamut

  • *
  • Posts: 5
    • View Profile
Re: [FF8PC-Steam] Batch extraction of backgrounds
« Reply #9 on: 2018-12-13 17:27:56 »
Thanks but it seems some runtime library is needed? Because i got this


Maki

  • 0xBAADF00D
  • *
  • Posts: 624
  • 0xCCCCCCCC
    • View Profile
Re: [FF8PC-Steam] Batch extraction of backgrounds
« Reply #10 on: 2018-12-14 08:08:23 »
Thanks but it seems some runtime library is needed? Because i got this


I have no single idea why this happens. I attached all needed libraries.
Does anyone know how to solve this? I'm Qt first timer...
Anyway, I uploaded the fields for you:
https://www.dropbox.com/s/y3l4tyu5byojfr5/backExport.7z?dl=0

SaintBahamut

  • *
  • Posts: 5
    • View Profile
Re: [FF8PC-Steam] Batch extraction of backgrounds
« Reply #11 on: 2018-12-15 16:46:10 »
Done some tests, literaly results are: good, bad and the ugly

as seen below, results quality vary alot









LeonhartGR

  • *
  • Posts: 2577
  • ~Whatever...~ Enjoy life!
    • View Profile
    • LeonhartGR Productions
Re: [FF8PC-Steam] Batch extraction of backgrounds
« Reply #12 on: 2018-12-16 14:52:56 »
Hmmm... It seems that sometimes the effect is too noisy.

blippyp

  • *
  • Posts: 100
    • View Profile
Re: [FF8PC-Steam] Batch extraction of backgrounds
« Reply #13 on: 2018-12-19 17:53:24 »
First off, I apologize for the length of this message.
tldr; I want to help, MY proposed image example will be at the end.

SaintBahamut, I sure hope you're still around, cause I'd really like to work together on this if you're willing. Like you, I also decided to play this game recently again and was pleasantly surprised to see that mods existed for it (I only ever played it on playstation before, but purchased it recently on steam). And just like you, I was pretty discouraged with the backgrounds among a few other things, but at the moment, mostly the backgrounds. I'm no artist or graphic designer, but I do play with GIMP quite a bit for fun, rebuilding images from scratch without using the original in the final product. I also decided to begin a project to redo these backgrounds as well, but have only been working on this now for a couple of days. It took me about a day to figure out how to extract the backgrounds and the crazy but rather ingenious process required to actually mod the textures due to the efforts of Tonberry (forgot the original designer's name), but two thumbs up for his determination on getting that accomplished.

I certainly won't claim to be very familiar with this entire process yet, so I'm going to explain things as I see it just in case I'm mistaken.

Currently, there are two major mods for textures as far as I'm aware (Eden and AngelWing), I played the game up until just before the part when you first fight Seifer and Edea so far with the AngelWing textures, but only just installed the Eden textures. As far as I understand it, Eden is apparently better, but the hashes are messed up so people just install Eden and use the hashes from AngelWing. I haven't begun playing again to really see what kind of change this has made, but apparently its what people are doing. From what I can tell (and am guessing) both of these mods basically just applied Waifu2 on all the images, I'm guessing Eden did a better job of it and/or also has some other custom edits involved.

Given the images I saw you post, I took one of the ones I thought people probably weren't liking, but also see the potential as to why you've posted this. I don't really understand what you're doing with your images entirely, but there is certainly more detail there than Waifu2 is providing. So I decided to play with one of your images using the process I've been using and I'd like to see if you also think what I'm suggesting has potential.

Before I go further I'll tell you a little about myself and what I've noticed so far from this post. For starters, the background images you're using aren't accurate, and I'll explain that more in a bit, but I promise you won't like what I have to say. I'm currently an unemployed programmer, have been using computers heavily for well over 30 years now, so I'm not a newb by any means, but I am clearly new to modding FFVIII. I have no official graphical design or artist training, but consider myself pretty good at it as far as a hobbyist is concerned despite my lack of training with virtually no artistic skill. So for me, this is a 'fun hobby', I don't claim to be an expert at this.

With that said, from what I've seen so far, I can tell you this: Altering the backgrounds for this game isn't a simple matter of pulling the textures out of the game and running a filter on them. If you haven't gotten a good look at them yet, when you do, you will soon realize that you actually have to 'build' some of these backgrounds using the textures provided, and some of them actually have 2 or more actual backgrounds that need to created and altered. This is due to the animation effects that happen in the game. An example of what I'm talking about is in the archive that was sent to you with the backgrounds. Some of those backgrounds have a map slapped in the center of the screen and clearly can't be used, and that's just one example, they get much trickier than that.

Now having said that, I'm well aware that we would need to build these backgrounds and am willing to do this, I will post an example of one of these images that I've already done as well, I still don't have any clue how many of these backgrounds need to be generated (I'm still extracting the files myself with Unmass, I really wish there was a better process for this, but nothing I've tried so far works, I have downloaded that Deling program posted up above but received the same error you did. I'm sure this would be rather simple to fix, but I'm almost done extracting them with Unmass at this point, so don't see the point in trying to solve another solution that isn't a simple download/install/click and be done with it process).

Okay, okay, enough babbling, I'll now explain/demonstrate how I think I can help you with a little 'how I got here' to begin with.

So the original game's textures are something like 320x200'ish or whatever. Tonberry wants us to enlarge them four times bigger making the average image approx 1280x896, some of the images have large widths, but are always 896 pixels tall as far as I can tell so far. The original graphics for this game are quite nice at the original resolution, the trick is getting them bigger with so little to work with however.

Waifu2 will produce an image like:


Here is your image you posted:


This is a 'current' example of what I planned on doing with my backgrounds using this same image. Personally, I think this is much better than the Waifu2 produced image on it's own, but preference varies. This isn't what I'm suggesting at this time, but what I'm currently going with.


After seeing your image and the detail on it, I decided to run it thru the process I'm using (basically a bunch of filters and layers), hoping that it would produce a better image despite all the noise some of your images were producing, but the results weren't as expected, and certainly wouldn't want to use this either.


As someone suggested previous (sorry, don't remember your name), but I did try combining your image with a Waifu2 layer. It looks better, but overall not by much, so I decided to join your image I made with my image, both run through the filtering/layering process I've been using:


Personally, I think this looks pretty damn good. It adds the details that your images are clearly providing but Waifu2 is absolutely lacking, plus the filtering that I like with my own version.

Each image has to be taken with a grain of salt. This image isn't the best, but it's not bad and with a little effort I can come up with all kinds of variations on how to make this look differently, so there's no need to insist on keeping with the process I'm currently using. I honestly wouldn't be surprised if after a time I came up with a much better process and wanted to just redo all the images, but for now, I think this looks pretty awesome considering it was designed from a 400x224 image.

Take into account that each image can have some TLC applied, in this example, I didn't really like the sky much, but preferred the sky in the Waifu2 image much more, so I masked it into the image, which required some 'manual' work, but it wasn't hard and literally only took a couple of minutes. THIS is the final image I want you to see and consider. There's room to grow here, but I think this looks great (well, for someone who isn't an artist and compared to the Waifu2 image most of us are currently using anyway).


So after all this nonsense, what am I suggesting? I like your images, I want them. From what I understand you're using some commercial filter that I don't have, at the very least, I would love to hear what it is and how you're using it. If you're interested I'd like to discuss this further on Discord or something, create a mega account we can use to combine our efforts, build all the screens we can, and if successful simply provide it as a texture package.

Initially, the hard part as far as I can tell so far is figuring out how to generate all the proper images, which isn't impossible but isn't super easy either. Once we have those, I don't know what you need to do to the images, but I'd like a copy of every image so I can edit them as well. Currently, I'm manually applying the filters and layers, but once a process is decided on, I can turn that entire process into it's own 'filter' in GIMP with a little effort, but even if there's problems with that and I have to do them all by hand, I'm willing to do that, or would at least love to try.

Now I'm gonna make you mad though, but feel it needs to be said. Here is an example of the original game using AngelWing:


This is a pretty complicated background. For one, it's a duplicate (it's during the time when you're about to confront Seifer and Edea for the first time), I haven't verified it yet, but there is another background in the data files for this area and it has a moon and the lights are brighter, I'm assuming it's for any other time other than this moment in the game. See all those people there? They all have images for them as well, and they're really nasty, and I'm guessing there is a lot of this that needs to be fixed up. They've been clearly cleaned up I think, but not by much, there isn't much to go on, which is a huge problem as far as making this game look 'complete'.

I rebuilt this background as well, I'll show you what will happen with it, it looks good I think for the most part, but I would go back and darken the image a bit and/or remove some of the filtering I applied for some of the areas (the bushes mostly, maybe the sidewalks).


There are three 'kinds' of textures here for the npc's standing around, some are horribly blurry, that was due to lazyness on my part, I modified the background and simply ran the rest of the images thru a simple resize in GIMP (didn't even use Waifu2, it takes a while), they're obvious, at the very least, we can make them look like they did in AngelWing though. Look at Squal and Irvine though, those characters are done really well (much better than I can do). I also ran one of those images thru my filter though which you can see on the guards and the two people in the lower left corner the best. IMO, this looks better than what was done by AngelWing, but they still look absolutely horrible now with an updated background.

I don't really have much of a point with these last two images, except that once this can of worms is opened (redoing the backgrounds), all those little details (sprites/animations) in the game really begin to stand out. I suppose the point I'm making here is that doing the backgrounds is only a step, maybe it's all we'll do if you're willing, but those other details will also have to be looked at and will drive people nuts (I may even make an attempt at actually redrawing them myself afterwards) just so all the graphics look and feel the same in the game, might even have to degrade the images of the characters like squal a bit just to make them match the rest of the game.

Anyways, this is a really long message. So I'll stop for now and hope to hear from you in some way or any feedback from anyone would be cool even. Even if SaintBahamut doesn't want to work with me on this, I'm going to continue this myself either way, and with a little time and effort, who knows what I'll come up with. But as it stands, I would really like to use SaintBahamut's images as well, I just think they look much better combined with mine.
« Last Edit: 2018-12-19 18:13:17 by blippyp »

blippyp

  • *
  • Posts: 100
    • View Profile
Re: [FF8PC-Steam] Batch extraction of backgrounds
« Reply #14 on: 2018-12-20 05:39:46 »
I'm realizing that posts on this board are few and far between  :-o
Figured I'd make another edit, much happier with the way this one turned out though.

Waifu Version


SaintBahamut Version


Our Combined Efforts


Considering I'm still doing this by hand and haven't wrapped this into a one-click filter yet (that would be awesome) so there might be some differences, but afaik I used the exact same process for this, however, as a final filter I applied a 'sharpening effect' to the enitre image, which really cut down on the blurryness my image added, overall I think this looks quite good and is far better then the waifu version by far.

Here's the imgur where my pictures are getting posted if anyone else is interested, plus it's easier/faster to update this then wait for a moderator to accept my posts and causes less clutter here, I'll likely keep updating it with pictures until I get bored or get a response from SaintBahamut.

https://imgur.com/a/UzpMdew
« Last Edit: 2018-12-20 14:36:31 by blippyp »

Yagami Light

  • *
  • Posts: 173
    • View Profile
Re: [FF8PC-Steam] Batch extraction of backgrounds
« Reply #15 on: 2018-12-20 16:00:52 »
The latest edit you have shown is not even in the game I dont think, to get them in game you have to download unmass tool extract field files place pupu program inside the field folder with the mapdata next to it then start it up, export the files to a new folder somewhere and you have to upscale all the different pieces individually then import it back in, be warned there will be lines/seams if you aren't careful

SaintBahamut

  • *
  • Posts: 5
    • View Profile
Re: [FF8PC-Steam] Batch extraction of backgrounds
« Reply #16 on: 2018-12-20 18:00:25 »
Grand wall of text

Im completely aware of all technical stuff involved, just was not interested at export know-how yet. Thing im using gives mixed results, starting from 'holy-batman-good' (winhill) to awfull (white seed ship). So overally im bit dissapointed with results, just got lucky with initial attempts. So for now im at trying to find some compromise solution.
And if anyone is interested what im actually using.
https://github.com/xinntao/ESRGAN
Easy to use, just need python, couple of libraries and preferably nvidia card

blippyp

  • *
  • Posts: 100
    • View Profile
Re: [FF8PC-Steam] Batch extraction of backgrounds
« Reply #17 on: 2018-12-20 18:02:44 »
Actually, it is in the game, it's been a while since I fully played it though, so I just verified it in the data files to be sure. I believe it's when you're Laguna and living with what's her name about halfway thru the game.

As for the rest of your comments however, I already explained all of that in one way or another in my first post. The images used at the bottom is the game itself, one using AngelWing and the other with my own altered background, which also required being rebuilt using multiple files like you're suggesting, so I do understand what you mean but it's more time consuming than hard to do. Resizing them all individually is a very bad idea though, as that won't give a HD effect that we're all after. The hard part is determining which files should be included, but joining them is a simple matter, just do it with the originals before resizing them, export the alpha layer of the originals and resize those as well, then re-apply them (altered if need be, which is a very likely possibility as far as images like the one I posted, which should have the characters completely redone in them) before saving the final files for Pupu to import.

blippyp

  • *
  • Posts: 100
    • View Profile
Re: [FF8PC-Steam] Batch extraction of backgrounds
« Reply #18 on: 2018-12-20 21:12:32 »
And if anyone is interested what im actually using.
https://github.com/xinntao/ESRGAN

oh that's awesome, thx! I will certainly give that a go! :)

blippyp

  • *
  • Posts: 100
    • View Profile
Re: [FF8PC-Steam] Batch extraction of backgrounds
« Reply #19 on: 2018-12-21 03:37:43 »
I am having a ton of fun with this, thanks  ;D

Waifu2


My Edit

« Last Edit: 2018-12-21 03:43:27 by blippyp »

blippyp

  • *
  • Posts: 100
    • View Profile
Re: [FF8PC-Steam] Batch extraction of backgrounds
« Reply #20 on: 2018-12-25 05:19:53 »
For anyone who may actually be interested, which may be no one since no one seems to be showing any real interest, but figured I'd post this as an update regardless. I have been continuing my efforts, albeit somewhat slower on developing a mass process for filtering these images for what I consider to be an improved HD quality over WAIFU2 filtering. I still have plenty of images to export from the original data files using Unmass/Pupu, however I have begun developing a filter for automatically processing all the images. I still have to look into turning my filter into a process for mass batch processing, but have certainly made some headway so far and expect the base filter to be done shortly after the holidays. My current filter only requires a few more steps to be completed before adopting it to a mass process and ready to apply on all the images once I've determined which images need to be made, which I'll likely just use the AngelWing mod as a base to start with. Below are two images, the original image from the game at 4x it's original size without any modifications, and the second is my current expectations for my current filtering once it is completed. I have removed a lot of the filtering I was applying previously in hopes of removing some of the line art that was appearing, I think it looks better, but might alter the process again before finally doing a mass filter job. I certainly prefer this one over the last, but feel some of the detail may be missing that could be applied.

Original


Current Automatic Filtering Process


Up until this post, all the editing I've done has been manual, but this process is automatic and therefore can be easily applied to thousands of images requiring only the time needed for my system to process these images for them to be completed once they are collected and the filter I'm making is completed.
« Last Edit: 2018-12-25 13:54:25 by blippyp »

Yagami Light

  • *
  • Posts: 173
    • View Profile
Re: [FF8PC-Steam] Batch extraction of backgrounds
« Reply #21 on: 2018-12-25 10:29:31 »
I want to use ESRGAN with the manga109 dataset, I've downloaded all the programs needed but have no idea how to run it, a tutorial would be nice

blippyp

  • *
  • Posts: 100
    • View Profile
Re: [FF8PC-Steam] Batch extraction of backgrounds
« Reply #22 on: 2018-12-25 20:08:34 »
Considering I've only setup ESRGAN once myself and the fact that I'm hardly a Python expert, I'm far from the person to explain/make a tutorial on how to get it working. In fact, despite the fact that the filter I'm making right now for these images is being made with Python, I've never actually used it before now so I honestly have very little to offer on the subject. What I can tell you though from what I noticed while setting it up for myself, is that it was rather confusing and which version of Python you have installed will drastically alter how you install the necessary libraries for it and which commands/methods you use to install them to get ESRGAN working to begin with. Initially I tried to install it using Python 3.7 and ran into many issues trying with that version. I'm sure someone more familiar with Python can easily explain how to install it properly with that, but for me it was difficult and gave up quickly.

In the end I used 2.7. There were three libraries I needed to install, but can't remember which exactly, but they'll easily come up as you google on how to install each library. Begin with the instructions on the ESRGAN page and from there it will make sense. I used what I believed to be package/library manager for lack for a better description called Pip. Using that, I installed all the libraries necessary to use ESRGAN. While googling on how to install each library for Python 2.7 and using Pip, you should come across some pip commands that you run from the command line that automatically downloads and installs them. I used the same process to install all three libraries, just with slightly different commands referring to each library. Once they were all installed, I simply ran ESRGAN as explained on the developer's page.

Wish I could help you more than that, but that's the general 'how to' on how I installed it myself, hope that helps and hopefully I didn't leave anything out.
« Last Edit: 2018-12-26 03:34:46 by blippyp »

blippyp

  • *
  • Posts: 100
    • View Profile
Re: [FF8PC-Steam] Batch extraction of backgrounds
« Reply #23 on: 2018-12-27 06:31:18 »
Final stages of designing that filter was certainly a much bigger headache than I anticipated, but here are the results so far. The only thing missing from this process currently minus the ability to do a batch filter on many images (which from initial research probably won't be hard), is the initial merging of the ESRGAN and WAIFU2 files, which I use as the base for my filtering, but that's a simple process that I'm certain will cause no headaches and will be part of my process for turning this into a batch process. The real problem now should be finishing exporting the rest of the files and rebuilding the backgrounds that are broken apart, but as I mentioned earlier, I will be using the AngelWing mod as a base for which files to build from so it shouldn't be too hard to at least get a start on that once I've exported them. There are a few minor changes from this filtering than my manual edits also, since there were some steps I did in GIMP that don't actually have a filter for it, but GIMP itself was doing whatever it was doing and I had to try to mimic what I thought GIMP was doing to the images. An example of this would be performing a white balance on an image, so some of the steps have been slightly altered, but I think it's a fair representation, if not even better in some situations, for example my sharpening effect is more effective overall, and I think I prefer my filtered process more tbh.

Original


Filtered


Just as my previous manual edits and as I anticipated, the sky required some masking from the original WAIFU2 image in order to look good, I'm not sure how much of this type of manual work would be required from the images, but it's a relatively simple process, but if I have to do 500 of them then it could obviously take a while regardless. My plan is to save the masks I generate for each image, so when the process is completed, and if I decide to make/use a different filter at any point I can simply click on my batch process and all that sky work and other manual tweaks like that will be automatically done as well with the batch. I am also still anticipating that some of the images will require some other manuals edits, if not even have the filtering on them completely altered to look decent and match the rest of the images, but I won't know more about that until I start trying to actually produce the batch and see what the results are like when I go over each image separately.
« Last Edit: 2018-12-27 06:37:31 by blippyp »

blippyp

  • *
  • Posts: 100
    • View Profile
Re: [FF8PC-Steam] Batch extraction of backgrounds
« Reply #24 on: 2018-12-27 08:56:56 »
Background comparison between the original graphics and Project Eden, Project AngelWing (with latest AllFIX Patch) and my own Upscaling techniques.

Original


Project Eden


Project AngelWIng (including latest AllFIX Patch)


Current blippyp Upscaling


It should also be noted that I also have reshade running in these shots, which tends to make my images darker.

Current blippyp Upscaling with initial skin texture updates as well to match backgrounds
« Last Edit: 2018-12-27 10:59:24 by blippyp »