Author Topic: [PC] External Texture Support - Tonberry: Enhanced (2.04)  (Read 333187 times)

mavirick

  • *
  • Posts: 76
    • View Profile
Re: [FF8PC] External Texture Support - Tonberry: Enhanced (2.04)
« Reply #150 on: 2015-06-23 17:26:54 »
Mavirick, would it possible to have a Tonberry texture support for FFVII steam version since the game works in a similar way ? How much work would it take ?

Yes, I believe so. The FFVII Steam release uses DirectX 9 so I think you could use the exact same D3D9Callback.dll. There is some question as to whether the textures are the same size and are handled the same way (they are written to the GPU upside down in Tonberry, don't ask me why), but assuming they are mostly similar to FFVIII then the same process could be used: use Tonberry debug mode to find in-game textures, upscale them or repaint them, use hash of in-game texture to replace with new texture. I don't own the Steam version of FFVII, but if someone wanted to try Tonberry 2.04 with it I'd be interested in their findings. You'd want to empty \textures, tonberry\hashmap, and tonberry\objmap, set debug_mode=yes in tonberry\prefs.txt, and create a proper tonberry\debug folder structure for saving images.

The biggest problem would be the hashing algorithm, which Omzy wrote and then reinforced by a set of handchosen pixels that seemed to reduce collisions to an acceptable level. This algorithm would probably not work very well with FFVII and its entirely separate set of textures, which would mean tons of collisions and general misery for modders. In a perfect world, you'd want to do what the eventual plan for FFVIII is:

  • Use Tonberry with debug mode on to collect a large majority of in-game textures. The more textures the better--the best way to do this would be to crowdsource it, assigning people different areas to run around and then send all the collected textures to one person. One diligent speedrunner could also collect most of them in a reasonable amount of time, assuming he had the harddrive space (likely 100+GB of images), and if I've learned anything from qhimm it's that you should never underestimate the patience of a determined modder (I'm looking at you, FatedCourage :P).
  • Run some analysis on the set of in-game textures to find an ideal set of pixels for hashing. I've developed a process that I think does an excellent job of this, and was able to reduce collisions among the FFVIII textures I'd collected to ~2% (versus the 20+% in the current hash algo). However, the new hash algorithm was never completed/implemented for a variety of reasons, not the least of which was the release of The Witcher 3  ::)
  • Use pixels from analysis to create an (almost-)optimal hashing algorithm that can replace every texture in the game with whatever modders want without any significant collisions. Use the existing Tonberry structure with hashmaps and a directory for replacement textures.
  • Release the collection of in-game textures with their hashes (or an app for generating the hashes, like BerryMapper) and let modders go to town.
  • Release an updated Tonberry that uses the new hashing algorithm.

There are several other improvements in development (if a somewhat delayed development) for Tonberry, including logic that can search for 1-4 objects in all four corners of a texture so that modders like FatedCourage don't have to trawl the game for every possible combination of character textures to include in the hashmap. The important thing, I think, is that the replacement strategy that Omzy came up with and the texture cache that JeMa and I developed are both iron tight. They can and should be used for other games.

Izuela

  • *
  • Posts: 2
    • View Profile
Re: [FF8PC] External Texture Support - Tonberry: Enhanced (2.04)
« Reply #151 on: 2015-06-24 04:25:40 »
I am having trouble following the installation instructions. I think I got Tonberry in place but I don't understand how to add the mods. Feels like I missed the first half of a conversation, or something. Is there a more detailed tutorial somewhere?

fatboysteak

  • *
  • Posts: 2
    • View Profile
Re: [FF8PC] External Texture Support - Tonberry: Enhanced (2.04)
« Reply #152 on: 2015-06-24 05:49:19 »
Me too I am very confused

FatedCourage

  • *
  • Posts: 337
  • I'll never regret knowing you.
    • View Profile
    • Youtube Channel
Re: [FF8PC] External Texture Support - Tonberry: Enhanced (2.04)
« Reply #153 on: 2015-06-24 07:13:27 »
I am having trouble following the installation instructions. I think I got Tonberry in place but I don't understand how to add the mods. Feels like I missed the first half of a conversation, or something. Is there a more detailed tutorial somewhere?

Most of the mod threads usually tell you what you need to do to make them work. If you use the Rebirth Flame mod, I put instructions on the first page saying where you need to put everything to make it work. And I'll link that page here. http://forums.qhimm.com/index.php?topic=16022.0 And Mcindus and others also put instructions on theirs as well.

I don't own the Steam version of FFVII, but if someone wanted to try Tonberry 2.04 with it I'd be interested in their findings.

I tried it for you. It does pull textures like the backgrounds, text, battle effects and icons, and also character portraits. Doesn't pull any textures from the character models in or outside of battle though. It does give me their creepy little eyes, though.




Quote
One diligent speedrunner could also collect most of them in a reasonable amount of time, assuming he had the harddrive space (likely 100+GB of images), and if I've learned anything from qhimm it's that you should never underestimate the patience of a determined modder (I'm looking at you, FatedCourage :P)

That actually sounds very plausible. Though, I'd be uploading for days. And I'd feel sorry for the person having to sift through all that. Er... I'm a very determined person when it comes to anything I take part in. Perhaps too much at times. ...I also lead a very boring life.  8-)

mavirick

  • *
  • Posts: 76
    • View Profile
Re: [FF8PC] External Texture Support - Tonberry: Enhanced (2.04)
« Reply #154 on: 2015-06-24 13:58:43 »
I tried it for you. It does pull textures like the backgrounds, text, battle effects and icons, and also character portraits. Doesn't pull any textures from the character models in or outside of battle though. It does give me their creepy little eyes, though.

Hm, that's interesting that you don't get character models... Doesn't seem right to me. I don't know why they would work any other way. It's also kind of a deal-breaker if there's no way around it. Did you have all the debug directories working? They could be an unsupported size or something.

That actually sounds very plausible. Though, I'd be uploading for days. And I'd feel sorry for the person having to sift through all that. Er... I'm a very determined person when it comes to anything I take part in. Perhaps too much at times. ...I also lead a very boring life.  8-)

There wouldn't have to be any sifting, necessarily, just some code to remove any duplicates from the set and then pixel analysis on what's left over. And as far as your determination goes, I'm certain I'm not alone in saying that we're thankful for it around here ;)

Kaldarasha

  • *
  • Posts: 2437
  • Prince of Model Editing
    • View Profile
Re: [FF8PC] External Texture Support - Tonberry: Enhanced (2.04)
« Reply #155 on: 2015-06-24 14:33:09 »
For FF7 are only the backgrounds and the Menu textures interesting. The rest can be replaced directly.

FatedCourage

  • *
  • Posts: 337
  • I'll never regret knowing you.
    • View Profile
    • Youtube Channel
Re: [FF8PC] External Texture Support - Tonberry: Enhanced (2.04)
« Reply #156 on: 2015-06-24 20:12:51 »
Hm, that's interesting that you don't get character models... Doesn't seem right to me. I don't know why they would work any other way. It's also kind of a deal-breaker if there's no way around it. Did you have all the debug directories working? They could be an unsupported size or something.

I tried a few times and got nothing. Hard to say... Yeah. I opened them all up. Unsupported got FMV shots, and nomatch2 had more eyes in it. But no character or enemy textures in any.

Quote
There wouldn't have to be any sifting, necessarily, just some code to remove any duplicates from the set and then pixel analysis on what's left over. And as far as your determination goes, I'm certain I'm not alone in saying that we're thankful for it around here ;)

Doesn't sound too bad then. Ah, thanks. I'm glad you and others are. :D But compared to Omzy's Project Eden and the HorizonPack from Mcindus, Rebirth Flame seems pretty tame other than gathering the textures/codes.

mavirick

  • *
  • Posts: 76
    • View Profile
Re: [FF8PC] External Texture Support - Tonberry: Enhanced (2.04)
« Reply #157 on: 2015-06-24 20:49:38 »
I tried a few times and got nothing. Hard to say... Yeah. I opened them all up. Unsupported got FMV shots, and nomatch2 had more eyes in it. But no character or enemy textures in any.

Hm... is it possible that there aren't any character textures? I haven't played in a long while, but just looking back at screenshots it appears almost as if the characters are just polygons filled with solid colors. Lighting and shaders gives it the gradient look, but I'm not seeing any actual textures:




I'm not entirely sure that using solid colors instead of wrapped textures would change the way Tonberry works, but it's a possible explanation.

FatedCourage

  • *
  • Posts: 337
  • I'll never regret knowing you.
    • View Profile
    • Youtube Channel
Re: [FF8PC] External Texture Support - Tonberry: Enhanced (2.04)
« Reply #158 on: 2015-06-24 21:28:04 »
Hm... is it possible that there aren't any character textures? I haven't played in a long while, but just looking back at screenshots it appears almost as if the characters are just polygons filled with solid colors. Lighting and shaders gives it the gradient look, but I'm not seeing any actual textures:




I'm not entirely sure that using solid colors instead of wrapped textures would change the way Tonberry works, but it's a possible explanation.

That was my initial thought also. When I saw how people were making the models and such, I questioned whether or not it's just colored polygons as well. So you might be right. Still, Tonberry does/eventually can work to a certain degree on most other things. And I guess it's up to others whether they see potential in any of it or not. Also, since I don't have a save near the world map, I can't test anything there.

Aavock

  • *
  • Posts: 217
    • View Profile
Re: [FF8PC] External Texture Support - Tonberry: Enhanced (2.04)
« Reply #159 on: 2015-06-25 02:38:48 »
It's all polygons yes, the only textures for field and battles models are eyes, mouth, Barret's tatoo, Cloud's belt and such. You can change/replace them using Kimera.

mavirick

  • *
  • Posts: 76
    • View Profile
Re: [FF8PC] External Texture Support - Tonberry: Enhanced (2.04)
« Reply #160 on: 2015-06-26 00:27:53 »
For FF7 are only the backgrounds and the Menu textures interesting. The rest can be replaced directly.
It's all polygons yes, the only textures for field and battles models are eyes, mouth, Barret's tatoo, Cloud's belt and such. You can change/replace them using Kimera.

So am I correct in assuming Kimera cannot replace backgrounds and menu textures?

The only issue I foresee is using custom textures through previous mods in conjunction with Tonberry: if Tonberry is using a hashing algorithm based on FFVII vanilla textures but then gamers start throwing different combinations of modded textures at it, modders will not be able to predict and reproduce (and thereby try to circumvent) collisions. Someone's custom texture could very well hash to the same thing as a vanilla texture, which would cause Tonberry to try to replace it and things would start getting weird. Ideally, you'd use Tonberry for everything or not at all--additional textures won't slow it down significantly.

Obviously this is all further down the line, just something to think about. If Tonberry is to be used with FFVII, its addition to the game needs to be worth the slowdown (which admittedly should be very minor for modern computers) and the effort required to adapt existing texture mods to use it.

EDIT: As an additional and probably far-fetched thought, it's possible that the D3D9Callback interceptor could be modified to catch FFVII's calls to fill in polygons. If those could be replaced with calls to apply a texture to the same polygon... well, that would be very interesting indeed.
« Last Edit: 2015-06-26 00:34:52 by mavirick »

Kaldarasha

  • *
  • Posts: 2437
  • Prince of Model Editing
    • View Profile
Re: [FF8PC] External Texture Support - Tonberry: Enhanced (2.04)
« Reply #161 on: 2015-06-26 06:18:55 »
Quote
As an additional and probably far-fetched thought, it's possible that the D3D9Callback interceptor could be modified to catch FFVII's calls to fill in polygons. If those could be replaced with calls to apply a texture to the same polygon... well, that would be very interesting indeed.

That won't work this way. Even if you replace a specific color with a specific texture you will run in tons of problems. As example battle models have gradients. How do you want to replace them?
Texturing models is quite easy: giving the mesh a UV map and a texture and then place everything in back into the lgp (obviously it is more complicated in detail). I should mention that the directx9 renderer has a black texture bug which happens for no reasons.
If Tonberry only looks for menu textures and backgrounds it would be enough.

Stiverton

  • *
  • Posts: 4
    • View Profile
Re: [FF8PC] External Texture Support - Tonberry: Enhanced (2.04)
« Reply #162 on: 2015-06-28 01:12:23 »
Hey I am having some trouble with Carbuncle's texture.  There is a description of the problem and a picture in the Lunar Cry thread here: http://forums.qhimm.com/index.php?topic=15977.msg229570#msg229570

EDIT: Figured it out.
« Last Edit: 2015-07-01 04:01:44 by Stiverton »

OhNoJoeVo

  • *
  • Posts: 1
    • View Profile
Re: [FF8PC] External Texture Support - Tonberry: Enhanced (2.04)
« Reply #163 on: 2015-07-01 03:14:18 »
Hey guys! I seem to have an issue running Tonberry with FF8.

I've downloaded Final Fantasy 8, and ran it vanilla no problem.  I installed Tonberry and some mods and my Steam doesn't get past "Preparing to Launch Final Fantasy 8."   I removed the mod and the game runs fine.  I've also put back just Tonberry, and this is causing the game not to launch.  When I say this, the launcher doesn't even come up.

I've installed the Visual C++ 2013 package.
Direct X is up to date.
I have a Nvidia 765m with fully updated drivers.
My Intel Graphics driver is also fully updated.
I have disabled steam overlay both for FF8 and globally just to make sure.

I have installed the mod properly to C:\Program Files (x86)\Steam\SteamApps\common\FINAL FANTASY VIII

in that folder is the tonberry folder, texture folder and all else that comes with the mod..

I've got collisions and hash2map in the tonberry folder.
hashmaps in the hashmap folder
obj files in objmap folder
prefs file is untouched, but I confirmed that it is good.

This is my favorite childhood game D;.  I appreciate the help guys

EDIT:  I have removed ONLY the d3d9.dll so I could see if FF8 then launches... which it does; so hopefully that information helps out D;.
« Last Edit: 2015-07-01 11:19:53 by OhNoJoeVo »

olearyf2525

  • *
  • Posts: 255
    • View Profile
Re: [FF8PC] External Texture Support - Tonberry: Enhanced (2.04)
« Reply #164 on: 2015-07-06 21:33:23 »
Ah got it to work, copied MSVCP120.dll and MSVCR120.dll from another steam game which came with them in the folder.
Yes the TT game now looks a treat!
Thanks for your hard work guys!

I had to copy these two files from my League of legends folder to get ff8 launching after installing tonberry 2.04.

olearyf2525

  • *
  • Posts: 255
    • View Profile
Re: [FF8PC] External Texture Support - Tonberry: Enhanced (2.04)
« Reply #165 on: 2015-07-06 21:35:15 »
Hey guys! I seem to have an issue running Tonberry with FF8.

I've downloaded Final Fantasy 8, and ran it vanilla no problem.  I installed Tonberry and some mods and my Steam doesn't get past "Preparing to Launch Final Fantasy 8."   I removed the mod and the game runs fine.  I've also put back just Tonberry, and this is causing the game not to launch.  When I say this, the launcher doesn't even come up.

I've installed the Visual C++ 2013 package.
Direct X is up to date.
I have a Nvidia 765m with fully updated drivers.
My Intel Graphics driver is also fully updated.
I have disabled steam overlay both for FF8 and globally just to make sure.

I have installed the mod properly to C:\Program Files (x86)\Steam\SteamApps\common\FINAL FANTASY VIII

in that folder is the tonberry folder, texture folder and all else that comes with the mod..

I've got collisions and hash2map in the tonberry folder.
hashmaps in the hashmap folder
obj files in objmap folder
prefs file is untouched, but I confirmed that it is good.

This is my favorite childhood game D;.  I appreciate the help guys

EDIT:  I have removed ONLY the d3d9.dll so I could see if FF8 then launches... which it does; so hopefully that information helps out D;.

Just do what me and skillster did. If you try to launch ff8 from the game directory itself it should tell you that you are missing msvcp120.dll. So I did a C drive search for those 2 files, copied and pasted them in when i found them in my LOL directory and its working now.

Aavock

  • *
  • Posts: 217
    • View Profile
Re: [FF8PC] External Texture Support - Tonberry: Enhanced (2.04)
« Reply #166 on: 2015-07-12 21:42:45 »
I just tried using Tonberry v2.04 and Berrymapper with FFVII steam version, the debug mode works great and I pulled off Cloud's avatar, the picture is now 4x the size (512x512px) *.png file and the texture path is "textures\cl\cloud\cloud_0.png".

hashmap.csv:
Code: [Select]
cloud_0,18446744073709551615
objmap.csv:
Code: [Select]
cloud_0,4294967295,4294967295
Unfortunately when loading my save file, Cloud's avatar remains unreplaced and several fields layers are red or missing:


Mcindus

  • *
  • Posts: 916
  • Artist, Modder, Musician.
    • View Profile
    • Lunatic Pandora
Re: [FF8PC] External Texture Support - Tonberry: Enhanced (2.04)
« Reply #167 on: 2015-07-24 23:33:15 »
I just tried using Tonberry v2.04 and Berrymapper with FFVII steam version, the debug mode works great and I pulled off Cloud's avatar, the picture is now 4x the size (512x512px) *.png file and the texture path is "textures\cl\cloud\cloud_0.png".

hashmap.csv:
Code: [Select]
cloud_0,18446744073709551615
objmap.csv:
Code: [Select]
cloud_0,4294967295,4294967295

Are there by chance large areas of black in your 512x512 image you're hashing?  If so, it's not looking at enough pixels and is trying to replace anything that uses a transparency mask (so you get a bunch of black boxes everywhere)

From what I remember, the objmap codes that you're showing are exactly that - ones that are trying to replace every instance of a 'black' or transparency texture.  These I don't have a workaround for, but are what Omzy created the hash2map algorithm and collisions.csv exactly for these instances.   It's why I can't make the intro scenes HD in FF8 - every block of text including the squaresoft logo is using that same objmap code you provided.  Might even be the same hashcode too. 

Omzy would know more about how to isolate the pixels and then what algo to run to get the longer codes.

It's really a shame, because this is the graphic I made for the new Tonberry for whenever we get an actual release with the new hash algorithms.
« Last Edit: 2015-07-24 23:35:02 by Mcindus »

Aavock

  • *
  • Posts: 217
    • View Profile
Re: [FF8PC] External Texture Support - Tonberry: Enhanced (2.04)
« Reply #168 on: 2015-07-25 03:49:53 »
Nice picture Mcindus ;D

The only texture I wanted to import is Cloud's avatar and there's no large black areas at all; I've only changed the brightness for testing purpose:



The avatar still unreplaced in-game no matter what.

Mcindus

  • *
  • Posts: 916
  • Artist, Modder, Musician.
    • View Profile
    • Lunatic Pandora
Re: [FF8PC] External Texture Support - Tonberry: Enhanced (2.04)
« Reply #169 on: 2015-07-25 20:25:42 »
Nice picture Mcindus ;D

The only texture I wanted to import is Cloud's avatar and there's no large black areas at all; I've only changed the brightness for testing purpose:

The avatar still unreplaced in-game no matter what.

Can you send me the original bitmap that tonberry dumped for this?  If you're using FF7, try to remove hash2map.csv and collisions.csv and see if that helps?

mavirick

  • *
  • Posts: 76
    • View Profile
Re: [FF8PC] External Texture Support - Tonberry: Enhanced (2.04)
« Reply #170 on: 2015-07-25 21:53:35 »
It's really a shame, because this is the graphic I made for the new Tonberry for whenever we get an actual release with the new hash algorithms.

I lost all my textures when I uninstalled/reinstalled the game :( They were in my tonberry directory and uninstalling the game wiped the whole directory instead of just deleting the game files. I tried to recover them but I guess the reinstall recycled the memory so they were all corrupt.

What I really need is the tonberry dump from an entire playthrough. It would be a ton of images and the playthrough would be frustrating cause debug mode slows things down, but the texture analysis algorithms are written and with a large enough texture set I could generate a near perfect hash. The only thing thing to solve would be hardcoding some pixels in for the animated textures like characters blinking. One solution might be a way to designate additional pixels to look at for certain textures, so that modders could fix collisions without having to generate a whole new hashing algorithm.

Racthoh

  • *
  • Posts: 1
    • View Profile
Re: [FF8PC] External Texture Support - Tonberry: Enhanced (2.04)
« Reply #171 on: 2015-09-01 16:17:45 »
hello, i'm not sure if this is the right place to ask but i am having issue with 2 different mods sharing the same folder name in textures. the mods are project eden and seed reborn, both mods have a texture folder labeled gf and i am not sure how to proceed.

am i supposed to put extra folders in the texture folder to separate the textures of different mods? example [ \FF8\textures\Project_Eden\gf ] and [ \FF8\textures\SeeD_Reborn\gf ] instead of  [ \FF8\textures\gf ] ? or maybe can  merge all the folders inside both gf folders to the same gf folder?

edit: my question was answered in the tutorial video, should've watched the entire thing instead of pausing halfway.
« Last Edit: 2015-09-01 16:26:11 by Racthoh »

mavirick

  • *
  • Posts: 76
    • View Profile
Re: [FF8PC] External Texture Support - Tonberry: Enhanced (2.04)
« Reply #172 on: 2015-10-27 02:17:47 »
I just tried using Tonberry v2.04 and Berrymapper with FFVII steam version, the debug mode works great and I pulled off Cloud's avatar, the picture is now 4x the size (512x512px) *.png file and the texture path is "textures\cl\cloud\cloud_0.png".

hashmap.csv:
Code: [Select]
cloud_0,18446744073709551615
objmap.csv:
Code: [Select]
cloud_0,4294967295,4294967295
Unfortunately when loading my save file, Cloud's avatar remains unreplaced and several fields layers are red or missing:

[img...]

Sorry I never answered this 5way, I've been stretched thin on other projects for the past few months.

I'm pretty sure the problem you had with FF7 is the hashing function Tonberry uses (which is optimized for FF8) and especially the few hardcoded hashes used mostly for menu textures and the like. Many of the menu textures hash to the same thing and differ only by color (which Omzy's original hash function ignores) so there had to be special accommodations programmed directly into d3d9Callback.dll. It could also have to do with the texture size and resize code.

For the most part, if Tonberry can pull the texture into the debug folder then it can also replace it. If (and it's looking like the ball may get rolling again) we get a new hashing algorithm working in FF8 then we can look into applying the same process to FF7.

Aavock

  • *
  • Posts: 217
    • View Profile
Re: [FF8PC] External Texture Support - Tonberry: Enhanced (2.04)
« Reply #173 on: 2015-10-27 03:16:17 »
Thanks for the answer mavirick; yes, Tonberry does debug all kinds of textures so it was looking promising at first. Besides of the d3d9Callback.dll, does berrymapper needs to be optimized too for FFVII?

mavirick

  • *
  • Posts: 76
    • View Profile
Re: [FF8PC] External Texture Support - Tonberry: Enhanced (2.04)
« Reply #174 on: 2015-10-27 04:53:01 »
Thanks for the answer mavirick; yes, Tonberry does debug all kinds of textures so it was looking promising at first. Besides of the d3d9Callback.dll, does berrymapper needs to be optimized too for FFVII?

Basically, the method I've tested as a replacement for the hashing algorithm involves finding the set of the most "important" or "telling" pixels to differentiate between textures in the game, then packing their values into any good hashing algorithm. The one I've been using is Murmur2 because it's super fast and collision resistant, but really any hashing function will do.

You're basically trying to solve an optimization problem: maximize the speed of the hash (which is effectively the number of different pixels you use) while also minimizing the number of collisions (which show up as glitching textures in game). Obviously you could use every pixel and have zero collisions but it would be too slow, or use only one pixel and it would be very fast but have a ton of collisions. Theoretically, there is a perfect set of the minimum number of pixels needed to result in 0 collisions. This pixel set is based on the full texture set used by the game, and is--for all intents and purposes--impossible to find. But you can try to get close. Omzy hacked out a pretty good hashing process for FFVIII using some good ole' fashioned trial and error and that's the one we have today, but his process is virtually useless for other games (like FFVII).

TL;DR
All this is to say that yes, everything would have to change for FFVII--because the game uses a totally different texture set, Tonberry would need to hash a different pixel set. Berrymapper just uses the same hashing process as Tonberry to generate the hashes for the replaced and replacement textures. If the new hash function is completed it will come with a new Berrymapper as well. To make Tonberry game-independent, it would have to take the pixels used in the hash as an input (probably just a plaintext file), and the same goes for Berrymapper.
« Last Edit: 2015-10-27 04:56:14 by mavirick »