Author Topic: [PC] External Texture Support - Tonberry (v1.5)  (Read 251021 times)

Omzy

  • *
  • Posts: 205
    • View Profile
[PC] External Texture Support - Tonberry (v1.5)
« on: 2014-06-12 01:11:53 »
Tonberry v1.5
External Texture Support for Final Fantasy VIII
*Beta version, please let me know if there is something preventing you from using it


                   

This drop-in package allows:
I) Support for Hi-Res external textures (field files for now) to 4x original size in an FF8\textures directory
II) Future support for ANY texture (characters, monsters, battles, world map, cards, spells, summons, menus, and just about anything else you can think of)

Instructions (for players):
1. Extract Tonberry_v1.x.rar into your FF8 directory (typically 'C:\Program Files (x86)\Final Fantasy VIII')
2. Make sure to drop in the most up-to-date hash map files, from hashmap_v1.x.rar.
3. Download and place your properly formatted textures from their respective projects into the 'textures' folder (see below)
4. Install VC++ 2010 Redistributable (x86)
5. Play!
6. (Optional) If your Windows installation is not on drive C, change your prefs.txt file 'drive_letter' from C to your drive letter. If you're using textures of different sizes, change 'resize_factor' in prefs.txt.

Formatting Textures (for modders creating texture packs):
1. Create 4x textures with dimensions exactly 4 times the width and 4 times the height of the originals (PNGs supported only).
2. For field files, use PuPu to export and import your images (PuPu Updated and Working).
3. For non-field textures, PuPu will not work, and they will have to be renamed by hand or custom batch scripts.
4. Most textures won't require this, but field files have pages. If a field texture has a width of 1664 it has 13 pages (each 128x256 => 512x1024) and if it has a width of 1536 it has 12 pages. PuPu will automagically slice them up for you and name each page properly starting with a suffix of _0 and ending at _11 or _12 (_13 to _25 for 4-bit indexed textures).
5. Ensure your textures have the proper folder structure with 'textures\firsttwoletters\texturename\texturefile.png\'. E.g. for bghall_1 => 'textures\bg\bghall_1\bghall_1_0.png' for the first page, 'textures\bg\bghall_1\bghall_1_1.png' for the second page, etc. If you have a texture that doesn't require pages, the format would be the same, except with the page suffix '_0' (note that this has changed). E.g. mc00 (cards) => 'textures\mc\mc00\mc00_0.png'.

Adding Support for Other Types of Textures:
->How Tonberry Works
As mentioned above, Tonberry currently only recognizes field files. This is because in order to replace textures, it must read the original paletted textures from directx, generate a key that identifies the texture, and then lookup the texture in the '/textures' folder. Generating the key takes a lot of work on my end. I have to have up to 16 versions of each texture colored with each possible palette from the original source files. I've written programs to generate keys for field files, but it will take some time to support each additional file type. If you have a project that demands support for another file type, you can make it happen if you have some basic programming experience. If not, you'll have to convince your fellow modders to do that work for you if I can't.

Here's how to add support:
1. Find the game source files that contain the type of textures you are working on. They might be LZS compressed, have weird file extensions like md5, bin, etc, but they are all binary files of zeros and ones when it comes down to it.
2. Decode the files using the wiki documentation or asking around. Myst6re has apparently decoded most of them, as is evidenced in Deling (sorry if that's negative publicity myst but its true!). Also, contribute new knowledge to the wiki if you can along the way.
3. Find out if the texture type uses palettes (IMPORTANT!)
4. Write a simple program in python/c++/your favorite language that takes the source game files and exports the textures exactly as they are in the file. No need for map files or anything that will map the blocks to the screen, just dump the image data from the file (mostly applies to fields I think).
5. Export every possible paletted version of each texture. This means that if there is an area in the file with palettes, instead of finding which palettes go to which blocks in the image data, literally export an entire image that uses 1 palette, and do that for every palette.
6. Optimize your program to only export palettes that are used. For example, if you have 3 palettes pink, green, and yellow, but the blocks of the file only make references to pink and yellow palettes and never actually use the green palette, that version of the texture will NEVER be loaded to memory anyway, so don't export it.
7. Format your textures in the manner described above but with suffixes for each palette number. For example, palette 12 on mc00 would be mc00_0_12.png. There will be 2 suffixes including the page. For example, for the 3rd page and 8th palette of bggate_1, it would be bggate_1_2_7.png.
8. You now have a directory full of images that is ready to be hashed.
9. RAR it up and send it to me and I should have it working in short order.
10. Be prepared to make mistakes at any point, they are inevitable, but persevere in the name of art!

Examples of paletted texture files (what your output should look like):


*Please DO NOT post any direct links to files, always link to this forum thread. Especially, do not make any links to adfly or any other ad site
Download: Tonberry_v1.5.rar Mirror 1
Download: hashmap_v1.3.rar Mirror 1
Download: Tonberry_s1.5.rar (SOURCE CODE ONLY)

*hashmap contains most updated hash files, tonberry contains most updated DLLs*
*My policy on my mods is always that you have complete freedom to take them, use them, modify them, and even release derivative works--just give a line of credit in your release and do not draw any monetary profit from my links or releases. I think if you make something, including a derivative work, it is fine to solicit donations, but I encourage you to retain these same policies of openness for whatever you make so we can always further the modding community on every front.

Credits:
-Matthew Fisher at Stanford University for developing the original d3d9Callback interceptor: http://graphics.stanford.edu/~mdfisher/D3D9Interceptor.html
-Aali for developing the original work that allowed me to replace textures in FF7 (his OpenGL driver and Palmer, which PuPu is based on)
-Myst6re for her Deling field editor, which is quite an amazing tool (http://forums.qhimm.com/index.php?topic=13050.0)
-yarLson and sl1982 for their work on the FF7 field backgrounds, EQ2Alyza for her battle scene enhancements
-Salk for keeping me motivated to stay crunchy!

*Known issues:
-Memory issues--may crash after extended use. Temporary solution: Get RAMRush and set it to automatically optimize.
-Lag during some summons or other events that resolves by itself
-Possible hash collisions resulting in glitches where random textures are replaced by other textures.
-Works with the Steam release of FF8 only. I don't have any impetus to try to make it work with anything else, but I have posted the source above.

Debug Version
How to use:
Note the 'debug0' folder inside the 'tonberry' folder with category subfolders inside. If you rename this folder to 'debug' (erase the 0), Tonberry will dump all textures into these folders while your game is running. Be careful! You may amass gigs of textures in them if you are not aware and are just playing normally. Have a save prepared at a certain point and only play for a few minutes, depending on your hard drive size. To turn this feature off, simply rename the 'debug' folder to anything else like 'debug0'. Alternatively, you can rename only some of the subfolders to turn specific types off. Subfolder categories:
-unsupported: size/format is not yet supported (256x256 currently only supported)
-nomatch: not yet hashed, but will be in the future with modder help
-nomatch2: hashed once, failed second round of hashing (shouldn't happen)
-noreplace: hashed and intended to be replaced but the new texture failed to load or didn't exist
-replaced: successfully replaced
-error: texture not otherwise accounted for (shouldn't happen)

Next on my agenda:
1. Project Eden, the FacePalmer of FF8
2. Help people build support for other texture types
« Last Edit: 2018-04-24 17:21:07 by Covarr »

Mcindus

  • *
  • Posts: 929
  • Artist, Modder, Musician.
    • View Profile
    • Lunatic Pandora
Re: [REL] Tonberry v1.0 - External Texture Support for FF8
« Reply #1 on: 2014-06-12 10:29:42 »
:O

I'm speechless.

We're not worthy.


ps. - you can use Omega to convert a bunch of the .TEX and .TIM files to BMPs (the ones in menu.fs, anyway) for easy editing without having to write your own program ;)
unless i misunderstood that part...

LeonhartGR

  • *
  • Posts: 2577
  • ~Whatever...~ Enjoy life!
    • View Profile
    • LeonhartGR Productions
Re: [REL] Tonberry v1.0 - External Texture Support for FF8
« Reply #2 on: 2014-06-12 10:58:24 »
This is only for the Steam version right?

Omzy

  • *
  • Posts: 205
    • View Profile
Re: [REL] Tonberry v1.0 - External Texture Support for FF8
« Reply #3 on: 2014-06-13 04:23:25 »
Updated PuPu today to correctly import textures to be used with Tonberry. I also had to change the hash files to reflect the changes. Main post has been updated with new hash files in hashmap_v1.0.rar. If I have to change hash files in the future, which will happen a lot I anticipate, the most updated will be there. I hope not to have to change the DLL very often at all, until I implement the trigger pre-loads.
This is only for the Steam version right?
Yes, it only supports directx 9, which is what Steam uses to draw the graphics and what made this possible.

Edit: Here's a screenshot comparison, since you asked so nicely Leonhart  :-P Keep in mind, this is just a 10 second bilinear resize in photoshop, this won't come close to fractal
« Last Edit: 2014-06-13 05:13:21 by Omzy »

EQ2Alyza

  • 7th Heaven Crew
  • Global moderator
  • *
  • Posts: 3200
  • Dilly-Dally Shilly-Shally
    • View Profile
    • EQ2Alyza - YouTube Channel
Re: [REL] Tonberry v1.0 - External Texture Support for FF8
« Reply #4 on: 2014-06-13 06:17:53 »
The hashmap link is sending to a PuPu download. Is this correct?

LeonhartGR

  • *
  • Posts: 2577
  • ~Whatever...~ Enjoy life!
    • View Profile
    • LeonhartGR Productions
Re: [REL] Tonberry v1.0 - External Texture Support for FF8
« Reply #5 on: 2014-06-13 10:48:06 »
Thanks to all! :)

Omzy

  • *
  • Posts: 205
    • View Profile
Re: [REL] Tonberry v1.0 - External Texture Support for FF8
« Reply #6 on: 2014-06-13 12:40:49 »
The hashmap link is sending to a PuPu download. Is this correct?
Oops, fixed!

EQ2Alyza

  • 7th Heaven Crew
  • Global moderator
  • *
  • Posts: 3200
  • Dilly-Dally Shilly-Shally
    • View Profile
    • EQ2Alyza - YouTube Channel
Re: [REL] Tonberry v1.0 - External Texture Support for FF8
« Reply #7 on: 2014-06-13 13:08:11 »
Thanks. I pretty much have this figured out I think. I'll post a screen using the fractal methods from FacePalmer.

Mirrorman95

  • *
  • Posts: 144
    • View Profile
    • Mirrorman95 - YouTube Channel
Re: [REL] Tonberry v1.0 - External Texture Support for FF8
« Reply #8 on: 2014-06-13 15:12:47 »
Awesome, Steam FF8 support! Can we use this tool to port Magochocobo's Character model retextures into the game?

EQ2Alyza

  • 7th Heaven Crew
  • Global moderator
  • *
  • Posts: 3200
  • Dilly-Dally Shilly-Shally
    • View Profile
    • EQ2Alyza - YouTube Channel
Re: [REL] Tonberry v1.0 - External Texture Support for FF8
« Reply #9 on: 2014-06-13 16:41:48 »
For these images, I used the same upscale and filter/fractal methods from FacePalmer, except that I upped the smoothness to 75% just for testing purposes. It's a bit washed out in fullscreen, so I'd recommend keeping the 20-30 range.




I read about the black lines/borders somewhere, but now I can't find it again. Anything I can do about them?
« Last Edit: 2014-06-13 16:43:30 by EQ2Alyza »

Omzy

  • *
  • Posts: 205
    • View Profile
Re: [REL] Tonberry v1.0 - External Texture Support for FF8
« Reply #10 on: 2014-06-13 18:23:03 »
I read about the black lines/borders somewhere, but now I can't find it again. Anything I can do about them?
This occurs because the fractal method doesn't deal with edges well. If part of the texture has an edge that is next to blackness, the edge warps very slightly if you look closely. That's why it becomes so labor intensive and difficult without automation. You have to combine each sub-texture together to make all the different permutations of the fullscreen image and then slice each subtexture back out of the resized versions. I am running out of time because I have a big exam next month, so I'm going to try to start and finish Project Eden by the end of next week. An optimistic deadline, I know  :-X

Edit:
Awesome, Steam FF8 support! Can we use this tool to port Magochocobo's Character model retextures into the game?
Yes, but only once support is added. There are a couple problems with that right now, as you'll see in these images:

The left is the original and the right 2 are from Magochocobo. Notice that his mapping is very different than what's in the game's memory. These textures will need to be remapped to the appropriate sections to match the picture on the left. Also note that the size of the left image is 128x128. A 4x resize would be 512x512, not 1024x1024 like most other textures. This will also need to be changed. If all that is fixed and after support has been built in there's no reason it wouldn't work.
« Last Edit: 2014-06-13 21:27:35 by Omzy »

Mcindus

  • *
  • Posts: 929
  • Artist, Modder, Musician.
    • View Profile
    • Lunatic Pandora
Re: [REL] Tonberry v1.0 - External Texture Support for FF8
« Reply #11 on: 2014-06-14 00:54:16 »
This is so cool! :)

I'm working together with Omzy for a release of Triple Triad HD as well as attempting to re-create aspects of SeeD v2.0 for the Steam release!

I had almost given up hope.  I just had to finish mixing my bands Album, put it up on iTunes/Spotify, etc. and keep working on my personal goals until someone had enough patience, fortitude, and brilliance to put this together!

Thank you Omzy!

mmmmmmmm...... mods.

Aavock

  • *
  • Posts: 303
    • View Profile
Re: [REL] Tonberry v1.0 - External Texture Support for FF8
« Reply #12 on: 2014-06-14 02:59:45 »
Most impressive Omzy, thank you for your hard work; also, I'll be working on some "HD" Triple Triad textures as well. :mrgreen:

I've just recently noticed one thing, if you have the linear filtering option on (steam version) the TT cards textures will be filtered so my question is why is this texture filtered and not the TT board one for example (wich is exactly the same kind of texture file) ? Is it possible to get linear filtering on other textures like the board, board card selection, "you win", "draw" texts etc. ?
« Last Edit: 2014-06-14 03:01:28 by 5way »

Kaldarasha

  • *
  • Posts: 2449
  • Prince of Model Editing
    • View Profile
Re: [REL] Tonberry v1.0 - External Texture Support for FF8
« Reply #13 on: 2014-06-14 04:47:54 »
I would say the cards are 3D objects, that's why linear filtering can work on them.

Shunsq

  • *
  • Posts: 142
  • 20 years to mod a game, that's insanely long
    • View Profile
Re: [REL] Tonberry v1.0 - External Texture Support for FF8
« Reply #14 on: 2014-06-14 09:58:06 »
Hi Omzy,
Thank you for your dedication to this program. So you need the textures and the palette index for each one? I can do something i think.

Omzy

  • *
  • Posts: 205
    • View Profile
Re: [REL] Tonberry v1.0 - External Texture Support for FF8
« Reply #15 on: 2014-06-14 20:22:03 »
Thanks for the props guys! I have been thinking about this project ever since I made PuPu a couple years back.
So you need the textures and the palette index for each one? I can do something i think.
Yes, the textures properly named in folders is basically it. I may release a debug version of the DLL that allows people to dump the textures straight from memory instead of decoding all these file formats, which should speed things up suppport wise.

LeonhartGR

  • *
  • Posts: 2577
  • ~Whatever...~ Enjoy life!
    • View Profile
    • LeonhartGR Productions
Re: [REL] Tonberry v1.0 - External Texture Support for FF8
« Reply #16 on: 2014-06-15 03:45:49 »

Omzy

  • *
  • Posts: 205
    • View Profile
Re: [REL] Tonberry v1.0 - External Texture Support for FF8
« Reply #17 on: 2014-06-15 11:45:02 »
Omg Leonhart what the hell is that lol

LeonhartGR

  • *
  • Posts: 2577
  • ~Whatever...~ Enjoy life!
    • View Profile
    • LeonhartGR Productions
Re: [REL] Tonberry v1.0 - External Texture Support for FF8
« Reply #18 on: 2014-06-15 13:52:50 »

Omzy

  • *
  • Posts: 205
    • View Profile
Re: [REL] Tonberry v1.0 - External Texture Support for FF8
« Reply #19 on: 2014-06-15 15:27:27 »
Wish I could change my avatar to that now

magochocobo

  • *
  • Posts: 68
    • View Profile
    • My Gallery
Re: [REL] Tonberry v1.0 - External Texture Support for FF8
« Reply #20 on: 2014-06-15 18:40:53 »
Wow this its grat! so finally my textures can be used! :D
so if this  will work I can finish the kiros, ward and edea texture.
Ozmy the left texture of squall its different because its from the field model and my textures are for the battle models, I done a texture for the squall field model too. (the balle models have 2 textures and more  poligonal details)

Good luck with this!

Omzy

  • *
  • Posts: 205
    • View Profile
Re: [REL] Tonberry v1.0 - External Texture Support for FF8
« Reply #21 on: 2014-06-15 19:23:10 »
Ozmy[sic=p] the left texture of squall its different because its from the field model and my textures are for the battle models, I done a texture for the squall field model too. (the balle models have 2 textures and more  poligonal details)
Sorry bout that, didn't realize, so that should make it easier if they are already formatted properly.

Also, I've released hashmap_v1.1.rar above, which adds support for Mcindus' cards! Check out his release when it comes out.

Edit: After realizing the original release was compiled in Debug mode, I've recompiled it with Release mode and almost all of the trigger lag is gone, zlib.dll and ann.dll are no longer required, and the DLLs are a fraction of the size. Doh =p See main post for Tonberry_v1.1.rar
« Last Edit: 2014-06-15 23:24:58 by Omzy »

Mcindus

  • *
  • Posts: 929
  • Artist, Modder, Musician.
    • View Profile
    • Lunatic Pandora
Re: [REL] Tonberry v1.1 - External Texture Support for FF8
« Reply #22 on: 2014-06-16 02:49:46 »
This is why I'm here.

Thank you, Omzy! I can't really thank you enough. 

Have you guys tried it out yet?

Tripod v1.0

It seems that the background images don't load 100% of the time...
« Last Edit: 2014-06-16 04:04:15 by Mcindus »

Omzy

  • *
  • Posts: 205
    • View Profile
Re: [REL] Tonberry v1.1 - External Texture Support for FF8
« Reply #23 on: 2014-06-16 04:20:26 »
It seems that the background images don't load 100% of the time...
That just means theres a small hashing problem (missing line in hash1map.csv). Which ones aren't loading? I'll probably release a debug version tomorrow or so that allows dumping textures so you guys can find the problematic ones easier.

Mcindus

  • *
  • Posts: 929
  • Artist, Modder, Musician.
    • View Profile
    • Lunatic Pandora
Re: [REL] Tonberry v1.1 - External Texture Support for FF8
« Reply #24 on: 2014-06-16 05:27:29 »
That just means theres a small hashing problem (missing line in hash1map.csv). Which ones aren't loading? I'll probably release a debug version tomorrow or so that allows dumping textures so you guys can find the problematic ones easier.

The right portion of the screen seems to load less often, but I've had the whole background image not show up.  It it happens to the background, it happens to the mat.  Until you play another game I think.  I'm going to test more.

On the bright side, the cards always seem to work.