Show Posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.


Messages - Omzy

Pages: 1 2 3 [4] 5 6 7 8 9
76
Updated PuPu today to correctly import textures to be used with Tonberry. I also had to change the hash files to reflect the changes. Main post has been updated with new hash files in hashmap_v1.0.rar. If I have to change hash files in the future, which will happen a lot I anticipate, the most updated will be there. I hope not to have to change the DLL very often at all, until I implement the trigger pre-loads.
This is only for the Steam version right?
Yes, it only supports directx 9, which is what Steam uses to draw the graphics and what made this possible.

Edit: Here's a screenshot comparison, since you asked so nicely Leonhart  :-P Keep in mind, this is just a 10 second bilinear resize in photoshop, this won't come close to fractal

77
FF8 Tools / Re: [REL] PuPu - FF8 Field Importer/Exporter
« on: 2014-06-13 02:31:22 »
Updated main post with version 2 which now works with Tonberry  :D

*Note that the working directories are different, I've changed them and the prompts should make it harder to screw up now.
*Also, for the 1 person that already downloaded it, re-download, had a tiny bug I fixed.

78
Let me know if there is anything I can do to help. I would love to test this immediately.
Great! I posted my Tonberry release in the Tools section like an hour or 2 ago (before this thread) but it hasn't been approved by the forum moderator yet. I'll need lots of help with that project before this project.

79
Project Eden v1.0
A collection of 4x-resized background textures for FFVIII.
These are the result of my field resizing batch script which I will release once it has been tested more. As this is a beta release, many textures have bugs that you will notice in-game. Some of these bugs are from the photoshop script, and some of the bugs are from Tonberry's hashing. Over time, as players point out these bugs, both will be refined until we are at 100% completion. Note that there may still be some bugs in Tonberry that may affect your gaming experience, potential crashes or lag during summons have been noted.

Pardon a few of these screenshots which may be missing lighting or other little things--they should look normal in-game, for the most part.
(Click to Enlarge)





Download
Download the 1.3GB pack. Then, if there are other files that have been fixed, you can download those packs and apply them manually.
*If any of these links go dead, ask the community for a mirror.

Download: Project Eden (~1.3GB)
Torrent: Project Eden (~1.3GB) (Link down for now, anyone have mirror?)

Install
Requires the most updated version of Tonberry.

After downloading the big pack, you may download individual fixes and 'patch' the folders inside the big pack with the fix (copy&replace). Your directory structure should resemble this:
C:\Program Files (x86)\Final Fantasy VIII\textures\(lots of folders). Refer to Tonberry's documentation for folder structures.

Feedback
-If you have trouble installing or getting files downloaded, post in this forum so that others can respond with help.
-Please document glitches in this format when you find them, and keep the running list posted in each new reply so we can see all of them in one spot.

[Name of field file from PuPu/Deling] : [problem with Tonberry or Project Eden?] : [description of problem]

Glitched Field List
(alphabetical, re-post this quote if you are adding to the list)
Quote
Tonberry Hash Errors:
bcsaka_1 or bcsaka1a : Tonberry hashing error : flag animation is old texture
bgroad_4 : Tonberry hashing error: bushes on bottom are old texture

Project Eden Glitches:
bghall_3 or bghall3a: Project Eden glitch : black borders around water
bghall_4 or bghall4a : Project Eden glitch : wrong elevator layers
bgroad_4 : Project Eden glitch : white border around tree leaves
*See newest replies for updated list

80
Tonberry v1.5
External Texture Support for Final Fantasy VIII
*Beta version, please let me know if there is something preventing you from using it


                   

This drop-in package allows:
I) Support for Hi-Res external textures (field files for now) to 4x original size in an FF8\textures directory
II) Future support for ANY texture (characters, monsters, battles, world map, cards, spells, summons, menus, and just about anything else you can think of)

Instructions (for players):
1. Extract Tonberry_v1.x.rar into your FF8 directory (typically 'C:\Program Files (x86)\Final Fantasy VIII')
2. Make sure to drop in the most up-to-date hash map files, from hashmap_v1.x.rar.
3. Download and place your properly formatted textures from their respective projects into the 'textures' folder (see below)
4. Install VC++ 2010 Redistributable (x86)
5. Play!
6. (Optional) If your Windows installation is not on drive C, change your prefs.txt file 'drive_letter' from C to your drive letter. If you're using textures of different sizes, change 'resize_factor' in prefs.txt.

Formatting Textures (for modders creating texture packs):
1. Create 4x textures with dimensions exactly 4 times the width and 4 times the height of the originals (PNGs supported only).
2. For field files, use PuPu to export and import your images (PuPu Updated and Working).
3. For non-field textures, PuPu will not work, and they will have to be renamed by hand or custom batch scripts.
4. Most textures won't require this, but field files have pages. If a field texture has a width of 1664 it has 13 pages (each 128x256 => 512x1024) and if it has a width of 1536 it has 12 pages. PuPu will automagically slice them up for you and name each page properly starting with a suffix of _0 and ending at _11 or _12 (_13 to _25 for 4-bit indexed textures).
5. Ensure your textures have the proper folder structure with 'textures\firsttwoletters\texturename\texturefile.png\'. E.g. for bghall_1 => 'textures\bg\bghall_1\bghall_1_0.png' for the first page, 'textures\bg\bghall_1\bghall_1_1.png' for the second page, etc. If you have a texture that doesn't require pages, the format would be the same, except with the page suffix '_0' (note that this has changed). E.g. mc00 (cards) => 'textures\mc\mc00\mc00_0.png'.

Adding Support for Other Types of Textures:
->How Tonberry Works
As mentioned above, Tonberry currently only recognizes field files. This is because in order to replace textures, it must read the original paletted textures from directx, generate a key that identifies the texture, and then lookup the texture in the '/textures' folder. Generating the key takes a lot of work on my end. I have to have up to 16 versions of each texture colored with each possible palette from the original source files. I've written programs to generate keys for field files, but it will take some time to support each additional file type. If you have a project that demands support for another file type, you can make it happen if you have some basic programming experience. If not, you'll have to convince your fellow modders to do that work for you if I can't.

Here's how to add support:
1. Find the game source files that contain the type of textures you are working on. They might be LZS compressed, have weird file extensions like md5, bin, etc, but they are all binary files of zeros and ones when it comes down to it.
2. Decode the files using the wiki documentation or asking around. Myst6re has apparently decoded most of them, as is evidenced in Deling (sorry if that's negative publicity myst but its true!). Also, contribute new knowledge to the wiki if you can along the way.
3. Find out if the texture type uses palettes (IMPORTANT!)
4. Write a simple program in python/c++/your favorite language that takes the source game files and exports the textures exactly as they are in the file. No need for map files or anything that will map the blocks to the screen, just dump the image data from the file (mostly applies to fields I think).
5. Export every possible paletted version of each texture. This means that if there is an area in the file with palettes, instead of finding which palettes go to which blocks in the image data, literally export an entire image that uses 1 palette, and do that for every palette.
6. Optimize your program to only export palettes that are used. For example, if you have 3 palettes pink, green, and yellow, but the blocks of the file only make references to pink and yellow palettes and never actually use the green palette, that version of the texture will NEVER be loaded to memory anyway, so don't export it.
7. Format your textures in the manner described above but with suffixes for each palette number. For example, palette 12 on mc00 would be mc00_0_12.png. There will be 2 suffixes including the page. For example, for the 3rd page and 8th palette of bggate_1, it would be bggate_1_2_7.png.
8. You now have a directory full of images that is ready to be hashed.
9. RAR it up and send it to me and I should have it working in short order.
10. Be prepared to make mistakes at any point, they are inevitable, but persevere in the name of art!

Examples of paletted texture files (what your output should look like):


*Please DO NOT post any direct links to files, always link to this forum thread. Especially, do not make any links to adfly or any other ad site
Download: Tonberry_v1.5.rar Mirror 1
Download: hashmap_v1.3.rar Mirror 1
Download: Tonberry_s1.5.rar (SOURCE CODE ONLY)

*hashmap contains most updated hash files, tonberry contains most updated DLLs*
*My policy on my mods is always that you have complete freedom to take them, use them, modify them, and even release derivative works--just give a line of credit in your release and do not draw any monetary profit from my links or releases. I think if you make something, including a derivative work, it is fine to solicit donations, but I encourage you to retain these same policies of openness for whatever you make so we can always further the modding community on every front.

Credits:
-Matthew Fisher at Stanford University for developing the original d3d9Callback interceptor: http://graphics.stanford.edu/~mdfisher/D3D9Interceptor.html
-Aali for developing the original work that allowed me to replace textures in FF7 (his OpenGL driver and Palmer, which PuPu is based on)
-Myst6re for her Deling field editor, which is quite an amazing tool (http://forums.qhimm.com/index.php?topic=13050.0)
-yarLson and sl1982 for their work on the FF7 field backgrounds, EQ2Alyza for her battle scene enhancements
-Salk for keeping me motivated to stay crunchy!

*Known issues:
-Memory issues--may crash after extended use. Temporary solution: Get RAMRush and set it to automatically optimize.
-Lag during some summons or other events that resolves by itself
-Possible hash collisions resulting in glitches where random textures are replaced by other textures.
-Works with the Steam release of FF8 only. I don't have any impetus to try to make it work with anything else, but I have posted the source above.

Debug Version
How to use:
Note the 'debug0' folder inside the 'tonberry' folder with category subfolders inside. If you rename this folder to 'debug' (erase the 0), Tonberry will dump all textures into these folders while your game is running. Be careful! You may amass gigs of textures in them if you are not aware and are just playing normally. Have a save prepared at a certain point and only play for a few minutes, depending on your hard drive size. To turn this feature off, simply rename the 'debug' folder to anything else like 'debug0'. Alternatively, you can rename only some of the subfolders to turn specific types off. Subfolder categories:
-unsupported: size/format is not yet supported (256x256 currently only supported)
-nomatch: not yet hashed, but will be in the future with modder help
-nomatch2: hashed once, failed second round of hashing (shouldn't happen)
-noreplace: hashed and intended to be replaced but the new texture failed to load or didn't exist
-replaced: successfully replaced
-error: texture not otherwise accounted for (shouldn't happen)

Next on my agenda:
1. Project Eden, the FacePalmer of FF8
2. Help people build support for other texture types

81
Any screens yet?
Since it takes a lot of time and effort to write Photoshop scripts to do what FacePalmer did, I don't have any fractalized backgrounds made up yet, if that's what you're wondering. I'm about to release my Tonberry Beta, though, which is the DLL that supports external textures and should pave the way for a beautified game experience. Check the Tools section in a bit.

82
Nearing the finish line. Here's a proof of concept release that is very preliminary. It includes the textures starting from the first save point in Balamb Garden all the way to the front gate. Please do read the readme before using it. Let me know if you're unable to run it for any reason--I haven't tested it on any other machines so there very well may be a problem or two I didn't consider regarding portability.

Quote
Tonberry PRE-RELEASE ALPHA Readme
by Omzy
Born at http://forums.qhimm.com/index.php?topic=15222.0

Credits to Matthew Fisher at Stanford University for developing the original d3d9Callback interceptor: http://graphics.stanford.edu/~mdfisher/D3D9Interceptor.html

*This only works for the Steam FF8 release which uses DirectX
1) Drop the contents of this folder into your FF8 directory that includes your FF8_Launcher.exe
2) Run your launcher
3) Play!

Notes:
*) The textures bundled with this pre-release were improperly resized with photoshop in 30 seconds, do not use fractals (only bilinear), so are meant only as a proof of concept, not as replacers. You will notice there are lines that make the background look like bricks due to the poor quality resize
*) This hasn't been tested on low-end systems, only 1 medium-high end modern system running Windows 7 64-bit, there may be some barely perceptible lag but it isn't that noticeable but if you have a lower-end machine, it may suffer considerably, similarly to running an emulator
*) Anytime a switch/door/trigger occurs in any scene, there is a lag associated with the texture 'pop-in' when it loads. This may be addressed in further releases
*) The hashing algorithm is unfinished and will exclude some ~150 of the ~13000 texture pages used in the game. This will be noticed as parts of the background being high-res and other parts being low-res
*) Only field backgrounds are replaced at this point, but this will be expanded in the future to other texture types
*) Included is a WrapperParameters.txt which has a BaseDirectory that should equal the location of your FF8 directory. The default is "C:\Program Files (x86)\Final Fantasy VIII"

Download : Tonberry Pre-Release Alpha

Edit: Also, is there a project named Tonberry already? I looked but didn't find anything obvious, curious since its such an iconic ff monster.

***Edit2: I had a friend test my first upload and it was missing a few things so I've fixed that and re-uploaded for those of you who downloaded earlier. Don't forget to change your WrapperParameters.txt if you need to!

83
The way this all works is that whenever the game wants to display ANY texture, it calls a CreateTexture command first and then does the processing it needs to draw the texture onto triangles on the screen. Even the field backgrounds are drawn onto strips of triangles that produce a flat rectangle. So far I know that these commands are called when field backgrounds, character sprites (the character models you speak of), object sprites like save points, character portraits, menus with text, and cards are drawn. By extension, I believe they are also called for the world map, battle scenes and spell effects, and everything else, but I haven't proven those because I haven't recorded PIX data on those scenarios yet. So, if a CreateTexture command is called, then this method can replace that texture.

All that I have to know in order to replace a texture is what it looks like in memory. So far, the field backgrounds look exactly like the mim file contents in memory (with various palettes applied). I've already done the legwork and have written algorithms to display every necessary permutation with all necessary palettes for the field files. Character and object sprite textures look just like they do in myst6re's deling program when you open a field file and go to 3d models section. I don't have any experience with those files yet. So the bottom line here is that if any of you guys want non-field textures to work as well, I'll need to get copies of all of those images so that I can run my hash algorithm on them.

Generating these images is not hard--in fact myst6re has generated most of them by reading the ff8 files in her deling program. Its a matter of writing little programs that read the ff8 files and output them to image formats like PNG. In order to see what they look like in memory, you can download the DirectX SDK (June 2010) and use the PIX tool to record all the directx calls. When viewing a PixRUN file, you can click the texture references on all the SetTexture calls and it will display what the textures look like in memory. My guess is that most of them look exactly like they do in deling, because field files are the most convoluted files to display and tex files, etc, are very simple.

If anyone wants to help speed up the progress on those other types of textures, feel free to write python scripts that output images from game files or send me rar's of images. For now, I'm focusing on field files. This process actually becomes more reliable (from 98%->100%) once I have hash values for EVERY texture in the game. As it is now, it is theoretically possible for my hash algorithm to place a field texture over a character sprite, for instance, if they coincidentally have the same hash value. Rare, but possible.

84
Just wanted to update you all on my current progress. I've been able to resize a field file to 4x and get it working in-game. All of the animations and lighting effects seem to work. I've been working rather feverishly on this, more than I should readily admit.

My previous hash algorithm caused a lot of lag every time a texture was loaded since it was searching the entire hash. The new hash technique I'm working on should produce much more reliable hashing that is hundreds of times faster (17 iterations to find the texture vs 6256 iterations per texture). This represents the difference between an algorithm that searches linearly through a large data set and one that searches with a binary search algorithm. It should eliminate that lag altogether.

This method should work on all field backgrounds once I get polished, but also has the ability to work for all other texture-like files in the game including character/monster sprites, battle scenes, cards, menus, etc. In order to get it to work for those other texture types, I need to have directories full of those images extracted from the game files. I can see that myst6re's deling works to view most of those files, but extracting the textures from them is something I haven't tried. I'll cross that bridge when I come to it.

Stay tuned for a release within the next couple of weeks! (this will not include a resize pack, that is the next project)

85
I'm ready to test my new dictionary of hashes in-game but I need to gain a bit more d3d9 knowledge first. I'm planning on making use of a d3d9 interceptor DLL located here: http://graphics.stanford.edu/~mdfisher/D3D9Interceptor.html. I'm trying to find out which files I absolutely need and which ones I can change to test texture replacement in-game. Basically I need to capture the SetTexture calls and replace them with any test texture. Any help/advice would be appreciated to help me go faster  :-D

Here's the hash dictionary using the algorithm described above. It adds a numerical suffix to the filenames for each 128x256 block of the texture. There are a total of 6245 blocks in the game.
FF8 Field Hash Dictionary

Edit (note):
http://www.gamedev.net/topic/178004-idirect3dtexture9-and-manipulation-of-single-pixels/

Update (preliminary):
I have verified that my hash function has been successful at identifying my first test case, bghall1, from the VRAM. At first I was a little worried, because the hash value showed up 3 times, but those were bghall1, bghall1a, bghall1b, which are all identical textures. More to come when I have time!

Update2:
I've got the game successfully running the hash function on every texture that is called by D3D. First, I load my hash map from a file of all the known textures (described above). Then, I run the hash function whenever a SetTexture call is made. I then compare that hash value against the hash values in the hash map to find the closest match (the texture with the least different bits of comparison). I've done a test run from bghall1 (first save point) to the front gates of balamb garden and I've recorded these matches in a file to see if the correct textures were found. Here is that file, with a few annotations thrown in regarding where I was at each point in time: http://pastebin.com/9UpAynEd

Explanation of the results--
line structure: filename_blocknum,mismatchedbits,thistexturehashval,closestmatchedtexturehashval

I've only recorded 1/100 texture calls in the results because the file would be too long to navigate if I didn't. Whenever a background is supposed to be set, the hash function correctly identifies the texture and spits out the file name. However, whenever a texture with mostly black or monotone pixels is called, like in intro screens, and some other screens, including some animations that have a lot of black but a few spots here and there like lights, the hash function fails, usually identifying another texture file that has mostly black or monotone pixels. Additionally, whenever a character sprite texture is called, the results show either NO_MATCH or may sometimes identify the wrong texture as a match.

Keep in mind, these are results of a completely untested hash function that has a ton of room for improvement. The number of pixels sampled, number of comparisons between those pixels, randomness of the pixels, and thresholds for matches and mismatches can be varied. Also, secondary hash functions can be used on textures that are mostly black, for example, and eventually special cases can be worked on separately.

Next step: test texture replacement in D3D

Update 3:
Successfully replaced textures in-game with a simple 256x256 fire texture. This slows the system down tremendously if the texture file is loaded from the disk every time a SetTexture is called. This is because SetTexture is called roughly 150 times per frame to render all the various parts of the scene. Since all of the parts of the scene, except the character and object sprites, are loaded from a single field texture file, this 1664x256 (or 1536x256) texture file should be loaded only on the first texture call of the scene and then referenced in memory for each SetTexture call. This should drastically improve framerate and reduce errors in hashing during the other 149 SetTexture calls in the scene to 13/6245 = 0.2% of the original error rate. This all hinges on the ability to detect the correct texture on the first call during the frame.

Update 4: I have now successfully replaced textures in-game with their appropriate textures, using the hash method. It is not perfect, but it reliable chooses the correct texture > 90% of the time so far. I've expanded the pixel selection from 16 to 64 pixels to make it more reliable. It needs some optimization to avoid lag and some other fixes, but I really need to work through the next huge hurdle: allowing resized textures in-game. This, I fear, will be very difficult, since there are many operations performed on the passed-in textures in order to render them to the frame buffer. These operations include drawing all of the geometry of characters/objects, etc, and I'm so far unsure how to resize only the field textures yet keep everything else unaffected. I'll be thinking on that...

86
There is no "good" reason for the duplication you're seeing other than that its necessary to properly emulate the GPU. This is one of the big issues with replacing textures in FF8, field backgrounds are used as if they were 128x256 but when they're uploaded to a texture the next 128x256 pixels from the VRAM will be uploaded with it because that's just how the GPU works.
I understand, so the second block in memory I'm referring to doesn't really do anything, its just there because the commands that load the first block automatically load whatever's next for no good reason. So the palettes cycle on the first block, then the next one loads, etc.

87
Maybe this graphic will help. The blocks cycle through the VRAM in the order of the texture file. At each point, the palettes cycle over the image, changing the colors in VRAM until no more colors are needed, then the cycle continues with the next blocks. The palettes needed for the block in the first position are prioritized, that's the point of keeping the next block in the VRAM next state, because it hasn't finished all of its palettes yet, just the ones in common with the block before it.

Keep in mind, this is a theory based on a very small amount of data so far.


88
I've rewritten the hashing algorithm in C++ and have expanded it to 64 bits of comparison on 16 pixels. It compares the 16 pixels against each other in 4 different ways and generates a 64 bit hash value. If any two images differ by less than 10 bits, they are the same image with a different palette. Otherwise, they are different images. Typically, similar images with different palettes differ by only 3-6 out of 64 bits and images that are completely different differ by 20-45 out of 64 bits. When this algorithm is adapted to sample straight from VRAM instead of filling an image buffer first, it should run very fast.

Next stop, generate a dictionary of hash values mapped to filenames and try to see if it is accurate whilst the game is running. This dictionary will be placed in an external file in the game folder and read into memory when the game starts. That may take a bit longer to figure out.

*C++, uses CImg library
Code: [Select]
#include <iostream>
#include "CImg.h"
using namespace std;
using namespace cimg_library;

uint64_t hashval = 0;
int pixval[16];

void growhash(int columns)
{
    int lastpixel = 127;
    for (int i = 0; i < 16; i++)
    {
        if      (i < 1      ) { lastpixel = 127;                    }
        else if (i < columns) { lastpixel = pixval[i+(15-columns)]; }
        else if (i < 16     ) { lastpixel = pixval[i-columns];      }
        hashval *= 2;
        (pixval[i] - lastpixel) < 0 ? /*nothing*/: hashval++;
    }
}

void gethash(CImg<unsigned char> image)
{
    int pixelR, pixelG, pixelB;
    for (int i = 0; i < 16; i++)
    {
        int x = i*16;
        pixelR = image(x/2, x, 0);
        pixelG = image(x/2, x, 1);
        pixelB = image(x/2, x, 2);
        pixval[i] = (pixelR + pixelG + pixelB) / 3;
    }

    growhash(2*2*2*2);
    growhash(2*2*2*1);
    growhash(2*2*1*1);
    growhash(2*1*1*1);
    return;
}

int main() {
    char filename[] = "1.bmp";
    CImg<unsigned char> image(filename), visu(512,512,1,3,0);
    CImgDisplay main_disp(image, filename);

    gethash(image);
    cout << hashval << endl;

    while (!main_disp.is_closed())
    {
        main_disp.wait();
    }
    return 0;
}

Edit (notes):
VRAM order theory: VRAM loads the necessary parts of a scene by selecting the next necessary 128x256 portion of the field texture (block 1) and loading the next 128x256 block after it (block 2) to fill a 256x256 square of VRAM. It cycles through the necessary palettes for block 1 until it completes all portions of block 1. If there happened to be correct palettes for block 2, it will draw those portions but will likely need more palettes to complete block 2. It cycles and loads block 2 into block 1 slot and loads the next block into block 2 slot. The process repeats with the remaining palettes of block 1 loading and completing block 1, etc. The order of necessary blocks follows the file order in that blocks at the beginning of the file are prioritized first and blocks at the end are prioritized last. Animation blocks are not loaded at all unless the current frame demands a block containing a particular animation, in which case that block will be loaded when its turn in the file sequence comes.

This implies that instead of treating each texture file as a texture, each 128x256 block should be treated as its own texture since different combinations of the block will be loaded into the 256x256 VRAM square and recognized as a complete 256x256 texture by DirectX.

So, when checking filenames for DirectX textures, the first block should be identified and the second block should be identified separately with a different hash and in the dictionary these should be labeled filename_1, filename_2, filename_3...etc for all 128x256 blocks of the field texture file.

Rough total block count: 894 files x 1664/128/2 ~5000 blocks

89
Oh... I was actually referring to Unity or any already made engine. And yes... Q-gears is for VII  :|
I see what you're saying, I've played with Unity--it is the most awesome design tool around for sure. I could see someone rebuilding these games from scratch in Unity using code from things like Q-gears and rewriting systems entirely to support existing or modified Square assets, but it would take a dedicated team and a good chunk of time to pull off. Heck, being able to use Unity (porting Q-gears) might rally more developers to its cause, even. For now, I'm going to try to hack my way there--at least to see if its possible.

90
I've been playing with hash functions and came up with this, which samples 16 distinct pixels in the VRAM image and compares their grayscales to generate a hashvalue based on their differences. It uses a threshold value for how large the difference must be and that determines if a 0 or 1 is written. Based on only 1 frame of gameplay, this works, however it may need to be expanded to 32 pixels or additional comparisons instead of a linear comparison maybe odds vs evens, etc, if we are to expect different values for every texture/palette combo in the game. Hopefully this sort of technique can eliminate the need for distinguishing between palettes.

Written in python, forgive the poor syntax/lack of efficiency, I've never used it before so I just googled how to do things really quickly
Code: [Select]
import PIL
from PIL import Image

img = PIL.Image.open("5.bmp")
rgb_im = img.convert('RGB')

diff = []
binval = []

for i in range(0, 16):
diff.append(0)
binval.append(0)

currvalue = 255

for i in range(0,16):
x = i*16
pixel = rgb_im.getpixel((x,x)) # diagonal line of pixels to sample
value = (pixel[0] + pixel[1] + pixel[2]) / 3 #grayscale
diff[i] = currvalue - value # difference from last pixel value
currvalue = value
binval[i] = diff[i]
if binval[i] < 10: # convert to binary value based on threshold
binval[i] = 0
else:
binval[i] = 1

hashvalue = 0
for i in range (0, 16): #generate hash value from binary values
hashvalue *= 2
hashvalue += binval[i]
print hashvalue

a = input()

91
Textures are usually 256 * 256, either 8- or 16-bit. Add 16 different palettes of 256 32-bit colors each and you've got 70-150kb of data to hash every time the game tries to upload a texture. Now add to the mix the fact that palettes are not static, the game (and especially field files) often writes directly to the palette causing the whole texture to be reloaded (and depending on how the palette is modified you could get a million or so different hashes just for that one texture).
I wonder if a different hashing algorithm would make this easier. The hash can be computed from the values of specific pixels normalized to black and white and stored as a string of binary digits. Most backgrounds should have a different pattern of digits if enough digits are used and a few will fail this check if they are largely homeogenous like the credits having mostly black, etc. With carefully chosen pixels and enough of them we might achieve perfect hashing. This may eliminate dependence on palettes and be more specific to patterns. Did that make any sense or am I speaking crazy? I have yet to dig my fingers into the dirt so it might sound abstract right now but I'm going to do some experimenting with patterns and see if it works.

92
I thought D3D and DX interceptors like TexMod intercept textures based on hash checks (or a similar calculation), making it possible for multiple textures to accidentally get replaced by a singular texture. Texture mods with too many replacements break down with those types of programs for this reason.
When I referred to a matching algorithm above, that sort of thing is what I had in mind--a hash function. As long as it is written such that no keys(field files) have collision or fall into the same 'bucket', then that should work. This is possible, I think, since we know what all of the texture data is before-hand. We'll have to test such a function to make sure there are as many unique keys as there are hashes. http://en.wikipedia.org/wiki/Perfect_hash_function

93
Thanks a lot for the explanation Aali! It does seem a daunting task, but maybe over time we can arrive at a solution.

Let me recap my new understanding of how all this works, based on what Aali wrote and my findings in PIX:
1) The game engine has a texture to VRAM uploader for all textures (tex and fields)--there may actually be several loaders for multiple texture formats
2) When a scene is drawn, a loader takes a texture file from the game files and writes part of the texture to an area of VRAM
3) Each texture file contains multiple parts (e.g. curtains, bed, wall in the example of the waking scene), each of which is loaded into different areas of the VRAM
4) Each of these parts requires a different color palette, which may be one of 16? or so cached palettes that determine the colors of the texture part written to that segment of VRAM
5) The resulting VRAM holds several separately colored sets of overlapping information all derived from the original source image
6) Only a part of that VRAM may be used (the segments of the parts which are correctly colored)
7) Collectively, these segments of VRAM contain all of the information required to correctly write the parts of a scene to the final output

From the Direct3D perspective:
1) D3D recieves a call from a loader to load a part of the texture file into VRAM with a certain palette applied.
2) D3D recieves many of these calls and VRAM is populated with the various parts of the entire texture
3) D3D recieves calls to write these parts (in the order that the parts were loaded into VRAM?) onto the surface image
4) At the end of the frame, the parts magically have assembled into the correctly colored scene

In other words, a D3D interceptor will be able to read the assignments of each texture part to the VRAM and the resulting texture after the palette has been applied, but it will not know the origin of the texture (filename).

This means that there are 2 possible routes towards the solution:
1) Modifying the game engine's texture loaders as Aali has described, in order to obtain filenames for external textures that can be loaded (limited by the difficulties Aali mentioned)
2) Inferring from the D3D calls which field file the frame wants to display with some sort of matching algorithm and replacing all instances of texture parts with the correct parts (no palette work necessary since we are already working with output but determining the filenames and parts to replace may be near impossible)

Both solutions involve finding which file the game wants to display and which part of that file belongs in which part of the VRAM. I still haven't considered what it takes to plug-in higher resolution versions--does this use U/V coordinates and are there D3D calls that determine the mapping?

Again, still learning here, so I may have made mistakes, correct 'em for me if you please.

94
There is a mod for Dark Souls, which allow changes of the textures with higher res versions of them. Maybe the author of it has some ideas.
http://www.nexusmods.com/darksouls/mods/19/?
I would like to point out that actually replacing the textures is the easy part (in fact it is trivial), the problem is figuring out what to replace and what to replace it with.

Ok, I think I understand what you mean now, based on Kaldarasha's link and a follow-up link I found: http://graphics.stanford.edu/~mdfisher/D3D9Interceptor.html
Quote
This is a program that intercepts all Direct3D 9 (D3D9) commands issued by a running application. These commands are then dispatched to a separate program that interprets and responds to them. The commands can be forwarded to the operating system, discarded, or modified as desired...
...The idea that enables this interception is that Direct3D (like most Windows APIs) is dynamically linked. Furthermore, Windows first searches for dynamic link libraries (DLLs) in the application's local directory. Therefore, to intercept calls to the D3D9 dynamic link library, d3d9.dll, all that is required is to make a custom version of d3d9.dll and put it in the application's local directory, which is exactly what this framework does. All intercepted calls are passed to a separate DLL, d3d9Callback.dll. This DLL decides what actions to take as a result of the command stream. It is also allowed to modify or discard the command stream before the interceptor sends it to the "real" version of d3d9.dll that was the intended target of these calls, which dispatches the commands to the graphics card.

So in order to swap in new textures, you need to identify the commands that call the internal texture files and change them to commands that call external texture files, but this is the hardest part, as Aali points out (if I interpreted that right).

Edit (notes for myself):
PIX for windows allows analysis of any directx3d program for function calls, objects, and playback for each frame. http://www.chromium.org/developers/how-tos/debugging/pix-for-windows
Getting PIX to work with FF8 is difficult. In order to start an experiment you must select the target exe and the correct target is FF8_EN.exe, however, only the launcher allows you to launch the game so it fails. The workaround is to have FF8_EN.exe targeted, have the launcher ready to press PLAY, then press 'Start experiment' in PIX and then PLAY in the launcher as fast as possible and it will hook itself to the process. Unfortunately, for whatever reason, the game crashes when I load a save. It did not crash when starting a new game, so it played the intro FMV then entered the scene where Squall wakes up. You can then capture a frame of information with the F12 key.

PixRun from waking scene: https://drive.google.com/file/d/0ByMkI_Nb8OmJLTNXT2xzUWtFNDA/edit?usp=sharing
To view renders/textures, click Render tab in Details window, then use Find event tool in Events window (magnifying glass) and search for SetTexture. Click on texture references to show in right panel.

Unique field texture EIDs found in a single frame (frame 3947):
12445
12510
12656
12823
12960
13606 - complete frame

There are more EIDs of the same textures with different palettes.

https://github.com/GuentherVIII/Final-Fantasy-VIII-Launcher

95
I would like to point out that actually replacing the textures is the easy part (in fact it is trivial), the problem is figuring out what to replace and what to replace it with.
Thanks for joining the discussion Aali. Would you mind expanding on this--are you referring to the driver or the game files, your driver or directx?

Edit: sorry for the double-post, didn't think.

96
Only as a node GpuBlade use the CPU for graphic rendering, so it is ve~ry slow.
What we need is a program which updates the textures in the video memory and it should lead the game engine to believe that everything is at it should be. The rerelease of FF8 uses now DirectX9 so maybe a modified version of Textool would work.

I found the textool readme, which includes instructions:
Quote from: ficedula
Now you can go away and edit the BMP file all you like ... however, don't change the size *OR* the palette. Whatever palette the BMP has originally is what the game is going to use, so there's no point trying to change the colours in the BMP file.

This indicates that you can change the content of the textures but not the size since the game still expects the same texture size. Likewise, repackaging fields into the same files with the same size will not work since I am looking for a 4x rescale. (As a tangent, some people have asked why 4x? Why not 2x? My answer is that with Aali's driver, the resolution of the sprites/3d models actually is closest to the 4x textures and they look best when superimposed on 4x backgrounds). The convenient thing about Aali's driver is that it allows you to simply drag and drop your resized textures to an external directory where his driver looks first to replace textures. If a similar strategy can be incorporated using DirectX to allow external textures, that may be the best way. Does anyone know anything about the DirectX that is used by the Steam version of FF8? Can it be modified? Can additional files be added?

A fullscreen shader to smooth the backgrounds could be used with Aali's driver as well, if you know how to write one. In theory the shaders for Pete's Opengl plugin should work with Aali's Opengl driver.They would need of course some changes.

After looking into these shaders, it seems that they are the same algorithms used in the filters I described above, so the best you're going to get is the 4xBRZ, which in my opinion, isn't acceptable quality. I like this DirectX idea, though, if we can get more information on it.

97
I found some more information on what his GPU plugin allows:
Quote from: egdbla
Processing:

«Internal resolution»
Allows you to improve the image quality in 3D games, requires a powerful system.
( Possible values: 1 - 4. )

«Use 24-bit renderer»
This makes picture quality approach that of hardware accelerated plugins by eliminating color banding and smoothly painting the polygons.
"Disabled" — disabled.
"Primitives with dithering" — enabled only for primitives with dithering.
"Gouraud shaded primitives" — enabled for all Gouraud-shaded primitives.
"All primitives" — enabled for all primitives.

«Disable 'dfe' emulation»
Disable interlace emulation. All of the odd or even lines of the frame ( so-called fields ) are rendered in turns. This noticeably decreases the picture quality as that line alternation is clearly seen, and at the same time this gives rendering quite a speed boost because the fillrate needed is cut in half. This option disables this drawing and output mode and vastly increases picture quality, but reduces perfomance due to increased fillrate.

Postprocessing:

«Deinterlacing method»
Useful only when "Disable 'dfe' emulation" mode is disabled.
"Line doubling" — draws each line twice, causes a slight screen shaking, the best mode for dynamic games and the closest to a real TV picture.
"Field combination" — mixes odd and even fields, causes the "comb" effect, the best mode for static games.

«Filtering»
The entire frame buffer is filtered. You may choose a filter to your liking, but remember that they might be very CPU-intensive. 2D games usually look better with SuperSAIx2/SuperEagle, while HQ2x/HQ3x are more suitable for 3D games.

«Shader effect»
Use shaders written for gpuPeteOGL2. Put each shader in its own folder inside the «./shaders» directory alongside the emulator executable.

«Shader level»
( Possible values: 1(Minimum), 2(More), 3(Medium), 4(Maximum). )

«Bilinear filtering»
Reduces picture blockyness. Bilinear filtering is hardware-accelerated, speed reduction is almost unnoticeable.
Information on the SuperSal/SuperEagle/hq4x/4xBRZ filters can be found here on wikipedia: http://en.wikipedia.org/wiki/Image_scaling
Here's a nice tool that allows you to quickly compare a variety of upscale methods (does not include fractal): https://code.google.com/p/2dimagefilter/downloads/detail?name=ImageResizer-r129.exe&can=2&q=

Here are examples of all the major upscaling filters I am aware of:
original - http://postimg.org/image/49g9clby5/
4x SuperSal (2x run twice) - http://postimg.org/image/ilol1h0yz/
4x SuperEagle (2x run twice) - http://postimg.org/image/81rxh3cev/
hq4x - http://postimg.org/image/sjbjchosd/
4xBRZ - http://s7.postimg.org/vqbiixhmj/wake_4x_BRZ.png
4x fractal - http://postimg.org/image/fyh6ixf4t/
(for fractal, imagine characters and text box aren't there since fractal must be external textures only)

Most of these were invented to upscale 2d pixel art, not renders of 3d scenes or photographs. I think the 4xBRZ is the best-looking real-time filter but looks cartoony and misinterprets many boundaries. The reigning champion is still the 4x fractal but it is not suitable for real-time filtering due to both computational speed and licensing. The others aren't much of an improvement on the original. Consider also that if a filter was applied real-time it would apply to characters and text boxes, whereas external texture import would be applied only to selected textures like fields (backgrounds). I'm not sure what shaders could be applied on top of these filters to further refine the result.

Because of these things, I sense that external textures may still be the best way to go for quality's sake, but I welcome screenshots that can prove me wrong if there are any.

I wish this method works! Will it be just for the emulator then?

If this method works it will only work on the PSX emulator and edgbla's GPU plugin, meaning that support for other mods like Roses and Wine may need more work to allow them to function. I'm not really sure what that would entail or what the full implications would be and if anyone could spell that out, it would help.

98
When I click on the link it gives me an "unable to connect" note.

Amazing find btw!!! :)
Maybe you have a browser plugin, firewall, or security software that is blocking it since it is all in russian. My forum registration was sent to gmail spam since it wasn't in english.

99
Thanks for the referral Hellbringer, I sent edgbla a message to ask what his thoughts are. I don't think he personally speaks english but he uses translating programs and has a friend on his forums that can do rough translations that he posts alongside his russian text. Here's what I found in regards to his psx gpu emulation:

Quote from: edgbla
Picture enhancement methods.

Internal resolution.
The most powerful method is an increase of the internal resolution.
This method allows you to increase the resolution of the image independently
in width and height by an arbitrary integer factor. In the manual I wrote that
the best in my opinion is the resolution of 2x2.
This does not mean that I cannot add larger factors, although it is somewhat inconvenient.
I believe that further increase in resolution degrades the image, as "soapy" textures
with perfect geometry leave a bad impression. Generally there is no restriction,
I can make it 8x8, which is equal to setting "Ultra / Ultra" (I guess)
in the hardware plugin OGL2 by Pete Bernert, but I doubt you can find a
computer capable of at least 3 fps in this resolution. :D

24-bit renderer.
Console has a 15-bit renderer, that's why Gouraud-shaded polygons have so
unpleasant colors. When you turn this mode, renderer goes into 24-bit mode
and the picture in this case is not only looks as good as on hardware plugins,
but also doesn't have their drawbacks, you should turn "Gouraud shaded primitives"
option for this. If you turn the "All primitives" mode, I'll tear your hands away :),
because in this case you'll see the artefacts similar to hardware plugins, like
various different colored borders around the picture.

Texture bilinear filtering.
The translators can't translate my text.  :lol:

Filters.
Then goes filters, btw, very good ones: Super Eagle, Sal, Hq, Scale.
For those who like them, I don't. :)
But there are 'fans' of them, so I've decided to add them.
I should say, that they're also very hardware-dependant, especially while
working in pair with the increased internal resolution.

Shaders.
Realized a partial support for shaders written for gpuPeteOGL2,
Only glsl shaders are supported.

What I gather from this is that he is able to set the output resolution to whatever he wants using a scaling factor and can use a filter for the upscale in addition to shaders. This by itself sounds like a big improvement. If you can use bilinear interpolation you can do a sort of poor quality upscale on the fly, especially if you add a shader to improve the result. However, its unlikely that there is any filter/shader combination that can achieve the results of the fractal algorithm that has impressed me so much. Since that fractal algorithm requires rendering/processing time that will exceed what can be done on-the-fly frame by frame, it would still need an external texture feature. I'm assuming that having an external texture feature would require a driver or importer specific to final fantasy 8, given the nature of the field files. I have no problem working with the playstation ROM/emulator version of FF8 since I can still play it on a PC. Perhaps this dialog will result in using his PSX emulator with a specific driver/plugin allowing external texture import. I've already written PuPu so some progress has at least been made in completing the chain of file conversions.

Do you guys think I've interpreted this correctly or am I missing something? I'll admit that I don't know much about this subject, especially as far as limitations go, so correct me if I've made any erroneous conclusions.

100
Hi all,

I just started playing the Steam version of FF8 and the characters and objects look nice and crisp but the backgrounds are so pixelated that it makes me die a little on the inside. When I had that feeling playing FF7, I resized them myself with long-handed photoshop algorithms, but I could conveniently rely on Aali's driver doing the import nice and clean.

However, with FF8, according to Aali (in response to Mcindus' Triple Triad HD project),
FF8PC emulates the playstation GPU for most parts of the game. Thats bad news for several reasons but just to name a few it means the game has limited memory available for textures, doesn't use relative UV coordinates and does not readily accept textures in any other size or format than it was designed to. I don't think you'll have much luck, unfortunately.

My problem is that I'm so obsessive that I can't take no as an answer. I know that if I take the game apart myself binary bit by bit, I can build it again, theoretically, so that it will work however I want it to, given an infinitum of time. My question for those who have the knowledge of game engines, gpu architecture, executables, drivers, etc, what is the path of least resistance to allowing this game to accept larger textures? This may include some reverse engineering, recompiling executables, creating DLL's, etc.

Please use this topic as a place to sound off and throw your ideas around.

My hope is that when I have time on my hands in about a year, this question will be answered, or at least someone can provide me with the right direction to learn the things necessary to complete this task. I really believe it is a step-by-step endeavor, but right now the steps are not clear to me.

Pages: 1 2 3 [4] 5 6 7 8 9