Qhimm.com Forums

Miscellaneous Forums => Scripting and Reverse Engineering => Topic started by: rmco2003 on 2006-06-18 17:48:51

Title: Figuring out file formats
Post by: rmco2003 on 2006-06-18 17:48:51
There seems to be a few people on here that have done similar tasks for FF7, I'm working on a translation for another game and most of its menus are stored as graphics under the name SPR. It doesn't correlate to any of the known SPR formats on the web, and the most I could find is that there are patterns of 00 hex bytes at some offsets.

I'm not really asking anyone to spend their free time hacking these files for me, but could someone point me in the right direction to working out what these files are made up of?
Title: Re: Figuring out file formats
Post by: Sad Jari on 2006-06-18 18:00:46
*points to Tech-related*
Title: Re: Figuring out file formats
Post by: rmco2003 on 2006-06-18 18:27:29
Sorry thought Tech Related was for things only related to the FF series
Title: Re: Figuring out file formats
Post by: Alhexx on 2006-06-19 17:19:26
Could you maybe upload 3 or 4 of those files (e.g. on Mirex's Trashbinâ„¢ (http://bin.mypage.sk)).

Learning how to decode file formats isn't that easy. Even if you're familiar with data structures, it takes some experience to know what you have to look for in that file.

As I said, upload it somewhere, and I'll take a look at it.

 - Alhexx
Title: Re: Figuring out file formats
Post by: Sad Jari on 2006-06-19 17:37:18
Psst! Alhexx! The topic is in the wrong place. See the move-button? :-)
Title: Re: Figuring out file formats
Post by: rmco2003 on 2006-06-19 17:39:30
There are two types of files, one contains sprite data under the label SPR and the other contains animation data under the label ANI (I'm guessing). Here's 3 SPR files with their ANI files, and 2 SPR files without ANI files.

http://bin.mypage.sk/FILES/Files.zip
Title: Re: Figuring out file formats
Post by: Alhexx on 2006-06-20 13:00:31
Quote from: Jari
Psst! Alhexx! The topic is in the wrong place. See the move-button?  :-)
As you wish, my master.  :-P

rmco2003:
Okay, I'll try to take a look at the files this evening.

 - Alhexx
Title: Re: Figuring out file formats
Post by: mirex on 2006-06-20 13:14:17
Hi rmco2003
I have also copied this question to a different boards, if you don't mind, as they are specialized on file formats. You can find it here: http://ffab.mypage.sk/viewtopic.php?t=38

I was looking briefly through the data and looks like the .ANI files are quite empty (no images inside them, only some numbers, maybe animation times), and looks like that .SPR files contain multiple images inside.
Title: Re: Figuring out file formats
Post by: rmco2003 on 2006-06-20 15:09:23
Excellent :-D thanks for your help mirex, I originally thought each SPR file contained one image, but now that I think about it, it makes more sense to group images together if they have animations. I've been stuck on figuring these things out for a year now, it's about time something got done :wink:
Title: Re: Figuring out file formats
Post by: Tonberry on 2006-06-21 05:14:20
Hi! I looked into your files and was able to extract some images. I posted one of them into Mirex's Trashbin (http://bin.mypage.sk/FILES/Test1.JPG). There are no colors because I didn't convert from 16bpp.

I can't continue now, but tomorrow night I'll have more time to post more details, so I'll try to quickly explain the way images are compressed in the SPR files. As an example, I'll take "npc_priz_card.spr".

Code: [Select]
(11 bytes) Signature. ("DCSPRITE10" 00)
(4 bytes) Number of elements in the next 2 tables. (0x98) Maybe number of images.
Start of table 1:
    (4 bytes) Width for the first image? (0x00000140 = 320)
    ... (Repeat for every element in the table. 0x98 times.)
Start of table 2:
    (4 bytes) Height for the first image? (0x000000F0 = 240)
    ... (Repeat for every element in the table. 0x98 times.)
(4 bytes) Multiply by 2 to get pointer to a second structure. (*)
Array of pointers to images (0x98 elements):
    (4 bytes) Multiply by 2 to get pointer to first image. (*)
    ... (Repeat for every element in the table. 0x98 times.)
Start of first image: (Note this is the base offset for the above pointers marked with *.)
    ... (Here come the images.)
Second structure: (I haven't looked too much into it, but it seems similar to the above structure.)


With the example file, the first image starts at offset 0x00000733 and that address is the base for the pointers.

It seem images start with F0 00 00 00 and are followed by 0x74 bytes with value 0x00. (This is for the few ones I looked into, so I may be wrong.)

After this come the compressed data. (For the first image in the example file it's offset 0x000007AB.)

Data compression is as follows:

Code: [Select]
Scan line #1:
    (2 bytes) Number of sub-blocks for first scan line
    Sub-block #1:
        (2 bytes) Number of 0x0000 that should be injected.
        (2 bytes) Number of half-words (each one a pixel in 16bpp format) that follow and are supposed to be injected as they are.
        Pixels of sub-block:
            (2 bytes) 16bpp color for the pixel.
            ... (Repeat for every pixel.)
    Sub-block #2:
    Sub-block #3:
    ... (Repeat for each sub-block.)
Scan line #2:
Scan line #3:
... (Repeat for every scan line.)

That description doesn't guarantee that each scanline will be completed. The trick I used was to substract the total of pixels injected to the width (320 pixels) and inject that amount of pixels (with value 0x0000) to complete the scanline.


As an example (same file, same image):

Code: [Select]
Scan line #1: 02 00
  Sub-block #1: 91 00-03 00-AA A3-AA A3-AA A3
  Sub-block #2: 1B 00-04 00-AA A3-AA A3-AA A3-69 93
  <Inject pixels to complete width.>

Scan line #2: 02 00
  Sub-block #1: 90 00-07 00-2B BC-AA A3-AA A3-AA A3-AA A3-E7 82-AA A3
  Sub-block #2: 16 00-06 00-AA A3-AA A3-AA A3-AA A3-AA A3-AA A3
  <Inject pixels to complete width.>

Scan line #3: 03 00
  Sub-block #1: 90 00-08 00-8C CC-CD D4-AA A3-AA A3-AA A3-A9 A3-86 72-08 8B
  Sub-block #2: 04 00-0C 00-AA A3-AA A3-AA A3-AA A3-AA A3-AA A3-AA A3-AA A3-AA A3-AA A3-AA A3-A9 A3
  Sub-block #3: 03 00-09 00-A9 A3-AA A3-AA A3-AA A3-AA A3-AA A3-AA A3-AA A3-28 8B
  <Inject pixels to complete width.>

...


Hope this helps. Anyway, I'll complete the docs when I have more time, but if it's not clear, feel free to ask. :)

Title: Re: Figuring out file formats
Post by: rmco2003 on 2006-06-21 05:43:40
awesome :-D now I just have to find someone to code a converter tool.. I only know AS level VB, not much to go on :wink:

Isn't it ironic that I spent a year of sifting through webpages trying to find out how this file format works, and I post here and it's done in a couple of days.. :-P
Title: Re: Figuring out file formats
Post by: Tonberry on 2006-06-22 00:05:32
Quote
awesome now I just have to find someone to code a converter tool.. I only know AS level VB, not much to go on

I won't code a complete converter tool (as I'm looking into FF9 at the moment) but I can give you the code I'm writting to decompress the images from the SPR files when I'm done. (Even document it a bit. :wink:)

What I wanted to ask you is what version of VB do you know: 6.0 or .Net? I'm using C#, so the translation to VB .Net should be easy; and if it's 6.0 (or prior), that's what I mostly use at work.

Oh... just don't use the spec I gave above unless you are curious because it doesn't exactly apply to the smaller files you posted. (What I said about the 0x74 bytes with value 0x00 at the start of the images has a meaning.)


Quote
Isn't it ironic that I spent a year of sifting through webpages trying to find out how this file format works, and I post here and it's done in a couple of days..

Not really. In my experience, it's quite difficult to find the description of a file format on the Web. Yes, you'll find complete docs for most common file formats out there, but this seems to be a custom format.
In particular, games use a lot of custom formats to store data so unless there is interest in the game you won't find that info. But internally, games also use pretty common formats.

What I'm tring to say is that maybe SPR files are not common, but the way the images are compressed inside is a slight variation to a common compression algorithm for graphics (or data, really.)

Title: Re: Figuring out file formats
Post by: rmco2003 on 2006-06-22 05:08:57
I know Visual Basic 6, .NET's a bit confusing for me :-P

I think it's great you've helped me out even this far, but there's a couple of things I'm not sure how to do..

Once I recieve your code, would I be able to just reverse the procedure to generate working SPR files? Or would I be coding a recompression scheme from scratch?

I'm asking this because I'm not very familiar with converting between file formats, file headers, how things are stored, etc, and although I don't have a problem with coding my own routines, in fact I can't wait :wink:, I'd like to know that it's feasible to do so.
Title: Re: Figuring out file formats
Post by: Tonberry on 2006-06-23 01:44:30
Quote
I know Visual Basic 6

No problem, I translated everything from C# to VB .Net but I'll be able to rewrite it into VB6 this weekend. (I forgot I didn't have VB6 at home.)

Quote
Once I recieve your code, would I be able to just reverse the procedure to generate working SPR files? Or would I be coding a recompression scheme from scratch?

No, the code I'm writting is to decompress the images because I thought of doing that to help you, or anyone else, understand the format. I could decide to write a recompressor, but don't count on that as I'm not too fond of coding. :D
I have no problem helping you do it or even write some pseudocode. (As the saying goes: "Give a man a fish; you have fed him for today. Teach a man to fish; and you have fed him for a lifetime". That, and that I really hate programming.)

It's not complicated; you just have to know where to find the elements and these files have few things inside. The compression mechanism implies you have to search for the repetition of a color of your choice.

Quote
I'm asking this because I'm not very familiar with converting between file formats, file headers, how things are stored, etc, and although I don't have a problem with coding my own routines, in fact I can't wait, I'd like to know that it's feasible to do so.

It's as simple as reading/writing/seeking inside binary files. The best thing you could do is learn the format, try to rewrite it in your own terms and do the coding.

I think it's possible to rebuild a SPR from a group of images even from scratch, and that the game accepts it as valid. If you keep the same width and height as the original image it should accept them.
I'm not 100% sure about this, because the game could be expecting "something" from the data. As an example, if the game allocates M bytes of memory to load the compressed data into a buffer and your compressed data is bigger, you'll end up overwriting memory you weren't supposed to overwrite.

The easiest way to test this is to get the biggest compressed image, decompress it and save it uncompressed (but encoded) to the same SPR; then run the game.


Now, I have something else to ask of you because I'm having some problems. I posted to Mirex's trashbin (http://bin.mypage.sk/FILES/SPR_img.zip) some images. I need you to tell me what the colors should be (or give me a link to a picture with them) as I'm not much of a gamer and don't recognize the game.


I included the VB .Net code I used to generate them just in case you want to look a bit. As I said, I'll rewrite it to VB6. The only public function there (last one) takes the source file, the "table" from where you want to take the image (at the moment 0 works fine), the image index, a color to substitute the compressed pixels and the destination file. It exports the specified image to a BMP file. The other functions are auxiliary and private.

I'm not sure BMP is the right format to export to, but as there are a lot of editors/converters you can use, I thought it was fine. I say this because there are transparency and masks that other formats support better. Oh, I convert the images from 16bpp to 24bpp. I think there are 16bpp BMPs, but I'm not sure about the kind of support they have.
(I took the info on BMP from Wikipedia.)

As is, the code has some bugs:
- Colors are wrong. (Or I'm not used to those new colorful games. :D)
- Some re-interpretation for colors may be needed as I'm really using 15bpp and discarding the 16th bit.
- Output BMP is flipped because I forgot BMPs start at the bottom and go up.
- Only works fine with "table" 0. I have some problems with "table" 1.
- VB .Net arrays suck. (I'm used to C and VB6 kinds of arrays.)


Maybe I overcomplicated things with the description I gave in a previous post, so here is a simplification (I hope).

The general structure for SPRs is:

Code: [Select]
- Signature.
- Image count.
- Widths for all the images. (Array from 0 To image_count-1)
- Heights for all the images. (Array 0 To image_count-1)
- Pointer to second table ("table 1").
- First table ("table 0"):
    - Pointers to images. (Array from 0 To image_count-1)
    - Collection of compressed images. (image_count images.)
- Pointer to end of secod table or third table?
- Second table ("table 1"):
    - Pointers to images. (Array from 0 To image_count-1)
    - Collection of compressed images. (image_count images.)

And sometimes "table 1" data is missing. (I'm looking into it.)


Now, the trick to work with those files is to know where each element starts, its size and the encoding.

a) The signature seems to be the same in all the files you posted:
    - It starts at offset 0.
    - Its size is 11 bytes (including the 0x00).
    - The encoding doesn't matter, but in ASCII it's the string "DCSPRITE10" followed by the byte 00.
b) The image count tells the total number of images that are stored inside the SPR file.
    - It starts at offset 11. (Because the signature is fixed.)
    - Its size is 4 bytes.
    - It's the representation of a 32-bit integer.
c) The widths for the images are an array of integer with the widths in pixels.
    - It starts at offset 15.
    - Its size is '4*image_count' because each element is 4 bytes long.
    - Each element represents a 32-bit integer.
d) The heights for the images are an array of integers with the heights in pixels.
    - It starts at offset '15+4*image_count'.
    - Its size is '4*image_count' because each element is 4 bytes long.
    - Each element represents a 32-bit integer.
e) The pointer to the second table tells where in the file the pointers to the second group of compressed images start.
    - It starts at offset '15+4*image_count*2'.
    - Its size is 4 bytes.
    - It's the representation of a 32-bit integer.
      Note: This is not a direct offset in the file, but it has to be multiplied by 2 and added to a base offset instead. (* See below.)
f) The first table of pointers tells where each image from the first table can be found in the file.
    - It starts at offset '15+4*image_count*2+4'
    - Its size is '4*image_count' as each element is 4 bytes long.
    - Each element represents a 32-bit integer.
      Note: Each element is not a direct offset in the file, but they have to be multiplied by 2 and added to a base offset instead. (* See below.)
g) The collection of compressed images for the first table has all the images from the first table in compressed form.
    - It starts at offset '15+4*image_count*2+4+4*image_count'.
    - Its size is variable and it depends on the size of the images and the compression obtained.
    - Each element is a compressed image and is described somewhere else.
h) Not sure how to interpret the pointer to the end of the second table (or third table).
    - It's start is determined by the pointer explained in (e).
    - Its size is 4 bytes.
    - It's the representation of a 32-bit integer.
      Note: This is not a direct offset in the file, but it has to be multiplied by 2 and added to a base offset instead. (** See below.)
i) The first table of pointers tells where each image from the second table can be found in the file.
    - Its start is determined by adding 4 to the offset of (h).
    - Its size is '4*image_count' as each element is 4 bytes long.
    - Each element represents a 32-bit integer.
      Note: Each element is not a direct offset in the file, but they have to be multiplied by 2 and added to a base offset instead. (** See below.)
j) The collection of compressed images for the second table has all the images from the second table in compressed form.
    - It starts '4+4*image_count' bytes after the start of the array seen in (h).
    - Its size is variable and it depends on the size of the images  and the compression obtained.
    - Each element is a compressed image and is described somewhere else.

(*) As the size of the data inside the images is always a multiple of 2 bytes, the pointers seem to be expressed as half-words (16 bits) instead of bytes, so they have to be multiplied by 2.
    These pointers (multiplied by 2) are relative to a base offset. In the case of the pointers described in (e) and (f), the base offset is the start of (g).
(**) Similar to (*), but the base offset for the pointers in (h) and (i) is the start of (j).


I left out the explanation for the compression, but I'll explain it in some other post.

I think this explanation is more simple than the previous one (I may be wrong) and if you decide to make an editing tool for SPRs, I'll recommend you take your time to understand the general outline of the files. A pencil and paper, plus a hex-editor should help you a lot. :)
Try to find the start of all the components and then the blocks of data: you'll notice similar elements and such.


About file modification of SPRs, don't try to add new images to the existing files: just copy the data you can take from the original file (with the modifications necessary to create a valid SPR) and add/remove the data you want.
Another thing you can do is export all the images at once, modify them and rebuild the SPR. This is probably easier to do than writing functions for individual images, but you'll need more disk space at once.


-------------------------

Edit: Corrected the tables a little bit as I was missing a field.
Title: Re: Figuring out file formats
Post by: rmco2003 on 2006-06-23 05:31:33
Alright, here are the images with the correct colours, I couldn't get a front facing one of the prize one though:

(http://img.photobucket.com/albums/v484/rmco2003/1.png)
(http://img.photobucket.com/albums/v484/rmco2003/3.png)
(http://img.photobucket.com/albums/v484/rmco2003/4.png)
(http://img.photobucket.com/albums/v484/rmco2003/2.png)

[EDIT] pseudo code would be great, that would be very helpful :-P
Title: Re: Figuring out file formats
Post by: Tonberry on 2006-06-23 23:25:14
OK, thanks. The colors look right now and I solved some of the bugs. I'll start the VB6 version tomorrow, so you can have it on monday.

No problem about the pseudo-code, but start thinking on how you want to work with the extract/import functions; that is, if you want to be able to export/import/delete/update one image at a time or if you want to dump all the data once, modify it and then rebuild the SPR. (Or some other thing.)
Title: Re: Figuring out file formats
Post by: rmco2003 on 2006-06-23 23:42:06
Well I think extracting all of the images then rebuilding the SPR would be better for what I'd be using the code for. When you mention fixing bugs, what exactly were they, and did they relate to your comments about the amount of 00's after the fiie signiture? I'm just wondering about how I'd be able to account for variable lengths of those bytes... Have you managed to access multiple tables rather than just the first one?

Also I was wondering if you solved the problem of some of the SPR files not having table 1 data?
Title: Re: Figuring out file formats
Post by: Tonberry on 2006-06-24 15:39:00
I haven't posted any changes, but the things I fixed were:
- The colors.
- I can get images from both tables, without problems from the files you uploaded. The icons only use the first table (0). The sprites use the first table (0) to store the image with colors and the second table (1) to store the shadows. That means the widths and heights for each image are the same if you use table 0 or table 1.
- The amount of 00's is not after the signature. (Did I ever say so?) The signature includes only one 00 at the end (offset 0x0000000A).
- The thing I said about a lot of 00s was related to the image data, for which I haven't posted an updated reference, but the VB .Net version I posted had the solution. (See variable "height2".)
I said image data started with F0 00 00 00 and had 0x74 bytes with value 0x00 for "npc_priz_card.spr", but that is not the right interpretation of the data. The first 2 bytes of the image data are equal to its height and I think the real meaning is "number of compressed scanlines" but they are the same in the files you posted so I don't have a way to know. (So F0 00 = 240 is the number of rows you'll have to decode.) That's the reason the code shows a message box telling you the size for the height is not the same as "height2".
The other (now) 0x76 bytes with 00s are compressed scanlines. Every '00 00' is one scanline for which there are no sub-blocks, so the program has to jump to the start of the next scanline. They are trated as the other scanlines.
(So the real compressed data for the images starts 2 bytes after height2.)


Some additional notes:
- The compressed pixels seem to be only the ones that are invisible; so they define the mask for the sprite.
- Each table seems to refer to a layer in the drawing process of a sprite and there is the possibility that there are more layers in other files.

I'll give you an updated doc with the "final" format.

Remaining known bugs:
- Interpretation of "last table". It seems the pointer to the next table points to 00 00 00 00 when you are in the last table. (I'm almost sure, but I haven't added this to the code to test.)
- Flipped BMPs. That's trivial to solve and I'll do it for the VB6 version.


Additional note: All the offsets I've used are 0 based, but binary files are 1 based in VB6.
Title: Re: Figuring out file formats
Post by: rmco2003 on 2006-06-24 16:03:50
Alright then, I was just wondering if there would be any issues with any other files. Here's a link with 15 other SPR files, I know it's a lot but it should help to work out information about the first 2 bytes of the data.

Just to let you know these are much larger than the other files I posted, and they'll probably be the ones I'm editing.

[link removed]

#### MOD EDIT ####
Hi, this is your friendly neighborhood moderator. Discussion of formats are fine, screenshots are OK, and snippits of data is workable. However posting whole files can get us in trouble. Please don't.

ktanks ^_^

-Halkun

[EDIT]

Sorry about that, anyway I compiled your code with VB.NET express and extracting images from the files I posted works just fine, so it seems that your solution works fine.

[EDIT2]

There is one problem I noticed with extracting from attack information, namely it doesn't get anything from any table or image number I choose. Maybe these are the sprites without table data that you were talking about? Unless I'm remembering what you said wrong... I'd post one but I'm not sure if I'm allowed to (need confirmation halkun). If BMP is a bad format to convert to for transparency, etc, then I'd gladly accept another format, I'd be using Paint Shop Pro X to edit these files, and that program can work with a large amount of file formats, so I'd go for 100% compatibility with the data rather than a common file format.

Once again thanks a lot for your help, thanks to your .NET code (I installed .NET express on another computer) I've been able to create a list of all of the files I need to edit.

Also a final note, with some of the newer graphics, there is partial transparency (can't remember the exact term), sort of translucent, and the border around it completely transparent. It seems to use the same non-existant colour, I noticed this is a pure white, since the translucent colour is white. I noticed all of this when I extracted these menus and compared them to in-game screenshots.
Title: Re: Figuring out file formats
Post by: Tonberry on 2006-06-25 14:41:27
I think I finished the program. I left it in Mirex's trashbin (http://bin.mypage.sk/FILES/SPRExport.zip). It has all the bugs I could find, fixed. (I deleted the previous files, as there's no sense in keeping them.)

The structure of SPRs seems to be complete now (there weren't any changes) so I'll start with the "final" document. I tested all the files you posted and the program works fine.
I corrected a lot of bugs in the BMP generation, as the code I previously posted worked fine with images that had a number of columns divisible by 4 but had serious problems for other images.

I ended up assuming the first 2 bytes of the image data were the number of scanlines but left the message box to mark possible differences with the height. Anyway, I think it's unlikely you'll find files with any difference in those fields, because all images must have been compressed with the same tool and the only motive I see to make the numbers different is to compress a little bit more; but the compression only is used to encode the mask, not to save space.

I also took into account the possibility that there are more than 2 tables (layers) and the program assumes they have the same structure as tables 0 and 1.

The only function you'll find to work with SPRs is the one that exports one of them to BMP. If you want, you can add functions to get the number of tables, number of images, or anything else; just take the export function as an example.

The export function is SLOOOOOWWWWW because it doesn't have any kind of optimization: data is read/written byte by byte, function calls are used inside the loops and it's written in VB :D. It isn't a model for good programming practices, but you have a lot of comments. (More than I'm used to.)

Be very careful with the data types. I had to use some tricks to avoid the automatic type casts that VB6 does. Try to define all the integer variables as Long instead of Integer and be careful when you convert data to Long or define constants, because if VB converts them to Integer first, the numbers between 32768 and 65535 can be converted to negative numbers without you knowing. (VB isn't suited for this kind of job.)

You'll have to modify the examples in Form1 to your needs. Just don't uncomment all the lines at once or you'll get bored of waiting.

One last thing... when choosing the mask color to use, you are free to use any color, but if you choose a color with one of the following properties, you can be sure the mask won't conflict with any color the image has:
- Any component of red not divisible by 8.
- Any component of green not divisible by 4.
- Any component of blue not divisible by 8.
This is valid for the 24bpp representation (0-255 for each color) because when you convert from 16bpp to 24bpp you are not mapping to all possible colors.

Feel free to ask any question you have about the program.
Title: Re: Figuring out file formats
Post by: rmco2003 on 2006-06-25 14:51:24
Awesome Tonberry, I'll check it out when I get home, as I've only got .NET on this computer. Thanks for all of your help,  you'll be in the credits :wink:
Title: Re: Figuring out file formats
Post by: Tonberry on 2006-06-25 16:01:54
Quote
There is one problem I noticed with extracting from attack information, namely it doesn't get anything from any table or image number I choose. Maybe these are the sprites without table data that you were talking about? Unless I'm remembering what you said wrong... I'd post one but I'm not sure if I'm allowed to (need confirmation halkun).

Try the VB6 code first as it has less bugs. The problem you see may be with images that have a width not divisible by 4 as those images created corrupted BMPs. That's fixed with the VB6 version.
Try it with all the SPRs you can and make a list of the ones that didn't work.

The VB6 version recognizes SPRs with 0, 1, 2 or any number of tables. It just assumes the meaning for tables 2, 3, etc. is the same as for 0 and 1, but that may be wrong. Anyway, I didn't find anything besides table 0 and 1.


Quote
If BMP is a bad format to convert to for transparency, etc, then I'd gladly accept another format, I'd be using Paint Shop Pro X to edit these files, and that program can work with a large amount of file formats, so I'd go for 100% compatibility with the data rather than a common file format.

I don't think you'll have problems working with BMPs. The compatibility can be 100%, but you have to put each layer into a different BMP and you can select any color to be the mask.

It's not that BMP is bad, it's that the 24bpp version doesn't directly support masks/transparency as other formats do and I'm not sure how good the support is for other color representations. It doesn't support layers and there may be more than 2.
I understand you see this as something negative, but also think that BMP is a very simple format, quite good for learning how to convert from one format to another. Keep in mind that I'm only providing this code as an example.

Quote
Also a final note, with some of the newer graphics, there is partial transparency (can't remember the exact term), sort of translucent, and the border around it completely transparent. It seems to use the same non-existant colour, I noticed this is a pure white, since the translucent colour is white. I noticed all of this when I extracted these menus and compared them to in-game screenshots.

There are many possibilities for that:
- It's possible that the SPR format is not complete. (But I covered every byte.)
- Perhaps another layer is used for transparency. How many tables do you have for those images?
- Perhaps the transparency/translucency is programmatically added. Something like: if I'm drawing a button with size dx*dy, blend it with the background using a rectangle of size dx*dy. Are those images rectangular?
- It may be some kind of filter.
- I noticed some of the buttons you provided had an effect simulating a diagonal reflection of light crossing the button and a gradient for the borders. Maybe it's something similar.

Try changing the mask color to black/green/red or some unused color.

I'd suggest you get the BMP to SPR conversion working, modify one image and try it with the game to see what happens.

Title: Re: Figuring out file formats
Post by: rmco2003 on 2006-06-25 19:58:48
Yes I think they are rectangular, I'm going to be coding the conversion in the morning, I'll probably make your code save things like the amount of tables and layers in one SPR file to a text file, so that it can be rebuilt properly.

[EDIT] I've worked out now that the transparency is done in the second extracted layer, so it seems to be fine. One thing I'm not sure of though is when I'm making the SPR file, how would the layer thing work? In one of the files it extracts two images, the first image is the main one, and the second contains the transparencies, does this mean that they are stored as image 0, and image 1 in table 0? If this is so then it should be alright, I'm just wondering about how complicated the solution will be.

[EDIT2] I've now got the code writing the file signiture, and image amount to an spr file, I've also got code for reading and writing the widths and heights but I need to know where they are written in the BMP header. I've currently got it reading 4 bytes from offset 0x48 for the width, and 4 bytes from offset 0x59 for the height for each bitmap image. It seems to store the data fine, but could you confirm that the offsets are correct please?

[EDIT3] I'm also confused about the pointers for the tables, how exactly would I implement the whole table structure? Do they have a set amount of 32 bit integers or is it variable? What is the base offset you need to multiply the half words by?

Also relating to the second table, is each image stored in a seperate table? Or is there a size limit for the amount of images in each table?

You also mentioned that in some of the files the second table had an offset pointing to a group of 00's, does this mean it doesn't have anything in the second table? How can I tell what SPR files have second tables?

I'm trying to avoid running into size problems, where the game tries to read from the wrong location because the files haven't been padded, etc.

I'm seriously considering modifying your code to output structure information from the SPR files so that I can just read the data and transfer the BMP pixel data directly, I think it would be much easier, and safer.

Waiting for your thoughts :-P
Title: Re: Figuring out file formats
Post by: Tonberry on 2006-06-26 16:04:46
Wow, that's a lot of questions... and edits. :) I work so I won't be able to answer your questions too fast, but I'll do it eventually.

Quote
Yes I think they are rectangular, I'm going to be coding the conversion in the morning, I'll probably make your code save things like the amount of tables and layers in one SPR file to a text file, so that it can be rebuilt properly.

That's a good idea. I'd even recommend you modify the program to export all the data and images at once, but that's practically a rewrite.

Quote
I've worked out now that the transparency is done in the second extracted layer, so it seems to be fine. One thing I'm not sure of though is when I'm making the SPR file, how would the layer thing work? In one of the files it extracts two images, the first image is the main one, and the second contains the transparencies, does this mean that they are stored as image 0, and image 1 in table 0? If this is so then it should be alright, I'm just wondering about how complicated the solution will be.

That's the thing about BMPs and layers. As BMP doesn't support layers (and I didn't even know there where layers at first), you get 1 BMP for each layer.

About how those particular images+transparency are stored in the SPR, well that depends on how you extracted them:
- If you extracted them from the same table, the those will be different images and they'll be stored as image 0 and image 1 in table 0.
- If you extracted the first one from table 0 and the second one from table 1, then you'll have to make sure you store them as image 0 in table 0 and as image 0 in table 1.

In the case of the first files you posted (the ones with the prize), the sprites  had the colors in table 0 and the shadows in table 1, so in the file you have:

table0
  image 0 (normal)
  image 1 (normal)
  image 2 (normal)
  ...
table 1
  image 0 (shadow)
  image 1 (shadow)
  image 2 (shadow)
  ...

Quote
I've now got the code writing the file signiture, and image amount to an spr file, I've also got code for reading and writing the widths and heights but I need to know where they are written in the BMP header. I've currently got it reading 4 bytes from offset 0x48 for the width, and 4 bytes from offset 0x59 for the height for each bitmap image. It seems to store the data fine, but could you confirm that the offsets are correct please?

Here's a link: http://en.wikipedia.org/wiki/Windows_bitmap

That's where I took the info on BMPs from. It says the offsets are 19 (decimal) for the width in pixels and offset 23 (decimal) for the height. Those offsets are base 1, so that's what you should tell VB6. You have to read 4 bytes to get the width and 4 bytes to get the height.

Quote
I'm also confused about the pointers for the tables, how exactly would I implement the whole table structure? Do they have a set amount of 32 bit integers or is it variable? What is the base offset you need to multiply the half words by?

It's variable, but if you know the number of images that will be part of the SPR, you know the size. For each layer you have:
- A pointer to the next layer (4 bytes).
- A table of pointers to images in the current layer. Each pointer is 4 bytes long and there is 1 pointer for each image. (The total size for the array is 4*image_count)
- Images of the layer. (Variable size.)

The total size for the layer is 4+4*image_count+"the sum of the sizes of the compressed images".

No, there are no half-words in the tables, unless you are confussing the words you have to multiply by 2 with half-words.
A half-word is a 16 bit number in a 32 bit architecture, but the pointers you'll find in the tables are 32 bit, so they are words. The thing with those (32 bit) pointers is that to get a pointer in bytes, you have to multiply their value by 2.

Once you multiply the pointer by 2 to convert to a displacement in bytes, you have to add the base offset. The base offset for all the pointers in a layer is the offset of the first image of the layer.
That means that if you get to the offset of the layer (where the pointer to the next layer is), you'll have the base offset for the pointers '4+4*image_count' bytes afterwards. So:

base_offset = layer_offset + 4 + 4*image_count

You won't find "base_offset" nor "layer_offset" in the program, sorry; those are concepts I'm using now that I have a better idea of the format.

Quote
Also relating to the second table, is each image stored in a seperate table? Or is there a size limit for the amount of images in each table?

The second layer has a structure similar to the first layer:
- Pointer to next layer. (4 bytes)
- Table of pointer to images of second layer. (4*image_count bytes)
- Images for second layer. (Variable size.)

image_count is the same for all the layers and the widths and heights of the images are the same for all the layers. You'll notice that image_count, widths and heights are only defined once at the start of the file, before the first layer, so all the tables have the same number of elements (and size) and if image Z from table 0 has width dx and height dy, then image Z from table 1 will have dx*dy pixels. (The size in bytes for image Z in table 0 and 1 may be different because of the compression.)

Every table has the same amount of images because in reality they are different layers of the same images.

Quote
You also mentioned that in some of the files the second table had an offset pointing to a group of 00's, does this mean it doesn't have anything in the second table? How can I tell what SPR files have second tables?

As I said above, all the layers have the same structure:
- Pointer to next layer. (4 bytes)
- Table of pointer to images of current layer. (4*image_count bytes)
- Images for current layer. (Variable size.)

This allows you to go jumping from layer to layer using the pointer to the next layer. (Don't forget to multiply the value by 2 and add the base_offset of the layer.)

If you get to a layer where the value of the pointer is 0x00000000, then the last layer was the previous one and this layer doesn't have information. There are 4 more bytes after this pointer but the value is 0x00000000 too and there is no table nor images; you reached the end of the file.

It's a little strange, as they could have put the 0x00000000 in the previous pointer, but that's how it is.

So, to get the number of tables you have to jump from layer to layer until you reach 0x00000000.

Quote
I'm trying to avoid running into size problems, where the game tries to read from the wrong location because the files haven't been padded, etc.

I'm seriously considering modifying your code to output structure information from the SPR files so that I can just read the data and transfer the BMP pixel data directly, I think it would be much easier, and safer.

Be my guest. :) For what you want the data, it'll be easier if you load all the widths and heights into arrays and do the same for the pointers to the images, then dump the images to BMP sequentially, but that's a rewrite.

If you were using another programming language that supports unsigned values, you could define types for the headers or arrays for the tables to read bunchs of data. You can do that with VB6, but you have to be careful with the automatic type casting because you can end up with wrong values.

Title: Re: Figuring out file formats
Post by: rmco2003 on 2006-06-27 09:25:14
There are some files that definately aren't working, here's 5 of the problem spr's:

http://bin.mypage.sk/FILES/notworking.zip

This is the only problem I seem to be having with extraction, some sprs just don't work with the code..
Title: Re: Figuring out file formats
Post by: Tonberry on 2006-06-27 16:34:10
You shouldn't have poblems with the aura9. The other files have some differences to previous SPRs, but not much:
- The signature is different: "DCSPRITE0.5". That's the reason the export function returns. Be careful because this signature has 11 bytes like the previous one, but it doesn't include 00 at the end.
- The pointer to the next layer seems to point past the end of file when you are processing data from the last layer. (The previous version pointed to 0x00000000.) This will give you an error, so before jumping to the next layer, check to see if you'll end up outside the file.

That's it. I also noticed the image data was encoded a little bit different, but the function I wrote can decode it without modification.

To have the function working again, just compare both signatures and check you won't end up past the end of file when moving to the next layer.
Title: Re: Figuring out file formats
Post by: rmco2003 on 2006-06-27 17:23:52
Well I've only done a half arsed job, but it seems that the incorrect pointer for the tables doesn't matter. I just allowed the code to recieve SPR files with the older signature and it can extract the images I couldn't do before just fine now.
Title: Re: Figuring out file formats
Post by: Tonberry on 2006-06-27 22:57:33
Here is some code to encode one row from an image to the compressed SPR form. Everything is hardcoded so you'll have to modify it.

To write the information of a complete image to a SPR you have to:
- Write the number of rows to the file.
- For each row:
    - Compress the row using a subroutine similar to the one posted. (Don't forget to convert from 24bpp to 16bpp every pixel.)
    - Write the compressed row to the file.

All the elements from the image data are 2 bytes long each, so you'll have to use the function Write2 or something similar.


Code: [Select]
Option Explicit

Private Const CNST_VERSION = 10  ' SPR version to compress data to. Valid: 0.5 and 10
'Private Const CNST_VERSION = 0.5

Private Sub Command1_Click()
  ' This algorithm encodes a row from an image to SPR format.
  ' To encode a complete image, all individual rows must be processed by this subroutine.


  ' Input parameters.
  Dim width As Long          ' Image width.
  Dim row(0 To 319) As Long  ' Row of pixels to compress. (Array from 0 to width-1.)
  Dim mask_color As Long     ' Color used to encode the mask.
 
  ' Output parameters.
  Dim subblocks As Long                 ' Number of sub-blocks after compression.
  Dim compressed_row(0 To 480) As Long  ' Array that holds compressed row. (Worst case: array from 0 to width*3/2)
  Dim compressed_x As Long              ' Number of elements from 'compressed_row' that effectively hold compressed data.
 
  ' Auxiliary variables.
  Dim x As Long                   ' Column from 'row' being processed.
  Dim compressed As Long          ' Number of consecutive pixels with mask color.
  Dim uncompressed As Long        ' Number of consecutive pixels without mask color.
  Dim uncompressed_start As Long  ' Used to hold the place in 'compressed_row' where the number of consecutive pixels without the mask color must be written.
 
 
  ' Initialization of input parameters. (For the example.)
  width = 320
  mask_color = 0
 
  'Put some values into the row just to test.
  row(10) = 8
  row(11) = 9
  row(12) = 8
  row(55) = 7
  row(57) = 7
 
 
  ' Initialization.
  subblocks = 0
  compressed_x = 0
  x = 0
 
  ' For every column...
  Do While x < width
   
    ' Get the number of consecutive compressed pixels.
    compressed = 0
    Do While x < width
      If row(x) <> mask_color Then
        Exit Do
      End If
      compressed = compressed + 1
      x = x + 1
    Loop
   
    ' Add the number of consecutive compressed pixels to the compressed data.
    compressed_row(compressed_x) = compressed
    compressed_x = compressed_x + 1
     
    ' Hold the position where the number of consecutive uncompressed pixels must be stored.
    uncompressed_start = compressed_x
    compressed_x = compressed_x + 1
   
    ' Get the number of consecutive uncompressed pixels, while adding them to the compressed data.
    uncompressed = 0
    Do While x < width
      If row(x) = mask_color Then
        Exit Do
      End If
      compressed_row(compressed_x) = row(x)  ' *** TODO: Convert the pixel from 24bpp to 16bpp.
      compressed_x = compressed_x + 1
      uncompressed = uncompressed + 1
      x = x + 1
    Loop
   
    ' Store the number of consecutive uncompressed pixels.
    compressed_row(uncompressed_start) = uncompressed
   
    ' End compressed row data depending on SPR version.
    Select Case CNST_VERSION
      Case 0.5
        ' Version 0.5 simply adds last sub-block.
        subblocks = subblocks + 1
       
      Case 10
        ' Version 10 only adds last sub-block if it has uncompressed pixels.
        If (uncompressed = 0) Then
          compressed_x = uncompressed_start - 1
        Else
          subblocks = subblocks + 1
        End If
       
    End Select
  Loop
  ' End of subroutine.
 
 
  ' Show compressed data.
  Debug.Print subblocks
  For x = 0 To compressed_x - 1
    Debug.Print compressed_row(x) & "; ";
  Next x
  Debug.Print
End Sub
Title: Re: Figuring out file formats
Post by: mirex on 2006-06-28 09:47:13
Hi, I had no time to comment or look into the file format sooner. I see that you have decoded everything already.

For your information BMP has a possibility to store alpha transparency layer, it can be accomplished by storing 32bit BMP's. You can see these layers in Paint Shop Pro too. I use PSP6/7 and you can see the layer through menu / Mask / Load from alpha channel.

Tonberry: may I use your description and code SPR support into my program Biturn ? You will be credited of course.

rmco2003: I haven't seen the name of the game so far, please can you tell me what game is it ?
Title: Re: Figuring out file formats
Post by: halkun on 2006-06-28 12:56:55
taking a stab, it's either FF:Tacticts ot Xoneogears.

However sprites were a PSY-Q thing. So they might be generic.
Title: Re: Figuring out file formats
Post by: rmco2003 on 2006-06-28 13:06:35
It's just a random Korean MMORPG I'm working on. Mirex if you allow SPR conversion both ways I'll be very happy indeed :-P I think I've reached a point now where my ability isn't good enough to get this done, all I've managed to do is write the easiest things to a new file, and that's using existing code as a base..

Looks like I'm begging again.. :wink:
Title: Re: Figuring out file formats
Post by: mirex on 2006-06-28 13:39:32
rmco2003: you should continue on your program. I don't know how long will it take me untill its finished, I have only a little of free time.
Title: Re: Figuring out file formats
Post by: rmco2003 on 2006-06-28 15:35:14
I suppose, I'll just have to keep trying to figure out what to do until I get somewhere..
Title: Re: Figuring out file formats
Post by: Tonberry on 2006-06-28 16:07:53
Mirex: No problem, use it however you like. No need for credits unless it's custom.

Thanks for the info on BMP. I don't have PSP so I used Paint to test and it doesn't have 32-bit BMPs.


rmco2003:

This is the data you should save when decoding a SPR to be able to rebuild it later:

1) Signature.
2) Number of images.
3) Number of layers.
4) List with the widths.
5) List with the heights.
6) List with the names of the BMPs generated. (Choose the order. I recommend you list the images in the same order as they are stored in the SPR.)

I'll call this file the "SPR definition file" in this example. I'm assuming you won't change the width or height of the images. (If you'll change them, then don't save them and just get them from the BMPs.)

To rebuild a SPR you'll need that file and all the BMPs. Then you can do something like this:

1) Get the signature from the SPR definition file and write it to the new SPR.
2) Get the number of images from the SPR definition file and write it to the new SPR.
3) Write the list of widths to the new SPR.
4) Write the list of heights to the new SPR.
5) Compress all the BMPs in separate files.
5.1) Load one BMP into memory.
5.2) Create a new file with extension ".SPRI" (or other) and the same name as the BMP.
5.3) Write the height of the image to the SPRI file.
5.4) For every row of the BMP:
5.4.1) Compress the row.
5.4.2) Write the number of sub-blocks generated to the SPRI file.
5.4.3) Write all the compressed data to the SPRI file.
5.5) Get the size in bytes of the SPRI generated and save it to an array for later use.
5.6) Close the SPRI file.
5.7) Repeat for the next BMP, starting at 5.1.

At this point we have the header of the SPR, the sizes of the encoded images and the encoded images. That's enough to build the layers.

6) Add all the layers.
6.1) Calculate the pointer to the next layer and write it to the SPR file. To do this, simply add the size in bytes for all the encoded images of the layer and divide the number by 2.
6.2) For every image in the layer, calculate the pointer to the image and write it to the SPR file.
6.2.1) Take 0 as the pointer to the first image of the layer.
6.2.2) For image I, take the pointer to it as the pointer to previous image (I-1) and add the size of the encoded image (I-1) divided by 2.
6.2.3) Repeat for the rest of the images in the layer. (Go to 6.2.2)
6.3) Append all the SPRI files of the layer to the SPR file.
6.4) Repeat for the next layer. (Goto 6.1)
7) If the SPR version is 10 then write 8 bytes with value 0x00.
8 ) Close the SPR file.

Each subsection tells you how to complete a section.


That's a lot of things to do. I recommend you don't try to do everything at once, but write one step and compare the result to the original SPR. (Don't modify the BMPs at first as it will be easier to find the differences.)

Hope it helps.


Edit: Corrected a mistake in 6.2.2
Title: Re: Figuring out file formats
Post by: rmco2003 on 2006-06-29 07:57:04
I guess this is where my INI module comes in handy :-P

So much easier to record entries in an ini file than in a text log :wink:

Code: [Select]
[data]
signature=DCSPRITE10
imagecount=2
[measure]
width0=212
height0=330
width1=212
height1=330
[output]
image0=D:\Documents and Settings\Rhys\Desktop\dr2_sub_001_0000_i0.bmp
image1=D:\Documents and Settings\Rhys\Desktop\dr2_sub_001_0000_i1.bmp

[EDIT] I'm up to the point where I'm adding compressed data to the SPRI files. I can generate the files perfectly and add data to these files, but I'm having some issues with manipulating your code to compress one scanline. One of the problems is that the data is too small after compression to be realistic, think from something like 20k down to 2k, and when viewing the data there's a lot of blank space in between very small chunks of data. I'm just passing through to the files like this:

Code: [Select]
Put 3, , subblocks
For arraypos = 0 To (Width * 3 / 2)
    Put 3, , compressed_row(arraypos)
Next arraypos

I really doubt that code is right now.. maybe it's to do with the array size? I just saw the text "Worst case Width * 3 / 2" so I thought to put that there.

I'm also having some trouble figuring out how to convert the pixels from 24bpp to 16bpp, your functions require quite a few different variables, and I'm assuming I have to convert it all to RGB and then to 16bpp, and visual basic won't allow me to do Function1(Function2 each being converson functions of yours.

Perhaps I'm approaching this from the wrong angle, I'll have to take a deeper look into it tomorrow.
Title: Re: Figuring out file formats
Post by: Tonberry on 2006-06-30 00:02:36
It's nice to see you've already come this far. :)

Quote
One of the problems is that the data is too small after compression to be realistic, think from something like 20k down to 2k, and when viewing the data there's a lot of blank space in between very small chunks of data. I'm just passing through to the files like this:

The first layer of the first image of "npc_priz_card.spr" has a compressed size of approx. 9KB. When extracted to a 24bpp BMP the size is 226KB.

9/226 = 3.98%
2/20 = 10%

That means you are getting less compression than you could achieve.

To tell the truth, the compression depends a lot on how many pixels of mask there are in the image. If the image has no mask, you should expect a compression to about 67% (16bpp/24bpp) the size of the BMP. If you have a scanline with 320 pixels (960 bytes) as mask, assuming the width of the image is also 320 pixels, the algorithm for version 10 will encode it with only 2 bytes.

Quote
Put 3, , subblocks
For arraypos = 0 To (Width * 3 / 2)
    Put 3, , compressed_row(arraypos)
Next arraypos

Avoid using "Put" when writing to a binary file (in this application); use the functions "Write1", "Write2" and "Write4" instead. You can find them in 'AuxFileIO.BAS'.
In that example you are writing the value of 'subblocks' with a length of 4 bytes because 'subblocks' is a Long, but you should write only 2 bytes to conform to the SPR format. Use 'Call Write2(3, subblocks)' instead.
The same goes for the element of 'compressed_row'; just change it to 'Call Write2(3, compressed_row(arraypos))'.
That will also make the SPRI smaller.

You have to write only the elements of 'compressed_row' that go from 0 to 'compressed_x - 1'. The variable 'compressed_x' tells the number of elements in the compressed row.
The value 'width*3/2' is the maximium lenght a compressed row can have. I didn't really make the calculation so I may be wrong with that, but I think the worst case of compression is when a row has a sequence such as 'color-mask-color-mask-color-mask...'. In that case the algorithm doesn't compress the data, but expands it because it uses 2 values to encode the color pixel and 1 value to encode the mask. That's a total of 3 values for what was stored in only 2.

That bit of code should read something like this:

Code: [Select]
Call Write2(3, subblocks)
For arraypos = 0 To compressed_x - 1
    Call Write2(3, compressed_row(arraypos))
Next arraypos


Quote
I'm also having some trouble figuring out how to convert the pixels from 24bpp to 16bpp, your functions require quite a few different variables, and I'm assuming I have to convert it all to RGB and then to 16bpp, and visual basic won't allow me to do Function1(Function2 each being converson functions of yours.

How strange... I thought I'd put a function to do that, but it seems I forgot. Add this to 'Color.BAS': (I didn't test it.)

Code: [Select]
' Convert 24bpp to 16bpp.
Public Function Convert24bppTo16bpp(bpp24 As Long) As Long
  Dim R as Long
  Dim G As Long
  Dim B as Long

  Call Convert24bppToRGB(bpp24, R, G, B)
  Convert24bppTo16bpp = ConvertRGBTo16bpp(R, G, B)
End Function

Title: Re: Figuring out file formats
Post by: rmco2003 on 2006-06-30 06:02:33
I realised that my problem is that I was loading the BMP for read access, but I wasn't actually reading the rows into the array. I think I need to use a seek command then something like read4? The problem is I don't know where to seek to..
Title: Re: Figuring out file formats
Post by: Tonberry on 2006-06-30 17:24:45
According to the Wikipedia link, you can get the starting address of the bitmap from offsets 11-14 (base 1, so in VB do a seek to 11 and use Read4). The address you get will be base 0 so add 1 to it and do another seek to get to that address. (You'll probably end up at offset 55, base 1.)

There you'll have to read, using Read1, the three components for every pixel and convert them from RGB to 24bpp. The first '3*width' bytes have the pixels for the first row (the last one really, as BMP stores rows from bottom to top), then you'll have to check if '3*width' is divisible by 4 and if it's not, then read (and discard) as many bytes as necessary to make it divisible by 4. Look at what I did with the variable 'complete_row' in 'ImageBufferToBMP', but instead of writing, read. After that, process the next row.
Title: Re: Figuring out file formats
Post by: rmco2003 on 2006-06-30 17:52:46
So if I use

Code: [Select]
  For row = image_buffer.Height - 1 To 0 Step -1
    For column = 0 To image_buffer.Width - 1
      Call Read1(fh, blue)
      Call Read1(fh, green)
      Call Read1(fh, red)
      Call mdlColor.ConvertRGBTo24bpp(image_buffer.Color(column, row), red, green, blue)
    Next column

To read the rows, then it should work?

Also what do I do about this?

Code: [Select]
    For i = 1 To complete_row
      Call Write1(fh, 0)
    Next i

Is this a read too? Or do I do something else instead?

I'm just replicating your function and switching things around to make it follow your instructions.

Some sudo code or something would be extremely helpful for doing all of this seeking, and need to know things like are the images going to be upside down and all of the rest of it, which I'd probably end up doing seeing as I don't know the first thing about working with files in this way.

The code above is all I've been able to do so far, I've also seen you're making a new image buffer when making a BMP so I added that code to the loop I've made to process the compression code for each image so it makes a new buffer for each image.

I'm really confused now. :oops:
Title: Re: Figuring out file formats
Post by: Tonberry on 2006-07-01 01:44:41
Try something like this:

Code: [Select]
  For row = image_buffer.Height - 1 To 0 Step -1
    For column = 0 To image_buffer.Width - 1
      blue = Read1(fh)
      green = Read1(fh)
      red = Read1(fh)
      image_buffer.Color(column, row) = mdlColor.ConvertRGBTo24bpp(red, green, blue)
    Next column

    For i = 1 To complete_row
      discard = Read1(fh)
    Next i

  Next row

The 'For' that uses 'complete_row' is used to complete the rows with 00s to make it divisible by 4 bytes, as requiered by BMPs.
Don't forget to assign value to 'complete_row' as I did in 'ImageBufferTopBMP'.

Quote
I'm just replicating your function and switching things around to make it follow your instructions.

I know it seems hard, but as I said before, it'd be better if you try to understand what the function is suposed to do and write it in your own terms. The code I wrote is simply an example and it may not do the things the way you intend them to, that's why it's important you use it simply as a guideline.

Quote
Some sudo code or something would be extremely helpful for doing all of this seeking, and need to know things like are the images going to be upside down and all of the rest of it, which I'd probably end up doing seeing as I don't know the first thing about working with files in this way.

The code above is all I've been able to do so far, I've also seen you're making a new image buffer when making a BMP so I added that code to the loop I've made to process the compression code for each image so it makes a new buffer for each image.

I'm really confused now.

The images won't appear upside down because you started storing the first row of the BMP (the bottom row) in the last row of the image buffer. When saving the compressed rows just do the 'for' from 0 to 'height-1'.


Mmmm... I smell despair. :wink: Maybe I'm not that good explaining or you still need a little more practice programming, or both. If it's a real pain to you and you can wait some time, I'll complete it for you.
Please, don't misunderstand me, I have no problem explaining the details to you, but you stated your objective is to translate the images. Maybe you prefer to skip the programming. Let me know what you decide.

Title: Re: Figuring out file formats
Post by: rmco2003 on 2006-07-01 04:29:31
Well I know for a fact that my knowledge in programming is insufficient :wink: but I'll still give it a shot and see if I can get any further. I think dispair would be accurate :|
Title: Re: Figuring out file formats
Post by: Tonberry on 2006-07-01 14:02:08
OK, let's continue then. :)

About the pseudocode for seeking the start of the bitmap (that I didn't explain it in the previous post):
1) Seek to the position in the header of the BMP where it has a pointer to the start of the bitmap. (Thats offset 11, base 1.)
2) Read 4 bytes using 'Read4'. (Notice the value you read will probably be 54, because according to Wikipedia that's the first offset after the header.)
3) Add 1 to the value you read in (2) to change it from base 0 (specified by BMP format) to base 1 (used by VB6).
4) Do another seek to the offset given by the value you got from step (3) to jump to the start of the bitmap.

That should complete the loading of the BMP. I recommend you test the code that loads the BMP before continuing with the compression as it will make things easier if you find problems later.
To test the loading function, just write the image buffer you just generated to another BMP. You already have a function to write the header of a BMP and another function to write the bitmap from an image buffer and you know they work fine; if there is any error, it will be in the loading.

If the new BMP looks fine, then go on to the compression. If it doesn't, it's best to correct that before adding the compression.

Oh... and test it with BMPs of different sizes.
Title: Re: Figuring out file formats
Post by: rmco2003 on 2006-07-01 14:36:45
I can't get my head round this now, and I'm starting to get confused by my own code :oops: This is getting pretty hopeless now... I've put in your pseudocode but I don't even know what I'm meant to do with that offset I'm getting. I don't mind waiting if you can complete it :|
Title: Re: Figuring out file formats
Post by: Tonberry on 2006-07-01 15:25:10
All that seeking had the purpose to find the offset where the "real" information of the image is stored. That's the offset where you'll find the data (component colors) for the first pixel of the image, and after that, for the next pixel and so on. So, after getting to that offset you have to start reading the pixels with the code...

Code: [Select]
  For row = image_buffer.Height - 1 To 0 Step -1
    For column = 0 To image_buffer.Width - 1
      blue = Read1(fh)
      green = Read1(fh)
      red = Read1(fh)
      image_buffer.Color(column, row) = mdlColor.ConvertRGBTo24bpp(red, green, blue)
    Next column

    For i = 1 To complete_row
      discard = Read1(fh)
    Next i

  Next row

... which I haven't tested, so it may have some mistakes.

After that you'll get the image in the image buffer. Before adding the part that compresses the image to make it SPR compatible, test that the image you loaded into 'image_buffer' is fine. To test it simply write it to a new BMP using the functions you have to write BMPs. Compare the original BMP to the new one.

Once you are sure the image is loading fine, delete or (better) comment the code that tests the BMP is fine and start adding the SPR compression.


--------

As a side note, I'll tell you all that previous seeking is needed because the BMP is composed of a header+image_data. The usual way to read them is to read the header into a memory structure first, get the width, height, bits per pixels, offset where image_data starts, etc. Once all those values are known, it's time to get the image_data (that is, the colors of the pixels).
There's the possibility that an application that writes BMPs adds some info between the usual header and the image_data, making the header bigger, so you can't just jump to the image_data. First you have to find where it's located.

As a bigger side note and completelly offtopic, I'll comment that to better understand all this jumping it's better if you learn some programming language that allows the use of pointers; something like C or Pascal. Learning how to implement common data structures such as lists, double-linked lists, queues, stacks and trees using pointers is helful too. There was a book (probably many) by an author named Aho that had good examples in Pascal.
You can do the same with VB6, VB .Net, C#, Java or some other programming languages that allows for references instead of pointers, but it's not as instructive as using pointers (in my opinion.)
Also, keep in mind that VB6 is a much higher-level language than C and although it makes a lot of generic things easier (GUI, reports, database management, etc.), when you have to implement something too specific it's a real pain and you end up using Windows APIs or linking to code written in C or C++.
Title: Re: Figuring out file formats
Post by: rmco2003 on 2006-07-01 16:09:48
I don't think it's worth me trying to code this thing anymore, it's just not working out. I'm just getting more confused and I don't know if what I'm doing is even going to help at all. If you have any spare time to work on this for me I'd greatly appreciate your help, otherwise I'm just going to have to give up on making these files.
Title: Re: Figuring out file formats
Post by: Tonberry on 2006-07-01 19:35:38
OK, sorry. I'll try to complete it this week, just don't expect a complete application. I think I'll make a function to do the extraction and other to build the SPR back, following the same idea as above; that way you won't have to worry about the internals of the files.
Title: Re: Figuring out file formats
Post by: rmco2003 on 2006-07-01 20:11:29
That'd be great if you could. I can do things like opening and saving with common dialogs, GUI related things, and the like quite easily.
Title: Re: Figuring out file formats
Post by: Tonberry on 2006-07-08 05:35:52
I posted a new version of the code to Mirex's trashbin (http://bin.mypage.sk/FILES/SPRExport2.zip). I made a few changes to some of the functions that already existed and added two new public functions to work with SPRs:

- SPR_Export
  Given the name of a SPR file, the name of a SPR definition file and the color to use for the mask, creates the definition file and a BMP file for each image in the SPR.

- SPR_Build
  Given the name of a SPR definition file, the name of a SPR file and the color to use for the mask, creates the SPR file.

You can find some examples on how to use the new functions in Form1. The concept is similar to what we discussed in previous posts, but I changed it a little bit, as now all the images are extracted in one step (more or less) and the rebuilding process does not use any auxiliary files to store the compressed data.

It's not complete yet, as I've found a little problem. My idea was to create a couple of functions, one the inverse of the other: one to extract the images and the other to rebuild an identical SPR to the original one. The problem is that some SPRs end with a sequence of zeros, but the amount of zeros isn't allways the same. There are some cases where the amount of zeros seem to be related to the number of images and others where they don't seem related.

Despite that problem, the rest of the data is the same, so I'll ask you to get some SPRs, extract the images, rebuild the SPRs and try the ones that have a different size than the original with the game.
If you find any problem, just let me know.

Hope it works. :)
Title: Re: Figuring out file formats
Post by: rmco2003 on 2006-07-08 07:23:08
Awesome, finished on my birthday too :-P great timing :wink:

[EDIT] The spr creation functions work perfectly, I've already inserted some of my modified images fine and they operate just like the originals.

Thanks for all of your help tonberry, I couldn't of done it without you.
Title: Re: Figuring out file formats
Post by: Tonberry on 2006-07-08 14:56:37
Happy birthday! ;)

It's nice to know it's working, thanks. :)
Title: Re: Figuring out file formats
Post by: rmco2003 on 2006-07-28 19:47:11
Thanks again tonberry for your help, I was just wondering if anyone could spare some time to help me with some other file formats? There's some more formats for the game that I need help decoding, but mainly the ANI files that accompany the SPR files. I'd like to put together some sort of viewer for the game's various files so if anyone has some spare time could they look at these different formats?

There are 3 formats for the maps, named
.MAP
.OBJ
.TIL

and then there's the .ANI files for the sprites.

I'll start with what I have found, which is basically signatures.

In the .MAP files, it starts off with the text string "MAP1", followed by another text string which seems to be related maybe to other .MAP files that it requires.

In the .OBJ files, it starts off with the signature "DCOBJECT0.01", it is then padded by 14 bytes of the value "00", then there is data relating to the file for 8 bytes, and then there is further padding with the value "01", however it often has the value "00" mixed in with this padding too.

In the .TIL files, it starts off with the signature "DCTITLE0.03" followed by 2 bytes of padding with the value "00". There is then 6 bytes of data relating to the file, and then padding follows with the value "00004000".

In the .ANI files, it starts off with the signature "DCANIMATION0.5" followed immediately by the name of the SPR file it is related to, with no padding in between. It then has either data seperated by padding, or the whole thing is padding, but there is some repeating data after this.

I've uploaded samples of the files, and I've included the SPR files that relate to the ANI files.

The link to download the samples is from http://bin.mypage.sk/FILES/sub.zip

I'd really appreciate some help, mirex you said you looked briefly at the ANI files, do you think you could help with them?

[EDIT]

Could someone help me out please?