Figuring out file formats

[Tonberry]

You shouldn't have poblems with the aura9. The other files have some differences to previous SPRs, but not much:
- The signature is different: "DCSPRITE0.5". That's the reason the export function returns. Be careful because this signature has 11 bytes like the previous one, but it doesn't include 00 at the end.
- The pointer to the next layer seems to point past the end of file when you are processing data from the last layer. (The previous version pointed to 0x00000000.) This will give you an error, so before jumping to the next layer, check to see if you'll end up outside the file.

That's it. I also noticed the image data was encoded a little bit different, but the function I wrote can decode it without modification.

To have the function working again, just compare both signatures and check you won't end up past the end of file when moving to the next layer.

[rmco2003]

Well I've only done a half arsed job, but it seems that the incorrect pointer for the tables doesn't matter. I just allowed the code to recieve SPR files with the older signature and it can extract the images I couldn't do before just fine now.

[Tonberry]

Here is some code to encode one row from an image to the compressed SPR form. Everything is hardcoded so you'll have to modify it.

To write the information of a complete image to a SPR you have to:
- Write the number of rows to the file.
- For each row:
    - Compress the row using a subroutine similar to the one posted. (Don't forget to convert from 24bpp to 16bpp every pixel.)
    - Write the compressed row to the file.

All the elements from the image data are 2 bytes long each, so you'll have to use the function Write2 or something similar.

Code: [Select]

Option Explicit

Private Const CNST_VERSION = 10  ' SPR version to compress data to. Valid: 0.5 and 10
'Private Const CNST_VERSION = 0.5

Private Sub Command1_Click()
  ' This algorithm encodes a row from an image to SPR format.
  ' To encode a complete image, all individual rows must be processed by this subroutine.


  ' Input parameters.
  Dim width As Long          ' Image width.
  Dim row(0 To 319) As Long  ' Row of pixels to compress. (Array from 0 to width-1.)
  Dim mask_color As Long     ' Color used to encode the mask.
 
  ' Output parameters.
  Dim subblocks As Long                 ' Number of sub-blocks after compression.
  Dim compressed_row(0 To 480) As Long  ' Array that holds compressed row. (Worst case: array from 0 to width*3/2)
  Dim compressed_x As Long              ' Number of elements from 'compressed_row' that effectively hold compressed data.
 
  ' Auxiliary variables.
  Dim x As Long                   ' Column from 'row' being processed.
  Dim compressed As Long          ' Number of consecutive pixels with mask color.
  Dim uncompressed As Long        ' Number of consecutive pixels without mask color.
  Dim uncompressed_start As Long  ' Used to hold the place in 'compressed_row' where the number of consecutive pixels without the mask color must be written.
 
 
  ' Initialization of input parameters. (For the example.)
  width = 320
  mask_color = 0
 
  'Put some values into the row just to test.
  row(10) = 8
  row(11) = 9
  row(12) = 8
  row(55) = 7
  row(57) = 7
 
 
  ' Initialization.
  subblocks = 0
  compressed_x = 0
  x = 0
 
  ' For every column...
  Do While x < width
   
    ' Get the number of consecutive compressed pixels.
    compressed = 0
    Do While x < width
      If row(x) <> mask_color Then
        Exit Do
      End If
      compressed = compressed + 1
      x = x + 1
    Loop
   
    ' Add the number of consecutive compressed pixels to the compressed data.
    compressed_row(compressed_x) = compressed
    compressed_x = compressed_x + 1
     
    ' Hold the position where the number of consecutive uncompressed pixels must be stored.
    uncompressed_start = compressed_x
    compressed_x = compressed_x + 1
   
    ' Get the number of consecutive uncompressed pixels, while adding them to the compressed data.
    uncompressed = 0
    Do While x < width
      If row(x) = mask_color Then
        Exit Do
      End If
      compressed_row(compressed_x) = row(x)  ' *** TODO: Convert the pixel from 24bpp to 16bpp.
      compressed_x = compressed_x + 1
      uncompressed = uncompressed + 1
      x = x + 1
    Loop
   
    ' Store the number of consecutive uncompressed pixels.
    compressed_row(uncompressed_start) = uncompressed
   
    ' End compressed row data depending on SPR version.
    Select Case CNST_VERSION
      Case 0.5
        ' Version 0.5 simply adds last sub-block.
        subblocks = subblocks + 1
       
      Case 10
        ' Version 10 only adds last sub-block if it has uncompressed pixels.
        If (uncompressed = 0) Then
          compressed_x = uncompressed_start - 1
        Else
          subblocks = subblocks + 1
        End If
       
    End Select
  Loop
  ' End of subroutine.
 
 
  ' Show compressed data.
  Debug.Print subblocks
  For x = 0 To compressed_x - 1
    Debug.Print compressed_row(x) & "; ";
  Next x
  Debug.Print
End Sub


[mirex]

Hi, I had no time to comment or look into the file format sooner. I see that you have decoded everything already.

For your information BMP has a possibility to store alpha transparency layer, it can be accomplished by storing 32bit BMP's. You can see these layers in Paint Shop Pro too. I use PSP6/7 and you can see the layer through menu / Mask / Load from alpha channel.

Tonberry: may I use your description and code SPR support into my program Biturn ? You will be credited of course.

rmco2003: I haven't seen the name of the game so far, please can you tell me what game is it ?

[halkun]

taking a stab, it's either FF:Tacticts ot Xoneogears.

However sprites were a PSY-Q thing. So they might be generic.

[rmco2003]

It's just a random Korean MMORPG I'm working on. Mirex if you allow SPR conversion both ways I'll be very happy indeed I think I've reached a point now where my ability isn't good enough to get this done, all I've managed to do is write the easiest things to a new file, and that's using existing code as a base..

Looks like I'm begging again..

[mirex]

rmco2003: you should continue on your program. I don't know how long will it take me untill its finished, I have only a little of free time.

[rmco2003]

I suppose, I'll just have to keep trying to figure out what to do until I get somewhere..

[Tonberry]

Mirex: No problem, use it however you like. No need for credits unless it's custom.

Thanks for the info on BMP. I don't have PSP so I used Paint to test and it doesn't have 32-bit BMPs.


rmco2003:

This is the data you should save when decoding a SPR to be able to rebuild it later:

1) Signature.
2) Number of images.
3) Number of layers.
4) List with the widths.
5) List with the heights.
6) List with the names of the BMPs generated. (Choose the order. I recommend you list the images in the same order as they are stored in the SPR.)

I'll call this file the "SPR definition file" in this example. I'm assuming you won't change the width or height of the images. (If you'll change them, then don't save them and just get them from the BMPs.)

To rebuild a SPR you'll need that file and all the BMPs. Then you can do something like this:

1) Get the signature from the SPR definition file and write it to the new SPR.
2) Get the number of images from the SPR definition file and write it to the new SPR.
3) Write the list of widths to the new SPR.
4) Write the list of heights to the new SPR.
5) Compress all the BMPs in separate files.
5.1) Load one BMP into memory.
5.2) Create a new file with extension ".SPRI" (or other) and the same name as the BMP.
5.3) Write the height of the image to the SPRI file.
5.4) For every row of the BMP:
5.4.1) Compress the row.
5.4.2) Write the number of sub-blocks generated to the SPRI file.
5.4.3) Write all the compressed data to the SPRI file.
5.5) Get the size in bytes of the SPRI generated and save it to an array for later use.
5.6) Close the SPRI file.
5.7) Repeat for the next BMP, starting at 5.1.

At this point we have the header of the SPR, the sizes of the encoded images and the encoded images. That's enough to build the layers.

6) Add all the layers.
6.1) Calculate the pointer to the next layer and write it to the SPR file. To do this, simply add the size in bytes for all the encoded images of the layer and divide the number by 2.
6.2) For every image in the layer, calculate the pointer to the image and write it to the SPR file.
6.2.1) Take 0 as the pointer to the first image of the layer.
6.2.2) For image I, take the pointer to it as the pointer to previous image (I-1) and add the size of the encoded image (I-1) divided by 2.
6.2.3) Repeat for the rest of the images in the layer. (Go to 6.2.2)
6.3) Append all the SPRI files of the layer to the SPR file.
6.4) Repeat for the next layer. (Goto 6.1)
7) If the SPR version is 10 then write 8 bytes with value 0x00.
8 ) Close the SPR file.

Each subsection tells you how to complete a section.


That's a lot of things to do. I recommend you don't try to do everything at once, but write one step and compare the result to the original SPR. (Don't modify the BMPs at first as it will be easier to find the differences.)

Hope it helps.


Edit: Corrected a mistake in 6.2.2

[rmco2003]

I guess this is where my INI module comes in handy

So much easier to record entries in an ini file than in a text log

Code: [Select]

[data]
signature=DCSPRITE10
imagecount=2
[measure]
width0=212
height0=330
width1=212
height1=330
[output]
image0=D:\Documents and Settings\Rhys\Desktop\dr2_sub_001_0000_i0.bmp
image1=D:\Documents and Settings\Rhys\Desktop\dr2_sub_001_0000_i1.bmp

[EDIT] I'm up to the point where I'm adding compressed data to the SPRI files. I can generate the files perfectly and add data to these files, but I'm having some issues with manipulating your code to compress one scanline. One of the problems is that the data is too small after compression to be realistic, think from something like 20k down to 2k, and when viewing the data there's a lot of blank space in between very small chunks of data. I'm just passing through to the files like this:

Code: [Select]

Put 3, , subblocks
For arraypos = 0 To (Width * 3 / 2)
    Put 3, , compressed_row(arraypos)
Next arraypos
I really doubt that code is right now.. maybe it's to do with the array size? I just saw the text "Worst case Width * 3 / 2" so I thought to put that there.

I'm also having some trouble figuring out how to convert the pixels from 24bpp to 16bpp, your functions require quite a few different variables, and I'm assuming I have to convert it all to RGB and then to 16bpp, and visual basic won't allow me to do Function1(Function2 each being converson functions of yours.

Perhaps I'm approaching this from the wrong angle, I'll have to take a deeper look into it tomorrow.

[Tonberry]

It's nice to see you've already come this far.

One of the problems is that the data is too small after compression to be realistic, think from something like 20k down to 2k, and when viewing the data there's a lot of blank space in between very small chunks of data. I'm just passing through to the files like this:

The first layer of the first image of "npc_priz_card.spr" has a compressed size of approx. 9KB. When extracted to a 24bpp BMP the size is 226KB.

9/226 = 3.98%
2/20 = 10%

That means you are getting less compression than you could achieve.

To tell the truth, the compression depends a lot on how many pixels of mask there are in the image. If the image has no mask, you should expect a compression to about 67% (16bpp/24bpp) the size of the BMP. If you have a scanline with 320 pixels (960 bytes) as mask, assuming the width of the image is also 320 pixels, the algorithm for version 10 will encode it with only 2 bytes.

Put 3, , subblocks
For arraypos = 0 To (Width * 3 / 2)
    Put 3, , compressed_row(arraypos)
Next arraypos

Avoid using "Put" when writing to a binary file (in this application); use the functions "Write1", "Write2" and "Write4" instead. You can find them in 'AuxFileIO.BAS'.
In that example you are writing the value of 'subblocks' with a length of 4 bytes because 'subblocks' is a Long, but you should write only 2 bytes to conform to the SPR format. Use 'Call Write2(3, subblocks)' instead.
The same goes for the element of 'compressed_row'; just change it to 'Call Write2(3, compressed_row(arraypos))'.
That will also make the SPRI smaller.

You have to write only the elements of 'compressed_row' that go from 0 to 'compressed_x - 1'. The variable 'compressed_x' tells the number of elements in the compressed row.
The value 'width*3/2' is the maximium lenght a compressed row can have. I didn't really make the calculation so I may be wrong with that, but I think the worst case of compression is when a row has a sequence such as 'color-mask-color-mask-color-mask...'. In that case the algorithm doesn't compress the data, but expands it because it uses 2 values to encode the color pixel and 1 value to encode the mask. That's a total of 3 values for what was stored in only 2.

That bit of code should read something like this:

Code: [Select]

Call Write2(3, subblocks)
For arraypos = 0 To compressed_x - 1
    Call Write2(3, compressed_row(arraypos))
Next arraypos


I'm also having some trouble figuring out how to convert the pixels from 24bpp to 16bpp, your functions require quite a few different variables, and I'm assuming I have to convert it all to RGB and then to 16bpp, and visual basic won't allow me to do Function1(Function2 each being converson functions of yours.

How strange... I thought I'd put a function to do that, but it seems I forgot. Add this to 'Color.BAS': (I didn't test it.)

Code: [Select]

' Convert 24bpp to 16bpp.
Public Function Convert24bppTo16bpp(bpp24 As Long) As Long
  Dim R as Long
  Dim G As Long
  Dim B as Long

  Call Convert24bppToRGB(bpp24, R, G, B)
  Convert24bppTo16bpp = ConvertRGBTo16bpp(R, G, B)
End Function


[rmco2003]

I realised that my problem is that I was loading the BMP for read access, but I wasn't actually reading the rows into the array. I think I need to use a seek command then something like read4? The problem is I don't know where to seek to..

[Tonberry]

According to the Wikipedia link, you can get the starting address of the bitmap from offsets 11-14 (base 1, so in VB do a seek to 11 and use Read4). The address you get will be base 0 so add 1 to it and do another seek to get to that address. (You'll probably end up at offset 55, base 1.)

There you'll have to read, using Read1, the three components for every pixel and convert them from RGB to 24bpp. The first '3*width' bytes have the pixels for the first row (the last one really, as BMP stores rows from bottom to top), then you'll have to check if '3*width' is divisible by 4 and if it's not, then read (and discard) as many bytes as necessary to make it divisible by 4. Look at what I did with the variable 'complete_row' in 'ImageBufferToBMP', but instead of writing, read. After that, process the next row.

[rmco2003]

So if I use

Code: [Select]

  For row = image_buffer.Height - 1 To 0 Step -1
    For column = 0 To image_buffer.Width - 1
      Call Read1(fh, blue)
      Call Read1(fh, green)
      Call Read1(fh, red)
      Call mdlColor.ConvertRGBTo24bpp(image_buffer.Color(column, row), red, green, blue)
    Next column

To read the rows, then it should work?

Also what do I do about this?

Code: [Select]

    For i = 1 To complete_row
      Call Write1(fh, 0)
    Next i
Is this a read too? Or do I do something else instead?

I'm just replicating your function and switching things around to make it follow your instructions.

Some sudo code or something would be extremely helpful for doing all of this seeking, and need to know things like are the images going to be upside down and all of the rest of it, which I'd probably end up doing seeing as I don't know the first thing about working with files in this way.

The code above is all I've been able to do so far, I've also seen you're making a new image buffer when making a BMP so I added that code to the loop I've made to process the compression code for each image so it makes a new buffer for each image.

I'm really confused now.

[Tonberry]

Try something like this:

Code: [Select]

  For row = image_buffer.Height - 1 To 0 Step -1
    For column = 0 To image_buffer.Width - 1
      blue = Read1(fh)
      green = Read1(fh)
      red = Read1(fh)
      image_buffer.Color(column, row) = mdlColor.ConvertRGBTo24bpp(red, green, blue)
    Next column

    For i = 1 To complete_row
      discard = Read1(fh)
    Next i

  Next row

The 'For' that uses 'complete_row' is used to complete the rows with 00s to make it divisible by 4 bytes, as requiered by BMPs.
Don't forget to assign value to 'complete_row' as I did in 'ImageBufferTopBMP'.

I'm just replicating your function and switching things around to make it follow your instructions.

I know it seems hard, but as I said before, it'd be better if you try to understand what the function is suposed to do and write it in your own terms. The code I wrote is simply an example and it may not do the things the way you intend them to, that's why it's important you use it simply as a guideline.

Some sudo code or something would be extremely helpful for doing all of this seeking, and need to know things like are the images going to be upside down and all of the rest of it, which I'd probably end up doing seeing as I don't know the first thing about working with files in this way.

The code above is all I've been able to do so far, I've also seen you're making a new image buffer when making a BMP so I added that code to the loop I've made to process the compression code for each image so it makes a new buffer for each image.

I'm really confused now.

The images won't appear upside down because you started storing the first row of the BMP (the bottom row) in the last row of the image buffer. When saving the compressed rows just do the 'for' from 0 to 'height-1'.


Mmmm... I smell despair. Maybe I'm not that good explaining or you still need a little more practice programming, or both. If it's a real pain to you and you can wait some time, I'll complete it for you.
Please, don't misunderstand me, I have no problem explaining the details to you, but you stated your objective is to translate the images. Maybe you prefer to skip the programming. Let me know what you decide.

[rmco2003]

Well I know for a fact that my knowledge in programming is insufficient but I'll still give it a shot and see if I can get any further. I think dispair would be accurate

[Tonberry]

OK, let's continue then.

About the pseudocode for seeking the start of the bitmap (that I didn't explain it in the previous post):
1) Seek to the position in the header of the BMP where it has a pointer to the start of the bitmap. (Thats offset 11, base 1.)
2) Read 4 bytes using 'Read4'. (Notice the value you read will probably be 54, because according to Wikipedia that's the first offset after the header.)
3) Add 1 to the value you read in (2) to change it from base 0 (specified by BMP format) to base 1 (used by VB6).
4) Do another seek to the offset given by the value you got from step (3) to jump to the start of the bitmap.

That should complete the loading of the BMP. I recommend you test the code that loads the BMP before continuing with the compression as it will make things easier if you find problems later.
To test the loading function, just write the image buffer you just generated to another BMP. You already have a function to write the header of a BMP and another function to write the bitmap from an image buffer and you know they work fine; if there is any error, it will be in the loading.

If the new BMP looks fine, then go on to the compression. If it doesn't, it's best to correct that before adding the compression.

Oh... and test it with BMPs of different sizes.

[rmco2003]

I can't get my head round this now, and I'm starting to get confused by my own code This is getting pretty hopeless now... I've put in your pseudocode but I don't even know what I'm meant to do with that offset I'm getting. I don't mind waiting if you can complete it

[Tonberry]

All that seeking had the purpose to find the offset where the "real" information of the image is stored. That's the offset where you'll find the data (component colors) for the first pixel of the image, and after that, for the next pixel and so on. So, after getting to that offset you have to start reading the pixels with the code...

Code: [Select]

  For row = image_buffer.Height - 1 To 0 Step -1
    For column = 0 To image_buffer.Width - 1
      blue = Read1(fh)
      green = Read1(fh)
      red = Read1(fh)
      image_buffer.Color(column, row) = mdlColor.ConvertRGBTo24bpp(red, green, blue)
    Next column

    For i = 1 To complete_row
      discard = Read1(fh)
    Next i

  Next row

... which I haven't tested, so it may have some mistakes.

After that you'll get the image in the image buffer. Before adding the part that compresses the image to make it SPR compatible, test that the image you loaded into 'image_buffer' is fine. To test it simply write it to a new BMP using the functions you have to write BMPs. Compare the original BMP to the new one.

Once you are sure the image is loading fine, delete or (better) comment the code that tests the BMP is fine and start adding the SPR compression.


--------

As a side note, I'll tell you all that previous seeking is needed because the BMP is composed of a header+image_data. The usual way to read them is to read the header into a memory structure first, get the width, height, bits per pixels, offset where image_data starts, etc. Once all those values are known, it's time to get the image_data (that is, the colors of the pixels).
There's the possibility that an application that writes BMPs adds some info between the usual header and the image_data, making the header bigger, so you can't just jump to the image_data. First you have to find where it's located.

As a bigger side note and completelly offtopic, I'll comment that to better understand all this jumping it's better if you learn some programming language that allows the use of pointers; something like C or Pascal. Learning how to implement common data structures such as lists, double-linked lists, queues, stacks and trees using pointers is helful too. There was a book (probably many) by an author named Aho that had good examples in Pascal.
You can do the same with VB6, VB .Net, C#, Java or some other programming languages that allows for references instead of pointers, but it's not as instructive as using pointers (in my opinion.)
Also, keep in mind that VB6 is a much higher-level language than C and although it makes a lot of generic things easier (GUI, reports, database management, etc.), when you have to implement something too specific it's a real pain and you end up using Windows APIs or linking to code written in C or C++.

[rmco2003]

I don't think it's worth me trying to code this thing anymore, it's just not working out. I'm just getting more confused and I don't know if what I'm doing is even going to help at all. If you have any spare time to work on this for me I'd greatly appreciate your help, otherwise I'm just going to have to give up on making these files.

[Tonberry]

OK, sorry. I'll try to complete it this week, just don't expect a complete application. I think I'll make a function to do the extraction and other to build the SPR back, following the same idea as above; that way you won't have to worry about the internals of the files.

[rmco2003]

That'd be great if you could. I can do things like opening and saving with common dialogs, GUI related things, and the like quite easily.

[Tonberry]

I posted a new version of the code to Mirex's trashbin (http://bin.mypage.sk/FILES/SPRExport2.zip). I made a few changes to some of the functions that already existed and added two new public functions to work with SPRs:

- SPR_Export
  Given the name of a SPR file, the name of a SPR definition file and the color to use for the mask, creates the definition file and a BMP file for each image in the SPR.

- SPR_Build
  Given the name of a SPR definition file, the name of a SPR file and the color to use for the mask, creates the SPR file.

You can find some examples on how to use the new functions in Form1. The concept is similar to what we discussed in previous posts, but I changed it a little bit, as now all the images are extracted in one step (more or less) and the rebuilding process does not use any auxiliary files to store the compressed data.

It's not complete yet, as I've found a little problem. My idea was to create a couple of functions, one the inverse of the other: one to extract the images and the other to rebuild an identical SPR to the original one. The problem is that some SPRs end with a sequence of zeros, but the amount of zeros isn't allways the same. There are some cases where the amount of zeros seem to be related to the number of images and others where they don't seem related.

Despite that problem, the rest of the data is the same, so I'll ask you to get some SPRs, extract the images, rebuild the SPRs and try the ones that have a different size than the original with the game.
If you find any problem, just let me know.

Hope it works.

[rmco2003]

Awesome, finished on my birthday too great timing

[EDIT] The spr creation functions work perfectly, I've already inserted some of my modified images fine and they operate just like the originals.

Thanks for all of your help tonberry, I couldn't of done it without you.

[Tonberry]

Happy birthday!

It's nice to know it's working, thanks.