Ye Olde PSX .DAT files redeux (4SBWY_1.DAT)

[Cyberman]

I've been looking at these of late (like within the last 2 weeks to be precise)
I modified my program to dump these little images of the structure.

We currently know a few things about these.
1 The first 7 Double Words are pointers to the playstations memory
2 The first section of the 7 sections of this file is the script section.

I'm not sure how to deliniate the sections properly myself.  So as a GUESS I took the pointers and assumed they were a certain amount apart. IE the data is decoded and dumped into memory directly.  I suspect this is not entirely true.

<removed cute cudely images of DATA!>

The sections aren't properly aligned with the pointer information AND all the pointer informat is DWORD aligned. The data is not aligned acording to that it appears. So the only thing I can think of is it's decompressed and a header is parsed to determine the locations the locations the real data gets stuffed into memory.  Since we know what the file is and where it's supposed to be in memory a little debugging I suspect will find the data using a PS1 emulator that should be aligned at these locations.  That's my new plan at least

I suspect that section 1 may have some other information in it to find the beginings and ends of these sections. IE Section 1 does not start immediately so back to the drawing board.

For reference I am using \FIELD\4SBWY_1.DAT file to work on this.

Any thoughts on this anyone?

Cyb

[Micky]

Here is my experimental field viewer. It's a bit sloppy, but then again it is not finished.

Code: [Select]

// System
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>

// STL
#include <string>
#include <map>

// C
#include <cassert>
#include <errno.h>

#if 1
// GLUT
#include <GLUT/glut.h>

// OpenGL
#include <OpenGL/OpenGL.h>
#include <OpenGL/gl.h>
#include <OpenGL/glu.h>
#else
// GLUT
#include <glut/glut.h>

// OpenGL
#include <gl/gl.h>
#include <gl/glu.h>
#endif

using namespace std;

typedef unsigned char u8;
typedef unsigned short int u16;
typedef unsigned long int u32;

typedef signed char s8;
typedef signed short int s16;
typedef signed long int s32;

// - utility functions --------------------------------------------------------

u16 get_u16le(void const * const buf)
{
return ((u8*)buf)[0] | (((u8*)buf)[1]<<8);
}

u32 get_u32le(void const * const buf)
{
return ((u8*)buf)[0] | (((u8*)buf)[1]<<8) | (((u8*)buf)[2]<<16) | (((u8*)buf)[3]<<24);
}

#ifndef O_BINARY
#define O_BINARY 0
#endif

int load( const string & name, void ** data )
{
        size_t size = 0;
        *data = NULL;
        int fd = open( name.c_str(), O_RDONLY|O_BINARY, 0 );
        if (fd >= 0) {
                struct stat sb;
                if ( fstat( fd, &sb ) >= 0) {
                        assert( sb.st_size > 0 );
                        void *tmp = malloc( sb.st_size );
                        if ( tmp!=NULL ) {
                                if (read( fd, tmp, sb.st_size ) == sb.st_size) {
                                        *data = tmp;
                                        size = sb.st_size;
                                } else {
                                        free(tmp);
                                }
                        }
                }
                close( fd );
        } else {
printf("%i\n", errno);
}
        return size;
}

void save(const std::string & name, const void *data, int size)
{
        int fd = open( name.c_str(), O_WRONLY|O_CREAT|O_TRUNC, 0666 );
        if (fd >= 0) {
write( fd, data, size );
                close( fd );
        }
}

int load_lzs(const string & name, void ** data )
{
// read compressed data
u8 *buf;
int tsize = load( name, (void**)&buf );
if (buf == NULL) {
*data = NULL;
return 0;
}

int isize = get_u32le(buf) + 4;

if (isize != tsize) {
free(buf);
*data = NULL;
return 0;
}

// decompress;
int osize = (isize + 255) & ~255;
u8 * obuf = (u8 *)malloc(osize);
int iofs = 4, oofs = 0;
u8 cmd=0, bit=0;
while (iofs < isize) {
if (bit == 0) {
cmd = buf[iofs++];
bit = 8;
}
if (cmd&1) {
obuf[oofs++]=buf[iofs++];
if (oofs==osize) {
osize+=256;
obuf = (u8*)realloc(obuf, osize);
}
} else {
u8 a = buf[iofs++];
u8 b = buf[iofs++];
u16 o = a | ((b&0xF0)<<4);
u8 len = (b&0xF)+3;

int rofs =  oofs - ((oofs - 18 - o) & 0xFFF);
for (int j=0; j<len; j++) {
if (rofs < 0) {
obuf[oofs++]=0;
} else {
obuf[oofs++]=obuf[rofs];
}
if (oofs==osize) {
osize+=256;
obuf = (u8*)realloc(obuf, osize);
}
rofs++;
}
}
cmd>>=1;
bit--;
}
free( buf );

*data = obuf;
return oofs;
}

// - field class --------------------------------------------------------------

class CFieldBackground {
private:
struct TexInfo {
int page, pal;
bool hc;
bool operator<(const TexInfo & rhs)
{
return memcmp(this, &rhs, sizeof(TexInfo)) < 0;
}
};
map<TexInfo, int> texInfoMap;

u8 * mimData;
int mimSize;

u8 * datData;
int datSize;

u8 * bsxData;
int bsxSize;

struct Texture {
u8 * rgba; // data
int id; // OpenGL texture ID
};

struct Tile {
int u, v, pal; // info from the request, pal=-1 for RGBA
int s, t; // texture position in OpenGL texture
int texture; // Texture
};

int getTileI8(int u, int v, int pal);
int getTileRGBA(int u, int v);

int getTexture(u8 atr1, u8 atr2, u8 x);
public:
CFieldBackground();
~CFieldBackground();
void load(const std::string & name);
void draw();
};

CFieldBackground::CFieldBackground():
mimData(NULL), mimSize(0),
datData(NULL), datSize(0),
bsxData(NULL), bsxSize(0)
{
}

CFieldBackground::~CFieldBackground()
{
}

void
CFieldBackground::load(const std::string & name)
{
// read compressed data
string baseName = name.substr(0, name.find('.'));

// MIM
string mimName = baseName + ".MIM";
mimData = NULL;
mimSize = load_lzs( mimName, (void **)&mimData );

// DAT
string datName = baseName + ".DAT";
datData = NULL;
datSize = load_lzs( datName, (void **)&datData );
save(datName + ".bin", datData, datSize);

// BSX
string bsxName = baseName + ".BSX";
bsxData = NULL;
bsxSize = load_lzs( bsxName, (void **)&bsxData );
save(bsxName + ".bin", bsxData, bsxSize);
}

int
CFieldBackground::getTexture(u8 atr1, u8 atr2, u8 x)
{
TexInfo texInfo;
texInfo.page = (atr1 >> 4); // * 128
int px = texInfo.page * 64 + x;
texInfo.pal = px < 512 ? px / 256 : ((px - 512) / 128) + 2;
texInfo.hc = false; // atr2 & 1;

if (texInfoMap.find(texInfo) != texInfoMap.end())
{
return texInfoMap[texInfo];
}
printf("%2.2x %2.2x\n", atr1, atr2);

int ofs = get_u32le(mimData);
int step = get_u32le(mimData+ofs);
int npal = get_u16le(mimData+10);
int pal_ofs = 12;
int pic_ofs = ofs+12;
int xsize = get_u16le(mimData+ofs+8)*2;
int ysize = get_u16le(mimData+ofs+10);

u8 *texture = new u8[256*256*4];

if (texInfo.hc) {
int page = texInfo.page * 128;

for (int y=0; y<256; y++) {
for (int x=0; x<256; x++) {
u16 col = get_u16le(mimData + (y*xsize+x*2+page) + pic_ofs);

int r = (((col      ) & 31) * 255 + 15) / 31;
int g = (((col >>  5) & 31) * 255 + 15) / 31;
int b = (((col >> 10) & 31) * 255 + 15) / 31;
int a = (col & 0x8000) || (col == 0) ? 0 : 255;

int addr = (y*256+x) * 4;
texture[addr+0] = r;
texture[addr+1] = g;
texture[addr+2] = b;
texture[addr+3] = a;
}
}
} else {
int page = texInfo.page * 64;
int pal = texInfo.pal * 512;

for (int y=0; y<256; y++) {
for (int x=0; x<256; x++) {
u8 idx = mimData[(y*xsize+x+page) + pic_ofs];
u16 col = get_u16le(mimData + idx*2 + pal_ofs + pal);

int r = (((col      ) & 31) * 255 + 15) / 31;
int g = (((col >>  5) & 31) * 255 + 15) / 31;
int b = (((col >> 10) & 31) * 255 + 15) / 31;
int a = (col & 0x8000) || (col == 0) ? 0 : 255;

int addr = (y*256+x) * 4;
texture[addr+0] = r;
texture[addr+1] = g;
texture[addr+2] = b;
texture[addr+3] = a;
}
}
}

int textureId;
glGenTextures(1, &textureId);
glBindTexture( GL_TEXTURE_2D, textureId);
#if 0
glTexParameteri( GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST );
glTexParameteri( GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST );
#else
glTexParameteri( GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR );
glTexParameteri( GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR );
#endif
glTexImage2D( GL_TEXTURE_2D, 0, GL_RGBA, 256, 256, 0, GL_RGBA, GL_UNSIGNED_INT_8_8_8_8, texture);

delete texture;

texInfoMap[texInfo] = textureId;
return textureId;
}

void
CFieldBackground::draw()
{
u32 memaddr[7];
for (int i=0; i<7; i++) {
memaddr[i] = get_u32le(datData+i*4);
}
u32 base = memaddr[0] - 7 * 4;
u32 section[7];
for (int i=0; i<7; i++) {
section[i] = memaddr[i] - base;
}

u8 * tilemap = datData + section[2] + get_u32le(datData + section[2]);
int count = (get_u32le(datData + section[2] + 4) - get_u32le(datData + section[2])) / 8;
#if 1
printf("---- %4.4x\n", count);
for (int i=0; i<get_u32le(datData + section[2]); i++) {
printf("%2.2x ", (u8*)(datData + section[2])[i]);
}
printf("\n----\n");
for (int i=0; i<get_u32le(datData + section[2] + 8)-get_u32le(datData + section[2] + 4); i++) {
printf("%2.2x ", (u8*)(datData + section[2] + get_u32le(datData + section[2] + 4))[i]);
}
printf("\n----\n");
#endif
for (int i=0; i<count; i++) {
s16 tx = (s16)get_u16le(tilemap + 8 * i + 0) + 256;
s16 ty = (s16)get_u16le(tilemap + 8 * i + 2) + 256;
u8 sx = tilemap[8 * i + 4];
u8 sy = tilemap[8 * i + 5];
u8 atr1 = tilemap[8 * i + 6];
u8 atr2 = tilemap[8 * i + 7];
// printf("%2.2x %2.2x|", atr1, atr2);

glBindTexture(GL_TEXTURE_2D, getTexture(atr1, atr2, sx));

glBegin(GL_QUADS);
glTexCoord2i(sx, sy); glVertex2i(tx, ty);
glTexCoord2i(sx+15, sy); glVertex2i(tx+16, ty);
glTexCoord2i(sx+15, sy+15); glVertex2i(tx+16, ty+16);
glTexCoord2i(sx, sy+15); glVertex2i(tx, ty+16);
glEnd();
}
//printf("\n");
#if 0
tilemap = datData + section[2] + get_u32le(datData + section[2] + 4);
count = get_u16le(tilemap);
for (int i=0; i<count; i++) {
s16 tx = (s16)get_u16le(tilemap + 14 * i + 0 + 2) + 512;
s16 ty = (s16)get_u16le(tilemap + 14 * i + 2 + 2) + 256;
u8 sx = tilemap[14 * i + 4 + 2];
u8 sy = tilemap[14 * i + 5 + 2];
u8 atr1 = tilemap[14 * i + 6 + 2];
u8 atr2 = tilemap[14 * i + 7 + 2];
// printf("%2.2x %2.2x\n", atr1, atr2);

glBindTexture(GL_TEXTURE_2D, getTexture(atr1, atr2, sx));

glBegin(GL_QUADS);
glTexCoord2i(sx, sy); glVertex2i(tx, ty);
glTexCoord2i(sx+15, sy); glVertex2i(tx+16, ty);
glTexCoord2i(sx+15, sy+15); glVertex2i(tx+16, ty+16);
glTexCoord2i(sx, sy+15); glVertex2i(tx, ty+16);
glEnd();
}
#endif
}

CFieldBackground g_FieldBackground;

// - OpenGL stuff --------------------------------------------------------------

void ReshapeWindowFunc(int width, int height)
{
glViewport(0, 0, width, height);
glMatrixMode(GL_PROJECTION);
glLoadIdentity();
glOrtho(0.0, width, height, 0.0, 0.0, 1.0);
}

void MainRenderLoop(void)
{
// render
glClearColor(0, 0, 0, 1.00);
glClear( GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
g_FieldBackground.draw();

glFlush();
}

void SpecialHandler(int key, int x, int y)
{
#if 0
switch (key) {
case GLUT_KEY_UP:
rot_x -= 4;
if (rot_x < -64) {
rot_x = -64;
}
break;
case GLUT_KEY_DOWN:
rot_x += 4;
if (rot_x > 64) {
rot_x = 64;
}
break;
case GLUT_KEY_LEFT:
rot_y -= 5;
break;
case GLUT_KEY_RIGHT:
rot_y += 5;
break;
}
glutPostRedisplay();
#endif
}

void KeyboardHandler(unsigned char key, int x, int y)
{
switch (key) {
case 27:
exit(0);
break;
}
}

int main(int argc, char *argv[])
{
if (argc != 2) {
fprintf(stderr, "fieldview <field.dat>\n");
} else {
// get current directory
char cwd[1024];
getcwd(cwd, sizeof(cwd));

// init opengl and glut
glutInit(&argc, (char **) argv);
glutInitWindowSize(640, 480);
glutInitDisplayMode(GLUT_SINGLE | GLUT_RGBA | GLUT_DEPTH);
glutCreateWindow(argv[0]);
glutDisplayFunc(MainRenderLoop);
glutIdleFunc(MainRenderLoop);
glutReshapeFunc(ReshapeWindowFunc);
glutKeyboardFunc(KeyboardHandler);
glutSpecialFunc(SpecialHandler);

// Setup OpenGL
glDisable(GL_LIGHTING);
glDisable(GL_DEPTH_TEST);
glDisable(GL_CULL_FACE);
glEnable(GL_TEXTURE_2D);
glEnable(GL_BLEND);
glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);
glEnable(GL_ALPHA_TEST);
glAlphaFunc(GL_GREATER, 0.0f /*1.0f/255.0f*/);

glDisable(GL_LINE_SMOOTH);

glShadeModel(GL_SMOOTH);

glMatrixMode(GL_MODELVIEW);
glLoadIdentity();

glMatrixMode(GL_TEXTURE);
glLoadIdentity();
glScalef(1.0f/256.0f, 1.0f/256.0f, 1);
glTranslatef(0.5f, 0.5f, 0.0f);

// Setup projection matrices
glViewport(0, 0, 640, 480);

glMatrixMode(GL_PROJECTION);
glLoadIdentity();
glOrtho(0.0, 640, 480, 0.0, 0.0, 1.0);

// if the filename is not absolute try to add the current work directory.
string path;
if (argv[1][0] != 0 && argv[1][0] != '/' && argv[1][1] != 0 && argv[1][1] != ':' && argv[1][2] != 0 && argv[1][2] != '\\') {
path = string(cwd) + "/";
}

// load the background data
g_FieldBackground.load(path + argv[1]);

// main (this will never return)
glutMainLoop();
}
return 0;
}


[sfx1999]

Hey Micky have you tried OpenGLUT? You can link it with your programs. It is under the MIT license.

[Cyberman]

Hmmm it seems 28 bytes was my problem go figure. Alignment is now somewhat sane, although computing the Width and Height has yet to be completed (since I keep having to reboot). I think I need to reinstall Borland C++ :/

Cyb

[Cyberman]

Thanks Micky your information got me this far.
4SBWY_1
http://www.geocities.com/cyberman_phillips/TimView/4SBWY_1.jpg
ANFRST_1
http://www.geocities.com/cyberman_phillips/TimView/ANFRST_1.jpg

Any interesting note there appear to be a few sections to this data likely sprite stuff et al. Micky's data of course is incomplete. Thanks to him I can at least view this much as well as find the dimensions of the Thing.
What I do is iterate through the tiles given to get Xmax and Xmin and Ymax and Ymin.
Then I size a bitmap and create a second one. The second bitmap contains all the tiles which are rendered as needed. When done I just copy a 16x16 block from the rendered tile palette to the drawing and this gets you the image.  White areas have no tiles associated with them in this view.  As you can see no sprite data is shown either.

The interesting thing is the values for the tiles aren't 0,0 as the upper left corner instead 0,0 is somewhere in the screen. My guess is that is the point is the origin of the screen in relative coordinates. Likely it relates to the walkmesh data after perspective transformation in the GTE.
For "fun" I will have to change this to an OpenGL view and add the walk mesh drawn on top of it. I believe the walk mesh data to be identical to that of the PC's so I will use what's available here for that (anyone for wire mesh view overlay?) I suppose I could just draw it overtop instead of using OpenGL (might make it easier since my OpenGL code still isn't working right).

Cyb

[sfx1999]

For the first one, try converting pure blue to no alpha.

[Micky]

Hey Micky have you tried OpenGLUT? You can link it with your programs. It is under the MIT license.

I selected GLUT mostly because it is already available on a lot of systems (Mac + Linux) or can be easily added (windows).
I don't think I need any of the features of OpenGLUT at the moment.

For the first one, try converting pure blue to no alpha.

I think the green/blue areas are overwritten by higher layers.

I suppose I could just draw it overtop instead of using OpenGL (might make it easier since my OpenGL code still isn't working right).

That should be quite easy with OpenGL. Just lock the Z-Buffer while drawing the background, then enable it for the mesh.
Did you find the correct algorithm for selecting palettes?

[Cyberman]

I wish I could say "SURE" save my windows computer DIED horribly and I'm backing data up from it as I type this. It means a install of win2K and all those fun security updates then reinstall all that software. This could take a little while I gave up doing a normal install over the stupid thing tell it works. So now I'm working on a wipe clean the slate of ye olde boot partition and install. SIGH.. I've got so much to back up.. thank goodness I have a whole disk to dump stuff into. (it had 30gigs cleared ... had) still busily backing up. Network is toast unless someone knows how to get the network up in 'DOS' setup mode.

So... hopefully I can back up everything and I won't have to start from scratch

Cyb

[Cyberman]

I've rebuilt enough of the application I've returned to this point.  (Weee)
Unfortunately it will be a week before the backgrounds are as good as they were LOL. Such is life I guess. In any case there appears to be lots of other data in that section you (Micky) have been using to extra data. Might be sprite data you think?
The information on the display appears to be segmented in groups of 64 horizontal pixels with an associtated palette. This comes out to be 128x256 sections.
The playstation has a display area of 1024 x 512.  The palette for the background appears to be at 0, 480 (leaving roughly 32 palettes possble), obviously the display for the background starts at 0,0.  At 384 (pixels) the actual background tile info starts. This gives a max of 1024-384= 640 pixels. OR 10 128x256 tile sets. with possibly 20 palettes. (hmmm).

I digress. I've made progress obviously (grin). however the offset computation to the tile set seems to not be quite right (wee). I guess it needs work.  I think how it works is something like this you have 640x256 physical pixels Or 1280x256 8 bit pixels in the tile set max. I believe they organized it in 128 8 bit pixel tile sets. However the pitch of the MIM data comes into play here. The palette is choosen depending if the 'page' exceeds the end pitch I think.  That's why if it's > 512 it wraps. However you can have larger tile sets than that so.. I think the palette count relates to the number of 128x128 sections that are used but starts top left to right and increases then (Y wise as you step through the palettes) by the horizontal pitch /64 to the next row. That's my thinking at the moment.

Cyb

[Cyberman]

First Micky you should be using a UINT16 for the attribute information instead of 2 bytes.
Second the page appears to use bits  6-10 or 4-10 this makes for a 5 or 7 bit field they are using for the page # or whatever it might be.  It also appears that using the scheme you have picks out every OTHER section of data. IE it alternates blocks.  Palette finding is not working worth beans with the current scheme. You can only find the first palette so I need to twiddle around more and might resort to some scheme to allow me to tweak the palette used for each section of tiles, so they look like something.
I suspect the alternate block sections between are sprite data or better put non background data. In any case I'm making 'steady' progress in getting the background data viewable, the palettes appear to be the tricky part.

[Cyberman]

Ok progress has moved along some
Here is ANFRST_1 again however I've included a tile image.
This pretty much looks identical as before, however I hand tweaked the palettes to look at the tiles.

Notes:
Some tiles are wrong, this means the method used for selecting tiles is .. wrong as well Edited <-- this has been fixed and the image reflects that. I used my Tile tracing code to find out where it was grabing the tiles from. Such is life!
This likely means I'll have to do some sort of forced tweaking of what tile sets are used with tile collection bits in the attribute word. That might be complicated but if you can rearrange things so they are correct then, you'll learn how it's supposed to work is my theory (it's how I am learning how to select the correct palettes at least).
http://www.geocities.com/cyberman_phillips/TimView/ANFRST_1_Image.jpg
Palettes:
Assume that the tiles are actually in 128x128 sections and in this case are arranged in an array of 10x2  (10 on top 10 on bottom).  Assume the top row is 0 -9 and the bottom row is 10-19 (tile set number). The palette ordering is (in tile set:palette order), 0:0. 1:0, 2:1, 3:1, 4:2, 5:3, 6:4, 7:5, 8:6, 9:7, 10:0, 11:0, 12:1, 13:1, 14:2, 15:8, 16:4, 17:9, 18:0, 19:7.  It's interesting to note there are 14 palettes and yet 4 aren't used. The tiles use palettes 0-9 what are 10 - 13 used for? I guess I'll have to check if there are 2 images in file.
http://www.geocities.com/cyberman_phillips/TimView/ANFRST_1_Tiles.jpg

Other notes:
The attribute word in the tile mapping structure does a few things however I've yet to know what it's all ment for however it appears bits 10-6 are the only ones used for tiles, palette information might be somewhere else (maybe).

Cyb

[Alhexx]

Sorry about that, Cyberman, but I had to replace your img-tags with url-tags, because I had to scroll horizontally at every single line I wanted to read (although I'm using 1024x768 screen resolution)

 - Alhexx

[Cyberman]

Hmmm well I suppose 1180 and 1280 size bitmaps might be an issue

LOL.

Cyb