File system.

[Akari]

To einherjar (because you already write it a lot)

I don't like the way how Files are created. I don't want to specify FileSystem every time I want read file. I think the best way will be create base file class and derive two classes from him. GameFile and RealFile (for example). And all classes will be derived from them. Specifinng filesystem in classes like TimFile are bad, because this file should always be readed from CD or others GameFileSystem, ie this is always ingame file.
Apart from that we have savemap (load saves in future), Xml files and images (bmp).

ps: do you need FileStream template? and for what purpose.
pss: i wang generally finalize Filesystem before cleanup and rewrite other part of common (config for example).

[einherjar]

To einherjar (because you already write it a lot)

I don't like the way how Files are created. I don't want to specify FileSystem every time I want read file. I think the best way will be create base file class and derive two classes from him. GameFile and RealFile (for example). And all classes will be derived from them. Specifinng filesystem in classes like TimFile are bad, because this file should always be readed from CD or others GameFileSystem, ie this is always ingame file.
Apart from that we have savemap (load saves in future), Xml files and images (bmp).

I agree; actually I had the same idea at the very moment I delivered the code

ps: do you need FileStream template? and for what purpose.

I do use it in .MAP file reading. I found awkward/dangerous to write:

Code: [Select]

s16 x = f.GetS16LE(offset); offset += 2;
s16 y = f.GetS16LE(offset); offset += 2;
s16 z = f.GetS16LE(offset); offset += 2;

// ...
u8 v0 = f.GetU8(offset); offset += 1;
u8 v1 = f.GetU8(offset); offset += 1;
u8 v2 = f.GetU8(offset); offset += 1;

So I just hide the offset handling in a wrapper, namely Filestream.  I
made it a templated class so that one could use the same mechanism with
any other file.

I know that you do not like streaming very much, but I think that
originaly, the game is meant to run on a Playstation 1, and the
developpers had to use streaming intensively, for two main reasons:
memory efficiency and CD access performance. I followed this
intuition, and used streaming when the file format was "natively"
"unstreamable". And this might a good way of avoiding loading screen,
don't you think?

pss: i wang generally finalize Filesystem before cleanup and rewrite other part of common (config for example).

Well hmmm I'm not sure to understand you here 

[Akari]

I do use it in .MAP file reading. I found awkward/dangerous to write:

Code: [Select]

s16 x = f.GetS16LE(offset); offset += 2;
s16 y = f.GetS16LE(offset); offset += 2;
s16 z = f.GetS16LE(offset); offset += 2;

// ...
u8 v0 = f.GetU8(offset); offset += 1;
u8 v1 = f.GetU8(offset); offset += 1;
u8 v2 = f.GetU8(offset); offset += 1;


This is still awkward/dangerous and very not undertandable, because when you look at u8 v2 = f.GetU8(); You cant tell where do you reading now and this will cause a lot of errors, expecially when format not understanded yet. And any rows moving are deadly.
I prefer

Code: [Select]

s16 x = f.GetS16LE(offset + 0);
s16 y = f.GetS16LE(offset + 2);
s16 z = f.GetS16LE(offset + 4);

// ...
u8 v0 = f.GetU8(offset + 6);
u8 v1 = f.GetU8(offset + 7);
u8 v2 = f.GetU8(offset + 8);

This way I always know in which part of data structure I reading and this will not kill anything if for some reason I deside to change  Y and Z.

So I just hide the offset handling in a wrapper, namely Filestream.  I
made it a templated class so that one could use the same mechanism with
any other file.

I know that you do not like streaming very much, but I think that
originaly, the game is meant to run on a Playstation 1, and the
developpers had to use streaming intensively, for two main reasons:
memory efficiency and CD access performance. I followed this
intuition, and used streaming when the file format was "natively"
"unstreamable". And this might a good way of avoiding loading screen,
don't you think?

This is not streaming. You still load whole file in memory and only then read it. On psx they have inly 4mb memory for everything. We don't have this restrictions and must get profit from it.

pss: i wang generally finalize Filesystem before cleanup and rewrite other part of common (config for example).

Well hmmm I'm not sure to understand you here 

Almost all part of common classes needs to be rewritten. For example config must be setted seperately for each game, logger must have different type of errors, we need some math library, render class needs to be rewritten since a lot of new obstacles appeared. Input already rewritten to support new type of input event.

[gigaherz]

It's jut my oppinion but I would use packed structs to read the data, and declare all unknown data as unknownX vars, this way you don't have to mess with offsets at all, just do something like:

Code: [Select]

struct some_data_struct {
    int8 field1;
    int16 field2;
};

some_data_struct sd;
f.Read(&sd,sizeof(sd));

or if you prefer dynamic objects:

Code: [Select]

some_data_struct *sd=(some_data_struct *)f.AllocAndRead(sizeof(sd));

this last one is a bit exotic though.
The only problem is the way to tell how to pack changes from compiler to compiler...

[Cyberman]

1) GetS16le <-- I know short is good but I prefer GetInt16le U and S are not descriptive enough.
2) Using offsets are 'nice' but in truth perhaps we should have descriptive offsets for what they are

Code: [Select]

s16 x = f.GetS16LE(offset + 0);
s16 y = f.GetS16LE(offset + 2);
s16 z = f.GetS16LE(offset + 4);

// ...
u8 v0 = f.GetU8(offset + 6);
u8 v1 = f.GetU8(offset + 7);
u8 v2 = f.GetU8(offset + 8);


Perhaps instead?

Code: [Select]

int16 x = f.GetInt16le(Position + l_XOff);
int16 y = f.GetInt16le(Position + l_YOff);
int16 z = f.GetInt16le(Position + l_ZOff);
uint8 v0 = f.GetUint8(Position + l_V0Off);
uint8 v1 = f.GetUiint8(Position + l_VOff1);
uint8 v2 = f.GetUint8(Position + l_V2Off);

It is more verbose and l_ is whatever 'structure' it's from.

3) I believe we are going to have compilor setting issues if we rely on data structures.  I know borland's compilors you must be sure to set your data alignment for the module you are going to use the structure in.

If we go more OOP in our data types we are creating then we can fille a structure by assigning the file to it or using an assign function or something like that.
Depending on how abstract we want our data loading to be.

I suppose we can have 'smart little objects' in our system and tell the object to read 'index' or item from the file and position into itself?
Instead of

Code: [Select]

           s16 dest_x =  GetU16LE(s2 + 0x00);
           s16 dest_y =  GetU16LE(s2 + 0x02);
           u8  src_x  =  GetU16LE(s2 + 0x04);
           u8  src_y  =  GetU16LE(s2 + 0x05);
           u8  clut   = (GetU16LE(s2 + 0x06) >> 6) & 0xF;
           backgroundManager->AddSpriteLayer1(dest_x, dest_y, src_x, src_y, clut, page_x, page_y);
           s2 += 0x08;

Maybe?

Code: [Select]

            backgroundManager->AddSpriteLayer1(s2);
            s2 += l_sprite_block_size;
It might have less impact on some things.

There is one issue that some keeps being overlooked. Although the PC might be a brute. Not every platform will have that much in the form of resources. Especially anything like the Palm or a WinCE machine.  The Palm has 8M of Ram just not all of it is available for one to use. It seems 'data' like notes etc. are kept in RAM cough. In any case the idea was not to design around a platorm but to make one.  Not all implementations will have a PC environment to play in.

Cyb

[Akari]

It's jut my oppinion but I would use packed structs to read the data, and declare all unknown data as unknownX vars, this way you don't have to mess with offsets at all, just do something like:

Code: [Select]

struct some_data_struct {
    int8 field1;
    int16 field2;
};

some_data_struct sd;
f.Read(&sd,sizeof(sd));

or if you prefer dynamic objects:

Code: [Select]

some_data_struct *sd=(some_data_struct *)f.AllocAndRead(sizeof(sd));

this last one is a bit exotic though.
The only problem is the way to tell how to pack changes from compiler to compiler...

You have just one trouble with this. The data will be reading as is. And you still needs swap bytes to get normal value.

[Akari]

1) GetS16le <-- I know short is good but I prefer GetInt16le U and S are not descriptive enough.
2) Using offsets are 'nice' but in truth perhaps we should have descriptive offsets for what they are

Code: [Select]

s16 x = f.GetS16LE(offset + 0);
s16 y = f.GetS16LE(offset + 2);
s16 z = f.GetS16LE(offset + 4);

// ...
u8 v0 = f.GetU8(offset + 6);
u8 v1 = f.GetU8(offset + 7);
u8 v2 = f.GetU8(offset + 8);


Perhaps instead?

Code: [Select]

int16 x = f.GetInt16le(Position + l_XOff);
int16 y = f.GetInt16le(Position + l_YOff);
int16 z = f.GetInt16le(Position + l_ZOff);
uint8 v0 = f.GetUint8(Position + l_V0Off);
uint8 v1 = f.GetUiint8(Position + l_VOff1);
uint8 v2 = f.GetUint8(Position + l_V2Off);

It is more verbose and l_ is whatever 'structure' it's from.

l_VOff1 and it's brothers will be used only in one place, so I think it's not needed to create variable just to create one.
About names of function - I must agree. U letter don't have any meaning. GetInt8 GetInt16LE GetInt32LE are much better.

3) I believe we are going to have compilor setting issues if we rely on data structures.  I know borland's compilors you must be sure to set your data alignment for the module you are going to use the structure in.

If we go more OOP in our data types we are creating then we can fille a structure by assigning the file to it or using an assign function or something like that.
Depending on how abstract we want our data loading to be.

I suppose we can have 'smart little objects' in our system and tell the object to read 'index' or item from the file and position into itself?
Instead of

Code: [Select]

           s16 dest_x =  GetU16LE(s2 + 0x00);
           s16 dest_y =  GetU16LE(s2 + 0x02);
           u8  src_x  =  GetU16LE(s2 + 0x04);
           u8  src_y  =  GetU16LE(s2 + 0x05);
           u8  clut   = (GetU16LE(s2 + 0x06) >> 6) & 0xF;
           backgroundManager->AddSpriteLayer1(dest_x, dest_y, src_x, src_y, clut, page_x, page_y);
           s2 += 0x08;

Maybe?

Code: [Select]

            backgroundManager->AddSpriteLayer1(s2);
            s2 += l_sprite_block_size;
It might have less impact on some things.

I try to design code to be less recource dependable. The background manager don't know anything about structure of file, it just work with data that provide to it by some reader (DatFile, XmlFile). Else we will need to implement loading method to Manager for each file type we want it to support... its very bad.

There is one issue that some keeps being overlooked. Although the PC might be a brute. Not every platform will have that much in the form of resources. Especially anything like the Palm or a WinCE machine.  The Palm has 8M of Ram just not all of it is available for one to use. It seems 'data' like notes etc. are kept in RAM cough. In any case the idea was not to design around a platorm but to make one.  Not all implementations will have a PC environment to play in.

The best way to make ffvii work are create PSX emulator or use some part of it to emulate some part of engine. Now q-gears needs at least 50mb of memory to load textures for background.
We simply can't support all platform on earth. There tooooo little of us. And there are tooooo little people need this. I don't want spend 50% of work for 1% of people. The morden PDA have >80mb RAM so this is not a problem. The best way in concentrate on Windows and Linux and create all functionality. After all things will be done there can be any thoughts on extentions and other ports.

[halkun]

Sooner or later texture management is going to become an issue.

In battle for example, the PSX takes advantage that both selecting an action and the spell/summon casting animation allows for the kernel to stream textures and animations off the disk at the same time. There is no way to know what spells will be cast during a battle and the idea of loading the entire /magic directory into memory is silly.

However, I'm all for loading a whole field file into memory uncompressed. If some programmer wants to make the system run on embedded systems, the code is availible.

[Akari]

Sooner or later texture management is going to become an issue.

In battle for example, the PSX takes advantage that both selecting an action and the spell/summon casting animation allows for the kernel to stream textures and animations off the disk at the same time. There is no way to know what spells will be cast during a battle and the idea of loading the entire /magic directory into memory is silly.

Dinamic loading and unloading textures are available. We just don't have Texture Manager or something that do this automaticly. Maybe we do this during rewriting common classes. The main problem are paletted textures, but I think to handle things during rewriting rendering class.

[einherjar]

You cant tell where do you reading now and this will cause a lot of errors, expecially when format not understanded yet.

So you prefer:

// for each record:
s16 x = f.GetS16LE(offset + 0);
s16 y = f.GetS16LE(offset + 2);
s16 z = f.GetS16LE(offset + 4);

// Goto next record
offset += /* how much? let me count... 4? oh no 8. Oh no wait we read a 2 bytes variable; OK: 6*/ 6;

to:

Code: [Select]

filestream s(f);
// for each record:
s16 x = s.GetS16LE();
s16 y = s.GetS16LE();
s16 z = s.GetS16LE();

Hmmmpf. OK, I will change it ASAP. I will remove Filestream as well.

You still load whole file in memory and only then read it.

Actually, yes, because when "opening" an lzs data block it is effectively
directly uncompressed into memory. But when I looked at how it is decoded,
I had the strong feeling that it was meant to be read as a stream as well.

On psx they have inly 4mb memory for everything. We don't have this restrictions and must get profit from it.

(And then we will need 1Gb RAM, the last P4 and a GeForce8800 to run q-gears. OK OK I am exagerating  -- just kidding )

Almost all part of common classes needs to be rewritten. For example config must be setted seperately for each game, logger must have different type of errors, we need some math library, render class needs to be rewritten since a lot of new obstacles appeared. Input already rewritten to support new type of input event.

Ah OK

[Akari]

So you prefer:

// for each record:
s16 x = f.GetS16LE(offset + 0);
s16 y = f.GetS16LE(offset + 2);
s16 z = f.GetS16LE(offset + 4);

// Goto next record
offset += /* how much? let me count... 4? oh no 8. Oh no wait we read a 2 bytes variable; OK: 6*/ 6;

to:

Code: [Select]

filestream s(f);
// for each record:
s16 x = s.GetS16LE();
s16 y = s.GetS16LE();
s16 z = s.GetS16LE();

Hmmmpf. OK, I will change it ASAP. I will remove Filestream as well.

It the same as

Code: [Select]

filestream s(f);
// for each record:
s16 x = s.GetS16LE();
s16 y = s.GetS16LE();
s16 z = s.GetS16LE();

// Goto next record
/* how much else I need to read? let me think there aligment bytes so i skeep 2? oh no 4. OK: 6 since there are short value whose value we still dont know.*/
s.GetS32LE();
s.GetS16LE();

It's not any better.
The best way is work with structures, but they not handle LE problem, so I think my way is better for understanding. Because if you know that size of each block 12 bytes then you add offset by 12 bytes regardless of what you read earlier. And you can have few offsets for paralel reading (look at the walkmesh - I need to read triangle info along with the access info).