Author Topic: Anyone want to write a PSX file extractor?  (Read 34066 times)

halkun

  • Global moderator
  • *
  • Posts: 2097
  • NicoNico :)
    • View Profile
    • Q-Gears Homepage
Anyone want to write a PSX file extractor?
« on: 2007-06-01 14:41:31 »
Ok, here's the idea.

FF7 for the PSX requires you to read data off the disk. The problems is that a PSX disk is not a standard data CD-ROM, but in fact is a "multi-mode" disk. In order to fit more data, some of the files on the disk use a "mode 2" format. In mode 2, the error-correction data is removed and real accessible data is put in it's place. The problem is most CD-ROMs assume that there is error-correction data throughout the disk, and when it lands on a "mode 2" sector, can't read the data because the error-correction has been "corrupted"

As for right now in my q-gears insatller, I'm using a simple XCOPY command to transfer the files from disk 1. If anyone wants to writ a real mode-2 copyier for windows, I'll include it. Here's some specs for you to follow...

1. The command must take arguments of   "program [sourcedir] [destdir]" so it's nice and simple. It should just do a tree copy of the FF7 disk.
2. It needs a GUI progress bar.
3. It need to be able to prompt for the next disk, and then only copy over the files that are different than the others.

ANyone up to it. I can answer questions if anyone has them.

gigaherz

  • *
  • Posts: 105
    • View Profile
    • gigaherz's shitty stuff
Re: Anyone want to write a PSX file extractor?
« Reply #1 on: 2007-06-01 17:24:42 »
The problem is most CD-ROMs assume that there is error-correction data throughout the disk, and when it lands on a "mode 2" sector, can't read the data because the error-correction has been "corrupted"
If you set the drive to read raw sectors, then you tell it you don't want correction, so you have to do the checking yourself, if you want to make sure the sectors are right.

Right now I'm a bit busy, but I'll look at it later. I think I can do one, I'll just have to take a look at the ioctl api to see how can I read raw 2352-byte sectors from the disc, once I know that it should be easy :P

EDIT: Status update:

I managed to have working raw reading code using windows ioctl calls, and I can detect if a specific sector is form1 or form2, so right now I can read anything, once I have the LBAs.

Next step is to add ISO9660 parsing code, so I can find the files.

EDIT2: Status update:

Listing works. I can see (printed the filenames to the stdout) the list of all the directories and files in the first ff7 cd (recursively).

Next step: sleep.

Next step(2): Extracting (copying) files (if the destination file doesn't exist).

EDIT3: Status update:

Extraction seems to work, but I have no way to play the videos so I can't test them.

Right now the program is a console app, so the next step would be adding a small window with a progressbar, once I know extraction works.

Beta build: http://www.uploading.com/en/files/FU8C5L0Y/CdExtractor.zip.html

If anyone wants to test it, and tell me if the resulting files work.

« Last Edit: 2007-06-02 14:20:22 by gigaherz »

gigaherz

  • *
  • Posts: 105
    • View Profile
    • gigaherz's shitty stuff
Re: Anyone want to write a PSX file extractor?
« Reply #2 on: 2007-06-02 14:21:56 »
I'm not sure if this forum marks modified messages as "new", but in any case, bumping it up.

halkun

  • Global moderator
  • *
  • Posts: 2097
  • NicoNico :)
    • View Profile
    • Q-Gears Homepage
Re: Anyone want to write a PSX file extractor?
« Reply #3 on: 2007-06-02 16:11:18 »
I'll take a look at it after work today...


gigaherz

  • *
  • Posts: 105
    • View Profile
    • gigaherz's shitty stuff
Re: Anyone want to write a PSX file extractor?
« Reply #4 on: 2007-06-02 21:59:53 »
While testing the MOV file extraction (it seems only MOVs have form2 sectors), I compared my output with the one given by windows and noticed something weird: either windows or the driver "convert" the files with form2 sectors so they include a RIFF CDXA header, and the data contents as raw sectors.
That means that, for a STR player to read the sectors correctly it must be aware of that "special" format.

My extractor doesn't do any conversion so the output is a plain STR file.
I don't know if this is correct, and I can't check, because none of the psx movie players I found in internet play any of the movies in ff7.

EDIT: I just found a better version of PSmplay that does play all the ff7 movies as windows gives them, but fails to play my "raw mov"s.
Still, I don't know if my files are correct or not, because all the pc programs could assume the format given by windows, and not support raw form2 data in a video.

EDIT: I always used the same source drive, so I never noticed the program had this drive letter hardcoded (even if you used a diff letter in the cmdline)!
as I suppose you won't want to change your drive letters to V:, I'll upload a new build in a few mins.

http://www.uploading.com/en/files/LP490OIB/CdExtractor.zip.html
« Last Edit: 2007-06-03 02:49:43 by gigaherz »

G

  • *
  • Posts: 104
  • , kupo.
    • View Profile
Re: Anyone want to write a PSX file extractor?
« Reply #5 on: 2007-06-05 11:08:59 »
 :-(
hmmm...
Why to reinvent a bicycle?
Those of you, who are interested in mode2 files grabber - look in Q-Gears SVN q-gears/trunk/src/utils/moviedump/moviedump.cc - an utility to dump ff7 movies from image - for making it work with CD-ROM it's just enough to replace fstream iF calls with appropriate calls for reading sector of CD-ROM.

Quote
My extractor doesn't do any conversion so the output is a plain STR file.
I don't know if this is correct, and I can't check, because none of the psx movie players I found in internet play any of the movies in ff7.
I wonder why you doesn't just check MOV files with Q-Gears but searching closed-source players to check?
I see, that there are only a few people, that are interested in progress of Q-Gears, and these people know, that movies are working in Q-Gears, ADPCM sound is working too, and there actually _is_ utilities in utils directory, which makes life easier. There are so many things that need to be done on engine - battle module, intro, ... But I see - no one wants to write something new - you just reinvent your bicycles...

halkun, sorry, that I don't modify moviedump yet - me & Zidane are reverse-engeneering AKAO format, and there is some progress on it.

Cyberman

  • *
  • Posts: 1572
    • View Profile
Re: Anyone want to write a PSX file extractor?
« Reply #6 on: 2007-06-05 13:08:20 »
I wonder why you doesn't just check MOV files with Q-Gears but searching closed-source players to check?
I see, that there are only a few people, that are interested in progress of Q-Gears, and these people know, that movies are working in Q-Gears, ADPCM sound is working too, and there actually _is_ utilities in utils directory, which makes life easier. There are so many things that need to be done on engine - battle module, intro, ... But I see - no one wants to write something new - you just reinvent your bicycles...

halkun, sorry, that I don't modify moviedump yet - me & Zidane are reverse-engeneering AKAO format, and there is some progress on it.
Hey G don't be discouraged.  I did write an entire VFS to access CD ROM image information. However I was so confused at the source code I just said "oh hell" with it. LOL.  In any case the intro is part of the PS1 boot up sequence which are a few MOV's used on start up (special MOV's) I believe also a few odd sized screen loads.  These are part of the PS exe initialization. The most interesting part would be to get the programed intro working (music and animation etc). However there is a bit of perspective needed here.  Not everyone has a lot of time, right now I should be writting invoices to get money.  Some of us sweat and work weird hours to stay alive.  That's reality. It's neither bad nor good. When I get time or a chance I look at what I can do.  Impatience I've learned is destructive.  I plan a lot of things ahead.  Since I know a bit about the battle models (see the wiki) and sceneary (thanks to Mickey) that part is not a big deal.  According to Akari the entire battle system is well known and all the opcodes for the battle engine are known.  So it's merely (he says so facetiously) coding up the battle engine framework. :D

Anyhow again don't be discouraged. Some of us would like to spend hours working on it! Life has conspired against us however.

Cyb

halkun

  • Global moderator
  • *
  • Posts: 2097
  • NicoNico :)
    • View Profile
    • Q-Gears Homepage
Re: Anyone want to write a PSX file extractor?
« Reply #7 on: 2007-06-05 14:34:48 »
I'm just trying to make things easier. Even though the original FF7 uses direct sector reads, I don't really see the reason why we should. Here is the problem broken down...

1) The PSX disk is a mode 2 disk.
2) Operating Systems such as Linux can't mount mode 2 disks properly, falling back on to mode 1 and ending up with a blank disk mount.
3) Windows device drivers are not smart enough to know when the sectors switch to XA format, causing the reads to fail because the rely on the error correction, whuch has been "written over"
4) PC CD-ROMS are slow and take time to "spin-up" causing access times to slow
5) There has always been a demand to "load everything to the hard disk" to remove the cd-rom problems alltogeather.

Akari's been using standard file reads with his code. I like how my installer just copies everything off the disk and it "just works" (well, to a degree anyway), now we can use normal file access without dealing with hardware abstraction layers. I'm also looking ahead when we have to extract/convert files from the later .bin blobs in ff8/9 and PE.

That's the idea I'm taking...

As part of the system being closed-source.....

This is just the installer, which is closed source. I licensed my windows one from a company. That was kind of the reason why I started this thread. The installer already solved the issue of us distributing parts of the FF7 disk. (Oops!) New we just need a better solution for setup...
« Last Edit: 2007-06-05 14:37:47 by halkun »

Cyberman

  • *
  • Posts: 1572
    • View Profile
Re: Anyone want to write a PSX file extractor?
« Reply #8 on: 2007-06-05 16:02:38 »
Personally what I would prefer to do is use a nice DB manager (ala SQLite or something) and create BLOB databases with all the binary data in it.  That would be the idyllic thing to me.
As for reading the data off the disk etc.  Do we know EXACTLY what files are used on the FF7 boot up sequence for the PS1?
We also know something about the FIELD directory and the reason Square used direct sector access (it's in another diatribe of mine regarding said file etc.)
There is a large file in that directory that contains everything you wanted to know about the field directory in it.
I think for installation wise since the disks are NOT identical in the FF7 PS1 it might be good to create big BLOB archive packs (not the same as an LGP archive but similar in concept).  It might work to have 'Same'/Union information in a single archive and DIFF/difference/different information in DISK related databases.  This would end up one BLOB for the main data and 3 smaller BLOB archives for what remains.

I prefer something like SQLite to manage a data base.  It's over kill? Yeah it is .. well at first it might appear to be.  However being able to know what's in the data base and access it with a more standard interface (IE SQL statements) might be a godsend.  Especially when dealing with things that aren't labeled (IE FF8 Xenogears FF9 data).  It's just a thought.

Data extraction from the disks, can be done via ye olde PS1 plugins for emulators. These are ideal for this type of thing.  Since we know what we are looking for off the disk and can make a utility that will create whatever we want in the end this seems practical.  I don't know about distributing a PS1 plug-in though (details?)  Anyhow one should be able to invoke from the installer script the external program to perform the file gathering for the game.

To simplify my thoughts

So in summary:
  • Create a Q-gears specific utility program that slurps data from a disk.  This utility likely should have it's own specialized script for said extraction.  This utility should be able to identify which disk it is looking at to extract said data (IE FF7 disk 1 etc.)
  • Optionally utilize a real database system ala SQLite (public domain), and sqlite3x (c++ wrapper also PD) to create the databases for the games stored information on the target system. This adds the possibility of later improvements in Q-gears in terms of functionality and dealing with large volumes of data consistency of the interface to run the games etc. Unfortunately SQLite is limited  to 2 telibytes (2^41 bytes) sized databases and Strings and BLOBs up to 2 gigabytes (2^31 bytes). 
  • The utility should be able to be invoked from an installer with it's correct script information for it to acquire data etc. from the CD.

I don't think everyone will like the idea, however it will do some things, first keep the data from being strewn all over, second keep the data better organized, and thirdly give the option of doing more with data later on.  Most groups consider a database expost facto in such things. Considering the volume of data we are dealing with, it might be better to do so before things get really unweildy. The Database idea perhaps will also help with the injection of XML scripts etc into the engine later on. Remember it's only an idea. :D  I just hate everything being piled like a junk yard into directories.

On a side note most of these objects are accessed via numerical values from the script no?

Cyb

Guspaz

  • *
  • Posts: 27
    • View Profile
Re: Anyone want to write a PSX file extractor?
« Reply #9 on: 2007-06-05 17:24:02 »
Personally what I would prefer to do is use a nice DB manager (ala SQLite or something) and create BLOB databases with all the binary data in it.  That would be the idyllic thing to me.
As for reading the data off the disk etc.  Do we know EXACTLY what files are used on the FF7 boot up sequence for the PS1?
We also know something about the FIELD directory and the reason Square used direct sector access (it's in another diatribe of mine regarding said file etc.)
There is a large file in that directory that contains everything you wanted to know about the field directory in it.
I think for installation wise since the disks are NOT identical in the FF7 PS1 it might be good to create big BLOB archive packs (not the same as an LGP archive but similar in concept).  It might work to have 'Same'/Union information in a single archive and DIFF/difference/different information in DISK related databases.  This would end up one BLOB for the main data and 3 smaller BLOB archives for what remains.

I prefer something like SQLite to manage a data base.  It's over kill? Yeah it is .. well at first it might appear to be.  However being able to know what's in the data base and access it with a more standard interface (IE SQL statements) might be a godsend.  Especially when dealing with things that aren't labeled (IE FF8 Xenogears FF9 data).  It's just a thought.

Data extraction from the disks, can be done via ye olde PS1 plugins for emulators. These are ideal for this type of thing.  Since we know what we are looking for off the disk and can make a utility that will create whatever we want in the end this seems practical.  I don't know about distributing a PS1 plug-in though (details?)  Anyhow one should be able to invoke from the installer script the external program to perform the file gathering for the game.

To simplify my thoughts

So in summary:
  • Create a Q-gears specific utility program that slurps data from a disk.  This utility likely should have it's own specialized script for said extraction.  This utility should be able to identify which disk it is looking at to extract said data (IE FF7 disk 1 etc.)
  • Optionally utilize a real database system ala SQLite (public domain), and sqlite3x (c++ wrapper also PD) to create the databases for the games stored information on the target system. This adds the possibility of later improvements in Q-gears in terms of functionality and dealing with large volumes of data consistency of the interface to run the games etc. Unfortunately SQLite is limited  to 2 telibytes (2^41 bytes) sized databases and Strings and BLOBs up to 2 gigabytes (2^31 bytes). 
  • The utility should be able to be invoked from an installer with it's correct script information for it to acquire data etc. from the CD.

I don't think everyone will like the idea, however it will do some things, first keep the data from being strewn all over, second keep the data better organized, and thirdly give the option of doing more with data later on.  Most groups consider a database expost facto in such things. Considering the volume of data we are dealing with, it might be better to do so before things get really unweildy. The Database idea perhaps will also help with the injection of XML scripts etc into the engine later on. Remember it's only an idea. :D  I just hate everything being piled like a junk yard into directories.

On a side note most of these objects are accessed via numerical values from the script no?

Cyb

Ugh. Databases are not filesystems. They are not the ideal method of storing data. It's one thing to use a database to index or catalog data, it's enough to start shoving gigabytes of binary data into a database. Besides, if you're extracting data, it's better to stick them in files on disk where they can be properly managed/cached with zero overhead for accessing them.

I imagine it'd be much slower to start shoving FMVs into a database.

gigaherz

  • *
  • Posts: 105
    • View Profile
    • gigaherz's shitty stuff
Re: Anyone want to write a PSX file extractor?
« Reply #10 on: 2007-06-05 21:07:51 »
:-(
hmmm...
Why to reinvent a bicycle?
Those of you, who are interested in mode2 files grabber - look in Q-Gears SVN q-gears/trunk/src/utils/moviedump/moviedump.cc - an utility to dump ff7 movies from image - for making it work with CD-ROM it's just enough to replace fstream iF calls with appropriate calls for reading sector of CD-ROM.

Quote
My extractor doesn't do any conversion so the output is a plain STR file.
I don't know if this is correct, and I can't check, because none of the psx movie players I found in internet play any of the movies in ff7.
I wonder why you doesn't just check MOV files with Q-Gears but searching closed-source players to check?
I see, that there are only a few people, that are interested in progress of Q-Gears, and these people know, that movies are working in Q-Gears, ADPCM sound is working too, and there actually _is_ utilities in utils directory, which makes life easier. There are so many things that need to be done on engine - battle module, intro, ... But I see - no one wants to write something new - you just reinvent your bicycles...

halkun, sorry, that I don't modify moviedump yet - me & Zidane are reverse-engeneering AKAO format, and there is some progress on it.
heh, you see, I'm selectively lazy. I can spend hours coding something "new" (even if it has been done before), but when it comes to use other's code I get lazy. I don't know why, but the effort of understanding others' codes seems worse than just coding everything from scratch.

Also, there's many reasons to reinvent a bicycle: it could have a poorly designed frame, the wheels could be of wrong sizes, the handler could be uncomfortable, the pedals could make it harder than necessary... same with software.

"It already works" isn't a reason not to improve it. And sometimes starting from scratch is the better way to improve things.

From what I said before, you'll notice I prefer to use some closed-source player than taking the time to make the qgears code load the videos for me.

Anyway, I'm busy ATM, exams time.

Cyberman

  • *
  • Posts: 1572
    • View Profile
Re: Anyone want to write a PSX file extractor?
« Reply #11 on: 2007-06-17 14:28:10 »
Ugh. Databases are not filesystems. They are not the ideal method of storing data. It's one thing to use a database to index or catalog data, it's enough to start shoving gigabytes of binary data into a database. Besides, if you're extracting data, it's better to stick them in files on disk where they can be properly managed/cached with zero overhead for accessing them.

I imagine it'd be much slower to start shoving FMVs into a database.
I've been a bit busy (Think finding a new job etc.).  So lets actually deal with this from a pragmatic point of view.
  • Databases are ideal for storing data, isn't that what they are for?  If you mean 'game related data'. Then you are only partially right.
  • Second there is a problem with 'sticking them in files on disk' you are making the assumption that using an OS's file system 'has to be faster', it in fact can be a lot slower.  Technically very few games these days are released without using a specialized database system to store the game. So the idea that "Database is an absurd idea", is going against what has become the most common method of handling information for games. That is a database.
  • There is no such animal as zero overhead.  Each file occupies a physical location on disk. In the the FAT32 system it can be from 2K to 64K as a minimum.  So a small file whose size is say 120 bytes may actually use 32K. This is FAT only because cluster sizes for NTFS in general are much smaller. 4K being the norm. In addition to that it takes a certain amount of time seeking about to acquire said data.  So no matter what you do there will be overhead, it's just can you deal with it or is it overbearing.  This is something that can be tested quite affectively in a few minutes if someone running a light computer can run Q-gears and said database system without even noticing then I would say, there is no problem.
  • FMV/Full Motion Video files are the exception not the rule of things, since they are at least 1M Byte in size, there are fewer than 100 of them per game. So yes it would be ludicrous to put them into databases.
The real reason you use databases is not because they are 'EASY' and 'CONVIENENT' if that's what you want, consider the number of relatively small files in the FIELD directory.  Square turned these into a database by making a special large file in that directory, that they used to find the length and sector start of each of the files that relate to field locations.  That's what there engine used with field locations, ours obviously does not. :)
So in summary the idea of using a database, is not absurd nor is it something 'no one else is doing', it's common as dirt in a field when it comes to handling vast quantities of game data.  In this case there is vast quantities of data.  I don't need to go very far to point out an example, see the PC version of FF7 (it uses a database archive system).  A simple flat file is all that's needed.   So instead of having a whole hierarchy if directories occupied by thousands of small files, one can have a few directories occupied by a few files.  I like the latter myself.

That was all I really wanted to discuss. It's not like using an archive/database system is suggesting we do something absurd at all.

In general this changes one problem for another, I think the trade off is less of a problem than the shear volume of files.

Cyb

Guspaz

  • *
  • Posts: 27
    • View Profile
Re: Anyone want to write a PSX file extractor?
« Reply #12 on: 2007-06-18 01:58:25 »
Ugh. Databases are not filesystems. They are not the ideal method of storing data. It's one thing to use a database to index or catalog data, it's enough to start shoving gigabytes of binary data into a database. Besides, if you're extracting data, it's better to stick them in files on disk where they can be properly managed/cached with zero overhead for accessing them.

I imagine it'd be much slower to start shoving FMVs into a database.
I've been a bit busy (Think finding a new job etc.).  So lets actually deal with this from a pragmatic point of view.
  • Databases are ideal for storing data, isn't that what they are for?  If you mean 'game related data'. Then you are only partially right.
  • Second there is a problem with 'sticking them in files on disk' you are making the assumption that using an OS's file system 'has to be faster', it in fact can be a lot slower.  Technically very few games these days are released without using a specialized database system to store the game. So the idea that "Database is an absurd idea", is going against what has become the most common method of handling information for games. That is a database.
  • There is no such animal as zero overhead.  Each file occupies a physical location on disk. In the the FAT32 system it can be from 2K to 64K as a minimum.  So a small file whose size is say 120 bytes may actually use 32K. This is FAT only because cluster sizes for NTFS in general are much smaller. 4K being the norm. In addition to that it takes a certain amount of time seeking about to acquire said data.  So no matter what you do there will be overhead, it's just can you deal with it or is it overbearing.  This is something that can be tested quite affectively in a few minutes if someone running a light computer can run Q-gears and said database system without even noticing then I would say, there is no problem.
  • FMV/Full Motion Video files are the exception not the rule of things, since they are at least 1M Byte in size, there are fewer than 100 of them per game. So yes it would be ludicrous to put them into databases.
The real reason you use databases is not because they are 'EASY' and 'CONVIENENT' if that's what you want, consider the number of relatively small files in the FIELD directory.  Square turned these into a database by making a special large file in that directory, that they used to find the length and sector start of each of the files that relate to field locations.  That's what there engine used with field locations, ours obviously does not. :)
So in summary the idea of using a database, is not absurd nor is it something 'no one else is doing', it's common as dirt in a field when it comes to handling vast quantities of game data.  In this case there is vast quantities of data.  I don't need to go very far to point out an example, see the PC version of FF7 (it uses a database archive system).  A simple flat file is all that's needed.   So instead of having a whole hierarchy if directories occupied by thousands of small files, one can have a few directories occupied by a few files.  I like the latter myself.

That was all I really wanted to discuss. It's not like using an archive/database system is suggesting we do something absurd at all.

In general this changes one problem for another, I think the trade off is less of a problem than the shear volume of files.

Cyb

I should clarify that I was thinking pretty much solely of FMVs when writing that post. Very large files that generally don't make sense to stick in a database. I also admit that I'm looking at this from a web programmer's point of view, where putting large files into a database is viewed as a big no-no for performance reasons.

However, I'd posit that many of those games you mention are really using archives as more of a virtual filesystem rather than a proper relational database. To me, the idea of sticking FMVs into a regular relational SQL-based database just seems... bad.

As for zero overhead, assuming the file allocation tables (or inodes, or whatever the filesystem in particular is using) are stored in memory, there really is almost no overhead to access a file on the disk, as far as I know. You know what sector on disk you want to read, the disk seeks and reads it. With a database, it would seem like it might have to query the disk multiple times (depending on caching) to find the information through a query.

Flatfiles or other archive-based methods, though, would probably have very little overhead, since you'd know where the data you want was in the file, so you could just seek right to it...

I think I just made my point really poorly :P

Hades

  • *
  • Posts: 43
    • View Profile
Re: Anyone want to write a PSX file extractor?
« Reply #13 on: 2007-07-19 21:16:00 »
Sorry for the double post, its been quite some time..

I found this little C program called PSXDump:

http://www.koders.com/c/fid5A766079DE4FA99E89613A02AE286559CB49A13E.aspx

I'm having some trouble getting it to compile so far but I thought it may be useful. Let me know if I am barking up the wrong tree.

========================HUGE EDIT=================================================

Hi guys, I've been working on this for a week now (more like going around in circles). I have the disk spinning but no data is coming through to the buffer.
Can anyone see anything that I might be doing wrong with this bit of code?

Code: [Select]
#include <iostream>
#include <linux/cdrom.h>
#include <sys/ioctl.h>
#include <sys/types.h>
#include <fcntl.h>
#include <fstream>

int main() {

std::string device = "/dev/cdrw";
std::cout << "Opening Device:" << device << ".." << std::endl;

int drive;

drive = open(device.c_str(), O_RDONLY | O_NONBLOCK);

if (drive == -1) {
std::cout << "Couldnt open the bloody device!!" << std::endl;
}
else {
std::cout << "Device Open." << std::endl << std::endl;
}

union {
        struct cdrom_msf mymsf;
        char buff[2536];
    } data;

unsigned int lba = 0;

std::ofstream output;
output.open("/home/karl/log.txt");

int minute;
int second;
int frame;

for ( lba = 0; lba < 403964; lba++ ) {

output << std::endl << "lba: " << lba << " in msf = ";
minute = int((lba + 150) / (60 * 75));
second = int((lba + 150 - minute * 60 * 75) / 75);
frame = lba + 150 - minute * 60 * 75 - second * 75;
output << minute << ":" << second << ":" << frame << std::endl;

//set msf
data.mymsf.cdmsf_min0 = minute;
data.mymsf.cdmsf_sec0 = second;
data.mymsf.cdmsf_frame0 = frame;

//make ioctl call
int result = ioctl(drive, CDROMREADRAW, &data);
std::cout << "ioctl reported " << result << std::endl;

output << "buffer  = " << data.buff << std::endl;
std::cout << "buffer in lba " << lba << " contains " << data.buff << std::endl;

}

output.close();

return 0;
}


The MSF addresses are incrementing correctly, and the disk has activity. I'm just not getting any data.
Thanks everyone.  :wink:
« Last Edit: 2007-07-30 22:10:04 by Hades »

Cyberman

  • *
  • Posts: 1572
    • View Profile
Re: Anyone want to write a PSX file extractor?
« Reply #14 on: 2007-08-05 03:43:42 »
No data? I think the problem is that you need to watch your sector size carefully.

They MUST be the right size.  2352 for each sector.
ALL CD ROM disks have sectors this size (just like ALL DVD's have 2064 byte size sectors). The problem comes from the implementation the manufacturer of the disk made and the OS translation of it. The first part of the disk sector is the most important information of all. It tells you what sector you are looking at and what information you are looking at.
Code: [Select]
typedef struct
{
UINT8 Sync[12];
UINT8 Minute,
Second,
Frame;
UINT8 Mode;
}
You have 12 bytes sync data. This data is necessary for the hardware to know it's reading a sector header and synchronize it's clock speeds to it (PLL {Phase Locked Loop} hardware needs this). The Minute Second and Frame information I've explained before.  Mode is the byte you need. If you don't have this header, you  aren't getting the entire sector.  The next data depends on the mode. There are actually 2 types of Mode 1 sectors the most common is what you are use to with the following data being 2K bytes and the rest of the sector considered nothing.  Mode 2 has a sub header that determines 2 types of sectors.  Mode 2 type 1 has 2K worth of data following the sub header. Mode 2 type 2 has the stuff used for audio in the FMV's.  If you can't find the mode byte  to know if it's type 1 or 2 it's likely because the sector size isn't right.  A raw sector should be 2352 in size.

It doesn't mean you were bad or evil. LOL it just means that if you read 2336 bytes you are missing the sync and sector information you need.  Have you tried your code with a RAW image generated under windows using Nero or clone CD etc?

Cyb

Hades

  • *
  • Posts: 43
    • View Profile
Re: Anyone want to write a PSX file extractor?
« Reply #15 on: 2007-08-05 09:27:03 »
Hiya Cyb,

I trust you are well mate :)

I amended the sector size, but I'm still not getting any data into my buffer. At the moment i'm only concerned with getting the IOCTL call to work as expected, I'll move onto detecting the mode of the sector and running it through your classes when I achieve this.

I found the following resource which is the most descriptive I have found:
http://www.gelato.unsw.edu.au/lxr/source/Documentation/ioctl/cdrom.txt#L347

I only wrote to the author of cdrom.h and recieved the following example code:

Quote
Yes, that is correct. I have no examples, but it should not be that
hard. You do need to just read it one frame at the time. so fill in
ala:

Code: [Select]
struct cdrom_msf msf;

       msf.cdmsf_min0 = minute;
       msf.cdmsf_sec0 = second;
       msf.cdmsf_frame0 = frame;

Quote
as input and call the ioctl, increment m/s/f appropriately. Internally
the cdrom driver will convert that to MSF, maybe the msf_to_lba()
conversion will help you:

Code: [Select]
int msf_to_lba(char m, char s, char f)
{
       return (((m * CD_SECS) + s) * CD_FRAMES + f) - CD_MSF_OFFSET;
}

Quote
where CD_SECS is 60, CD_FRAMES is 75, and CD_MSF_OFFSET is 150.

So something like this (untested) would read one frame at the time:

Code: [Select]
char *buf;
       struct cdrom_msf *msf;
       char m, s, f;

       buf = malloc(2352); /* raw framesize */
       msf = (struct cdrom_msf *) buf;
       m = 0;
       s = 2; /* lead-in */
       f = 0;
       do {
               memset(msf, 0, sizeof(*msf));
               msf->cdmsf_min0 = m;
               msf->cdmsf_sec0 = s;
               msf->cdmsf_frame0 = f;

               if (ioctl(fd, CDROMREADRAW, buf) < 0) {
                       perror("ioctl failed\n");
                       break;
               }

               /* buf now holds the frame */

               /* increment msf to next frame */
               if (f == CD_FRAMES) {
                       f = 0;
                       s++;
               }
               if (s == CD_SECS) {
                       s = 0;
                       m++;
               }
               f++;
       } while (1);


Here is what my code currently looks like:

Code: [Select]
#include <iostream>
#include <stdlib.h>
#include <linux/cdrom.h>
#include <sys/ioctl.h>
#include <sys/types.h>
#include <fcntl.h>
#include <fstream>

int main() {

std::string device = "/dev/cdrw";
std::cout << "Opening Device:" << device << ".." << std::endl;

int drive;

drive = open(device.c_str(), O_RDONLY | O_NONBLOCK);

if (drive == -1) {
std::cout << "Couldnt open the bloody device!!" << std::endl;
}
else {
std::cout << "Device Open." << std::endl << std::endl;
}

         union {
             struct cdrom_msf msf;
             char buffer[2352];
         } arg;

unsigned int lba = 0;

std::ofstream output;
output.open("/home/karl/log.txt");

char minute = 0, second = 2, frame = 0;

for ( lba = 0; lba < 403964; lba++ ) {


//set msf
arg.msf.cdmsf_min0 = minute;
arg.msf.cdmsf_sec0 = second;
arg.msf.cdmsf_frame0 = frame;

//make ioctl call
int result = ioctl(drive, CDROMREADRAW, &arg);
std::cout << "ioctl reported " << result << std::endl;

output << "buffer  = " << arg.buffer << std::endl;
std::cout << "buffer in lba " << lba << " contains " << arg.buffer << std::endl;


/* increment msf to next frame */
               if (frame == 75) {
                       frame = 0;
                       second++;
               }
               if (second == 60) {
                       second = 0;
                       minute++;
               }
               frame++;

}

output.close();

return 0;
}

Thanks for reading  :wink:

Cyberman

  • *
  • Posts: 1572
    • View Profile
Re: Anyone want to write a PSX file extractor?
« Reply #16 on: 2007-08-05 14:14:20 »
Fun stuff that so here is some more help
peops cdr driver
https://sourceforge.net/project/showfiles.php?group_id=39401
Comes with source :) this works under Linux and windows (different binaries however).

The next is Mooby
http://mooby.psxfanatics.com/
This comes with source as well, I believe it has a lot of useful information in it because I remember using it to figure out something to do with CD's under linux. :D
Vague I know.

Cyb

Hades

  • *
  • Posts: 43
    • View Profile
Re: Anyone want to write a PSX file extractor?
« Reply #17 on: 2007-08-07 08:37:14 »
Uh.. guys..

Could my script not be printing any data because I'm not using ios::binary?

==============EDIT================

I did it!!!! I DID IT!!!  :lol:

Hades Magical FF7 Linux dumper is coming soon!!'
« Last Edit: 2007-08-07 09:02:44 by Hades »

Cyberman

  • *
  • Posts: 1572
    • View Profile
Re: Anyone want to write a PSX file extractor?
« Reply #18 on: 2007-08-07 22:37:13 »
Ok .. I will not hand you a board to smack your head.
the other day I forgot to increment a for loop termination variable so I don't feel so bad now.

Cyb

Hades

  • *
  • Posts: 43
    • View Profile
Re: Anyone want to write a PSX file extractor?
« Reply #19 on: 2007-08-08 08:35:01 »
I have these moments a lot, I had one at work the other day where I accidentally pressed the middle mouse button (paste) and then committed my changes to the svn. I wasn't very popular that day haha.

I spent a couple of hours last night looking at the code you sent me. I also found the spec for ISO 9660 and had a read of the first few pages. I figured I should probably understand the file system before I start trying to make your classes work for me. :P

Cyberman

  • *
  • Posts: 1572
    • View Profile
Re: Anyone want to write a PSX file extractor?
« Reply #20 on: 2007-08-15 01:11:00 »
I have these moments a lot, I had one at work the other day where I accidentally pressed the middle mouse button (paste) and then committed my changes to the svn. I wasn't very popular that day haha.

I spent a couple of hours last night looking at the code you sent me. I also found the spec for ISO 9660 and had a read of the first few pages. I figured I should probably understand the file system before I start trying to make your classes work for me. :P
You have the  ECMA-119 document? That's a beast of a document to say the least, it does however cover MOST of the information for the headers (but Definitely not all).  The remaining balance is in the extra books ISO issued it wants lots of money for.  Basically I have most of what is needed from Pixel's small dissertation (or diatribe however which you like it). Those may help you understand what is going on in the classes I wrote better.

Cyb

Hades

  • *
  • Posts: 43
    • View Profile
Re: Anyone want to write a PSX file extractor?
« Reply #21 on: 2007-08-15 12:30:17 »
Hey Cyb,

Hope you are well.

Actually I got the ECMA-130 document from somewhere. I haven't had the chance to look at it properly yet due to doing a paid job for someone. Thanks for the link to the dissertation though I have a feeling I'm going to need it.

Myria kitty

  • *
  • Posts: 42
    • View Profile
Re: Anyone want to write a PSX file extractor?
« Reply #22 on: 2008-01-25 23:16:14 »
I know CD-ROM pretty well if you need advice.  I also know how to do this on Windows NT.

Cyberman

  • *
  • Posts: 1572
    • View Profile
Re: Anyone want to write a PSX file extractor?
« Reply #23 on: 2008-01-28 00:18:35 »
Forum Rules might help you out.
Cyb

Hades

  • *
  • Posts: 43
    • View Profile
Re: Anyone want to write a PSX file extractor?
« Reply #24 on: 2009-10-06 17:04:59 »
Hey guys (and Cyb!), been a while I know. Sorry I dropped off the face of the earth.

Has anyone finished this or found an alternative route? I could dig out the code and have another crack at it if not/