Let's make a univesral FF7/PSX file extractor (was an installer)

[halkun]

Ok. I've been playing with this for a while, and I need some help...

In a perfect world of puppy dogs and rainbows I would like to make an installer for the PSX version of FF7 so the files can be properly transferred from the PSX CD to a PC hard drive. This is, of course, a major problem for many many reasons... I'll list them out below...

1) Windows will stall on XA-formatted sectors and will error out.
2) Linux will not mount the drive at all, or will grab the data raw, assume the XA header is part of the data and garble the sector...

The reason why I would like to make an installer is to have a file manifest and finalized format for all files on the FF7 disks. This includes the movies. Now I got the idea that I could integrate jpsxdec as part of the install process and be done with the movies once and for all. (Converting them to raw AVI files)

This leaves the problem of a Linux installer, or can jpsxdec work on linux too (being it's a java program)? Any ideas?
 Has anyone ever had problems reading the PSX files on a windows PC other than the movies?

Also is there a list of all the XA formatted files on the PSX disk. I have a list for Disk one... I think I have them all)

Code: [Select]

BIKEGET.MOV
BISKDEAD.STR
BOOGDEMO.STR
BOOGDOWN.STR
BOOGSTAR.STR
BOOGUP.STR
BRGNVL.MOV
CAR_1209.STR
D_ROPEGO.MOV
D_ROPEIN.MOV
EARITHDD.MOV
FALLPL.MOV
FSHIP2.BIN
FSHIP2N.BIN
FUNERAL.STR
GOLD1.MOV
GOLD2.MOV
GOLD3.MOV
GOLD4.MOV
GOLD5.STR
GOLD6.MOV
GOLD7.MOV
GOLD7_2.MOV
HIWIND0.MOV
JAIROFAL.MOV
JAIROFLY.MOV
JENOVA_E.MOV
JUNAIRD.STR
JUNAIRU.STR
JUNELEGO.STR
JUNELEIN.STR
JUNIN_GO.STR
JUNIN_IN.STR
JUNON.STR
MAINPLR.MOV
MK8.STR
MKUP.STR
MONITOR.STR
MTCRL.STR
MTNVL.STR
MTNVL2.STR
NIVLSFS.MOV
NORTHMK.MOV
NULL1MIN.DAT
NVLMK.MOV
ONTRAIN.MOV
OPENING.BIN
OPENINGE.MOV
OPENMV.BIN
OVER.LZS
OVER2.SND
PLREXP.MOV
RCKTFAIL.MOV
SETO.STR
SMK.STR
SOUTHMK.MOV
STAFF.BIN
STAFF2.BIN
U_ROPEGO.MOV
U_ROPEIN.MOV


[Cyberman]

Most Java programs work the same on Angstrom and regular Linux as they do under windows.

I suppose I could make a little program to surf the PSX file ISO file system and copy complete images of files a host system.

It's easiest to do this with ISO images for me.

Reading the structure of the disk is difficult since it requires the drive to support RAW mode. Not all do.

Cyb

[halkun]

Alrighty... I have made progress.
     I have found how to convert any PSX disk into a BIN/CUE (or BIN/TOC) file on any Linux, making a raw disk format that is interchangeable between computer systems. This means that, starting with this format, we can extract all the files (including the movies) as if it was an archive. You don't have to worry about if a Linux drive can mount the disk, or if the driver can parse CD-XA data. Here's what's cool...

     1) We can throw away the CUE/TOC file as it just says that the whole disk is XA mode 2, something we already know.
     2) the BIN format preserves the all 2,352 bytes in each sector, as opposed to the ISO format that only saves 2,048 bytes per sector. You see, the ISO format destroys CD-XA data headers that are critical for movie playback. (This is why you can't drag/drop movie files from the CD in windows. These use a special XA mode-2 format that is incompatible with a typical CD-ROM iso9660 filesystem).

    The bad part all open source CD-ROM image readers I have seen can't read a BIN image. I'm in the process of modding a filesystem library that extracts files from ISO images and altering it so it can take BIN images instead....After that, I'm going to write an installer that copies the files from the BIN image to the hard drive in FF7's directory structure, and convert the movies from the image to AVI files in one go. That will be the final filesystem that a q-gears or q-gears like application and read universally on any system using simple fopen() commands.

Cool?

--- EDIT ---

You know what would be cooler? After removing all the duplicate files, putting them all into one archive blob that can be read. That would just be cake :3

[koral]

It is an interesting concept halkun
But wouldn't all this conversion and removal of duplicates etc, be quite a lengthy process (in terms of execution time)?
You wouldn't be allowed to distribute a pre-created data-set, so the person running Q-Gears would have to create it themselves from their own copy of the game.

I suppose most people wouldn't mind waiting though, especially if you integrate the steps into a Q-Gears installer or something 

[m35]

@halkun
Support for raw BIN/CUE disc images, and especially raw reading from a CD drive in the Linux world is pretty poor. I've done a lot of searching for as well for jPSXdec and didn't find much. I'm curious, what program did you find that could do raw ripping? And what file system library are you modifying for the raw BIN format? I have some ISO9660 parsing code in jPSXdec, but I'm interested in what else is out there that might be of use.

Also, I made sure jPSXdec is as cross platform as possible. I've tested it and it works fine on Windows, Mac (OS X), and Linux (Ubuntu)

[G3nius]

halkun
You can use libcdio (http://www.gnu.org/software/libcdio/) for reading sectors from CD XA or BIN image. If you use Java, you can write a wrapper for this library, compile it for each OS and access wrapped functions thru JNI. Maybe it's not exactly what you want, but it might help in the following searching.

[halkun]

The idea is to make a binary asset blob that is accessible by many different computers on many different platfroms... even ones without cd-roms.

[G3nius]

The idea is to make a binary asset blob that is accessible by many different computers on many different platfroms... even ones without cd-roms.

I got it. But you wrote about troubles with parsing data from BIN image (libcdio can work with images). I see you solved it already.

[halkun]

I took a look at cdio... that looks cool. I may find a way to make something like that lightweight.

[Cyberman]

halkun you aren't hoping this is going to be easy are you?

When I say ISO image I'm refering to a RAW image. I haven't used programs that just exact 2K sectors (like windows does).

There is a bit of a problem though with the XA format.  It uses the sector encoding (IE headers) with the PSX video files. This handles the multiplexing of the audio and video data.  The video data in general uses 2K of the sector (instead of 2.25k) (it can use the whole sector however I don't believe it does). Audio uses the full sector however. You will need to include some of the header data for the interleaved video and audio data to decode properly.  I don't think that will be a big problem however since the normal header isn't needed and just the mode 2 header is useful in this case.

If you want to play the video/audio data then you will have to have an ADPCM decoder for the audio and a MJPEG decoder with the proper matrix for the video.

Cyb

[halkun]

The idea is to steal that from the video ripper posted around here. The thing is I at least have the headers.The .STR and .MOV files will not be copied, but converted during the "install" process.

[Cyberman]

Do you plan to keep the sector spacing the same? Or to just grab what data is used in each sector? It might make sense to keep all the sector data the same size to make file parsing simpler (IE load a single sized block). I believe that the MDEC hardware allowed the use of different matrices. If so that might make things ... interesting.  I'll have to look at PCSX source.
So you plan on converting the video and audio data? Hmmm if you are planning on doing that then you may wish to go the 'extra mile' and perform a much more precise DCT on the data (a floating point one). There is loss in resolution when decoding if you use an integer DCT. The original data I'm fairly sure was compressed using a floating point DCT. If you plan on scaling the video output it would be best to use the lazacross3 method as well.  The biggest issue is that there is a lot of noise in the video data due to the low resolution used and they did not use any temporal compression (Raw frames at low sample rate 15fps).  You can increase the number of frames through temporal interpolation but this is not going to make it very sharp looking.
There just isn't enough of the original data left in the MJPEG data likely.  All the MJEG data is just Hoffman compressed 8x8 chunks right?

Cyb

[halkun]

As far as the file conversion goes. All I want to accomplish is to extract the files out of a raw bin/cue (without the cue) and copy them on the hard drive. Basically I want to turn them into "real" files for direct access. With movies, I want to convert them into a standard movie format. I'm Thinking uncompressed, raw AVI files. I'll have to extract the cam data out of the headers too.

My motivation is say like you have a computer that does not have a CD-ROM. I want to have a way to create a universal file structure for FF7 so you can gab the data off of  an SD card, or a memory stick for example . This makes an engine, such as q-gears, not have to bother with CD-ROM access, and rather just grab the data for the game directly from the provided game files that were "installed" previously.

[Cyberman]

As far as the file conversion goes. All I want to accomplish is to extract the files out of a raw bin/cue (without the cue) and copy them on the hard drive. Basically I want to turn them into "real" files for direct access. With movies, I want to convert them into a standard movie format. I'm Thinking uncompressed, raw AVI files. I'll have to extract the cam data out of the headers too.

All right. Ummm you do realize that the DCT Hoffman compressed data is in YUV space? I'm not sure if all systems accept raw YUV data is all. I'm not positive but you might be able to perform a more direct conversion since it's already JPEG compressed data.  That being said it's unlikely you would want any filtering for enhancement (since it doesn't really gain anything).  I noticed the videos have a lot of noise in them mostly in the black level and open contiguous expanses.  You'll still need to make a properly interleaved AVI. You would need to convert the ADPCM to 16bit PCM, possibly to MP3 192kbps CBR with 48khz sampling.  That might be a bit interesting.  Virtual dub source might help you with the AVI but you will need to do resampling of the audio data

My motivation is say like you have a computer that does not have a CD-ROM. I want to have a way to create a universal file structure for FF7 so you can gab the data off of  an SD card, or a memory stick for example . This makes an engine, such as q-gears, not have to bother with CD-ROM access, and rather just grab the data for the game directly from the provided game files that were "installed" previously.

Hmmmm do you have an idea how you can let the engine like Q-gears know where what is?

Cyb

[halkun]

I got the idea by hanging around the ScummVM guys. They pretty much have an engine and all you do is deposit the data in a directory for the engine to access. I think this is a keen idea, and it's the reason why scummvm had been ported to so many different systems, including ones with no cd-rom drives, floppies, or even hard drives for that matter.

I'l looking at Akari's Lua idea and think it's a bad direction. We should be reading the files proper. The problem is that XA formatted cd-roms are a pain to read by anything not a playstation. This way we have a universal installer to prep the files for reading....

That is, unless Akari is looking to make a universal installer that converts *everything*

[Cyberman]

I got the idea by hanging around the ScummVM guys. They pretty much have an engine and all you do is deposit the data in a directory for the engine to access. I think this is a keen idea, and it's the reason why scummvm had been ported to so many different systems, including ones with no cd-rom drives, floppies, or even hard drives for that matter.

Simplifies everything that is a certainty.

I'l looking at Akari's Lua idea and think it's a bad direction. We should be reading the files proper. The problem is that XA formatted cd-roms are a pain to read by anything not a playstation. This way we have a universal installer to prep the files for reading....

That is, unless Akari is looking to make a universal installer that converts *everything*

not being completely up to speed on the project, LUA is being used as a data import engine?
LUA is just a scripting language. Akari could use Python just as easily.  (or TCL) the real thing is the API.  Heck Akari could use Javascript even. Doesn't matter, the thing that makes and breaks the use of scripting engines and there data is the API provided. This is why MS is having so many security issues because of there extremely poor choice in API functionality.

Cyb

[m35]

I believe that the MDEC hardware allowed the use of different matrices. If so that might make things ... interesting.  I'll have to look at PCSX source.

Does it really? I have yet to run across any games that do this, but maybe there are some out there. None of the Final Fantasy games do that though. They all use the default MDEC quantization table, which is almost identical to MPEG1.

if you are planning on doing that then you may wish to go the 'extra mile' and perform a much more precise DCT on the data (a floating point one). There is loss in resolution when decoding if you use an integer DCT.

jPSXdec has a full double-precision Playstation video decoder. My comparisons with other decoders suggest the resulting colors are more accurate, although I haven't compared it to ffmpeg's since they updated it. The only way I know you could squeeze out even more quality is to use some algorithm that magically improves macro-block decoding, which is something ffmpeg has.

For a completely different approach, I have been pondering about the fourcc MJPG codec. Since many implementations of that codec simply stuff a full jpeg file into each frame, I wonder if the variable-length codes could be parsed and rewritten into jpeg files--but then include a color-profile in the jpeg to adjust for the slight difference between standard YCbCr color space and the PSX's. This would provide a completely lossless conversion that might work on modern operating systems. I haven't looked into this idea at all, but since not even all graphics programs handle jpeg color-profiles, there are probably even fewer MJPG codec implementations that could handle it.

[Cyberman]

I believe that the MDEC hardware allowed the use of different matrices. If so that might make things ... interesting.  I'll have to look at PCSX source.

Does it really? I have yet to run across any games that do this, but maybe there are some out there. None of the Final Fantasy games do that though. They all use the default MDEC quantization table, which is almost identical to MPEG1.

However that is unlikely due to the restrictions in licensing the PsyQ library and the use of the PsyQ library for PS1 development exclusively.

if you are planning on doing that then you may wish to go the 'extra mile' and perform a much more precise DCT on the data (a floating point one). There is loss in resolution when decoding if you use an integer DCT.

jPSXdec has a full double-precision Playstation video decoder. My comparisons with other decoders suggest the resulting colors are more accurate, although I haven't compared it to ffmpeg's since they updated it. The only way I know you could squeeze out even more quality is to use some algorithm that magically improves macro-block decoding, which is something ffmpeg has.

Hmmm it's a color depth issue really, a lot is lost in truncation (lost bits) double precision adds more resolution in low level areas (IE dark zones) than integer computations. This leaves dark areas less likely to look a dark gray and more like black.  (all this I am sure you are aware of I'm merely stating this so in case someone else reads this they don't go 'why's that?')

For a completely different approach, I have been pondering about the fourcc MJPG codec. Since many implementations of that codec simply stuff a full jpeg file into each frame, I wonder if the variable-length codes could be parsed and rewritten into jpeg files--but then include a color-profile in the jpeg to adjust for the slight difference between standard YCbCr color space and the PSX's. This would provide a completely lossless conversion that might work on modern operating systems. I haven't looked into this idea at all, but since not even all graphics programs handle jpeg color-profiles, there are probably even fewer MJPG codec implementations that could handle it.

I don't know but because the data isn't using a standard color matrix (or should I say commonly accepted?) matching it might be a bit tricky.  I would prefer to take the raw data instead of deblock decode then encode and reblock with yet more truncation error (LOL). So I am not sure what to do in those respect.  Are there any standard CLI based decoder/encoder that can bridge streams.  If the data remains in YUV/ YCbCr space then the precision should remain intact.  Can MJPEG codecs for most OS's work with the color matrix for the MDEC system Sony made?  I suppose that's the question.  Otherwise some data twiddling will be involved.

Cyb