Author Topic: Kickstarting Final Fantasy X Modding  (Read 300 times)

MetaLink

  • Fast newbie
  • *
  • Posts: 43
  • Karma: 0
    • View Profile
Kickstarting Final Fantasy X Modding
« on: 2017-10-26 16:39:56 »
I'm starting this thread to raise awareness to the fact that FFX is a lot easier to mod than expected, it has a non proprietary formats such has FSB for music sound effects and voices and webm for FMVs, the model and texture files are in dae.phyre and dds.pyhre respectively, which are former dds and dae files converted to the pyhre format


To get access to the game files first you need to extract the VBF files found in the game's data folder using this VBF unpacker by tek547 found in this thread http://forum.xentax.com/viewtopic.php?f=10&t=14340&sid=10a8543e65453ec5ad5bccdc5f4594ca&start=15

These are the folders that are in the ps3data directory these contain mostly meshes textures sounds FMVs and unused or old files from the PS2 version

battle - this folder seems to only contain font files
btlmap - this contains the level files from when u enter a battle
chr - contains monter, npc, random objects like barrels, playable characters, summons, and weapons
event - seems to contain textures only
flash - contains swf files used in the games boosters, ESC menu, the load and save screen, pretty much every icon or menu added in the PC version
fonts - contains one unknown file
help - contains textures used in the help menu and sphere monitors
lockit contains 8 .bin files
magic - contains textures for magic effects like fire
map - contains the non battle level files
menu - contains textures for the titlescreen, ingame menu and battle, also seems to contain some unused ps4 textures
savedataicons - contains the images seen in the load and save menu there are all png files
savesforviewer - contains 110 unknown files all numbered
shaders - contains shaders, i assume, in fx.phyre format
sound_pc - contains the arranged and original music, voices in english and japanese and sound effects, all these are FSB and FEV format, with 2 text files named common and loop for each bank
syncdata - contains 6 txt files no idea what this is for
texturevideo - contains 4 bin files named: texvideolchb, texvideolchb04, texvideoluca, texvideoluca08
video - contains the FMV files in japanese and english
yonishi_data - contains a few texture files


The map folder has each level on its own separated folder, i extracted them to see their contents and this is what i found

azit - al bhed home
bika - bikanel desert
bjyt - baaj temple
bltz - blitzball stadium
bsil - besaid island
bsmm - besaid beach (flashback)
bsvr - besaid village
bvyt - besaid temple
cdsp - al bhed boat & underwater ruins
djyt - djose temple
dome - zanarkand dome
dream - unknown
genk - moonflow
grid - sphere grid plane
guad - guadosalam
hiku - airship & world map
ikai - farplane
kami - thunder plains
kino - mushroom rock
klyt - kilika woods & temple
lchb - luca
lmyt - remiem temple
luca - luca square and pre-rendered backgrounds
maca - lake macalania
mcfr - macalania forest
mcyt - macalania temple
mihn - mi'hen highroad
mmmc - unknown
msmm - via purifico (maze)
mtgz - mt gagazet, caves, upper zanarkand
nagi - calms lands & cavern of the stolen fayth
omeg - omega ruins
ptkl - kilika town
sins - inside sin
slik - ss liki
ssbt - airship model
stbv - bevelle highbridge, via purifico (sewer), ffx-2 map
swin - ss winno
titl - main menu
zkrn - zanarkand ruins
znkd - dream zanarkand
zzzz - unknown

These folders also contain textures in dds.phyre these include the ones used in the meshes and also pre rendered backgrounds


In the pc folder found within the chr folder are the 7 playable characters + seymour's models. These include the high and low poly models

It also includes the PS2 version's models fully rigged, although these have no textures and the UV maps seem to be stretched, they are located in the following folders:

c811 Tidus
c812 Yuna
c813 Auron
c814 Kimahri
c815 Wakka
c816 Lulu
c817 Rikku


The audio files that are in FSB format can be opened with FMOD designer and can be played



Unfortunately to edit them you need to create a new FSB file from scratch

Someone managed to do it but he had to share the entire 20 GB VBF file, which is a problem: https://www.youtube.com/watch?v=5-RpYn6sfk8

The FMV should be fairly simple to replace they are just video files in webm format

You can convert and display the .dds.phyre files with this Noesis plugin: http://forum.xentax.com/viewtopic.php?f=18&t=16593
You can convert the .dae.phyre files with this: http://forum.xentax.com/viewtopic.php?f=16&t=16930, drag and drop them into ffx.exe, it will convert them to either .smd or .ascii. Be aware that most map files will have missing UV maps


There is another tool to convert the .dae.phyre files this one won't work on the map files and it only converts to obj, here is the source code in python


phyre.py

Spoiler: show
# -*- coding: utf-8 -*-

# extractMesh(inputFile[, objFile][, keywordArg1...])
#   Extract mesh from .dae.phyre file and convert to obj format
#   If objFile is not specified, the data is processed but not written out
#   See meshArgs0 below for the various optiona keyword arguments that
#   can be specified. Note that most are for debugging purposes and should not
#   be needed.
#
# extractDDS(inputFile, objFile[, keywordArg1...])
#   Extract DDS file from a .dds.phyre file and convert to .dds format.
#   See ddsArgs0 below for optional keyword arguments.


#------------------------------------------------------------------------------
#------------------------------------------------------------------------------
#------------------------------------------------------------------------------
#------------------------------------------------------------------------------
#------------------------------------------------------------------------------
#------------------------------------------------------------------------------

import os
import struct

# global variable names with default values
meshArgs0={
   'faceHeaderAddr': 0x0, # Starting address to search for face header
   'faceStartAddr': 0x0, # Starting address to search for face definitions
   'vertStartAddr': None, # Address of first vertex block (None=find automatically)
   'vertHeaderAddr': 0x0, # Starting address to search for vertex header
   'includeNormals': False,  # Include face normals
   'invertVertUV': True, # Invert vertical coordinate of UV maps
   'maxVert': 1.e3, # Warning if any vertices exceed max
   'uvBounds': (-.01, 1.01), # Warn if any UV coordinates outside bounds
   'normTol': 1.e-6, # Warn if normals aren't within norm tolerance
   'debug': False, # Debugging output (messy)
   'showWarn': True,  #Turn off warnings about expected values
   'maxWarns': 25 # maximum number of warnings per function call
}

ddsArgs0={'ddsStartAddr': None, # Start address for DDS data (None=find automatically)
         'width': None, # Forced width resolution (None=find automatically)
         'height': None, # Forced height resolution (None=find automatically)
         'encode': None, # DXT1/DXT3/DXT5/ARGB8, (None=find automatically)
         'mipMaps': None  # Number of mip maps in file (None=find automatically)
        }

       
meshArgs = []
ddsArgs = []

# Data to search for to find start of face index list
# Cannot be called as a keyword argument!
firstFace = struct.pack('3H', 0, 1, 2) # search for 0, 1, 2 as first face

encode0={'DXT5':  {'bbp':  8, 'minDim': 4}, \
        'DXT3':  {'bbp':  8, 'minDim': 4}, \
        'DXT1':  {'bbp':  4, 'minDim':  4}, \
        'ARGB8': {'bbp': 32, 'minDim':  1} \
       }

#------------------------------------------------------------------------------
#------------------------------------------------------------------------------
# Generic functions
def parseKeywords(options0, kwargs):
    # Parse provided keyword arguments, and fill in defaults to dict
   
    options = options0.copy()
    for key, val in kwargs.items():
        options[key] = val
    return options

#------------------------------------------------------------------------------
#------------------------------------------------------------------------------
# extractMesh functions

def extractMesh(inputFile, objFile=None, **kwargs):
    # Primary driver for extracting mesh

    global meshArgs
    meshArgs = parseKeywords(meshArgs0, kwargs)
   
    print("EXTRACTMESH")
   
    print("Reading phyre file %s..." % inputFile)
    with open(inputFile,'rb') as file:
        f = file.read()
   
    print("Extracting faces...")
    faceSet = extractFaceSets(f)
    if not faceSet:
        raise Exception("Faces could not be found")
   
    if meshArgs['debug']: print("Number of sets: %d" % len(faceSet))
   
    if meshArgs['vertStartAddr'] is None:
        nbyte = faceSet[-1]['nFace']*2*3
        meshArgs['vertStartAddr'] = faceSet[-1]['addr'] + nbyte
        if meshArgs['debug']:
            print("Computed start address of vertices: " + hex(meshArgs['vertStartAddr']))
    else:
        print("User-supplied start address of vertices: " + hex(meshArgs['vertStartAddr']))
   
    print("Finding vertex block addresses...")
    vertBlockAddr = findVertAddresses(f, faceSet)
    if vertBlockAddr is None:
        raise Exception("Vertex addresses could not be found")
       
    print("Extracting vertices...")
    vertSet = extractVertSets(f, faceSet, vertBlockAddr)
   
    print("Extracting UV maps...")
    uvSet = extractUvSets(f, faceSet, vertBlockAddr)

    if meshArgs['includeNormals']:
        print("Extracting normals...")
        normSet = extractNormSets(f, faceSet, vertBlockAddr)
    else:
        normSet = None
        print("Ignoring normals")   
       
    if objFile is not None:
        print("Writing object file to %s..." % objFile)
        writeObjFile(objFile, faceSet, vertSet, uvSet, normSet)
   
    print("Summary:")
    print("  Total Sets:     %d" % len(faceSet))
    print("  Total Faces:    %d" % sum(n['nFace'] for n in faceSet))
    print("  Total vertices: %d" % sum(n['nVert'] for n in faceSet))
    print("  ----------------------------------------------------------------------")
    print("  | ID | Faces | Verts | Face Addr | Vert Addr |   UV Addr | Norm Addr |")
    print("  ----------------------------------------------------------------------")
    for i in range(len(faceSet)):
        if meshArgs['includeNormals']:
            normAddr=hex(normSet['addr'])
        else:
            normAddr=""
        if uvSet is None:
            uvAddr=""
        else:
            uvAddr=hex(uvSet['addr'])
        print("  | %2d | %5d | %5d | %9s | %9s | %9s | %9s |" % \
          (i,  faceSet['nFace'], faceSet['nVert'], \
           hex(faceSet['addr']), hex(vertSet['addr']), \
           uvAddr, normAddr))
    print("  ----------------------------------------------------------------------")
   
#------------------------------------------------------------------------------
def extractFaceSets(f):   
    # Extract face sets by looking up header info
   
    faceSet = []

    print("  Finding start of face header blocks...")
    match = meshArgs['faceHeaderAddr']
    faceHeaderCatch=b'\xff\xff\xff\xff'
    match = f.find(faceHeaderCatch, match+1)
    while match > 0:
        block = struct.unpack_from('27I', f, match)
        nFace = block[13]/3
        nVert = block[12]+1
       
        valid = True
        if nFace % 1 > 0: valid = False
        if nFace <= 0: valid = False
        if nFace > 0xffff: valid = False # needs to fit in uint16
        if nVert < 3: valid = False  # always at least 3 vertices
        if nVert >= 0xffff: valid = False # needs to fit in uint16
        if block[22] != 0: valid = False # face block offset (0 for first block)
        if block[24] != nFace*2*3: valid = False # number of bytes in face block
       
        if valid:
            break
        else:
            match = f.find(faceHeaderCatch, match+1)
           
    pos = match
    if match < 0:
        print("    FAIL: Couldn't find face header block")
        return None
    if meshArgs['debug']: print("    Start of face header blocks: " + hex(pos))
    block = struct.unpack_from('27I', f, pos)
    iset = 0
    nFace = 0
    faceAddr = 0
    print("  Processing face header blocks...")
    while block[0] == 0xffffffff:
       
        # word-align for iset>0
        if iset>0 and (nFace%2) == 1:
            faceAddr += 2
       
        nFace = int(block[13]/3)
        nVert = block[12]+1
        if nFace <= 0 or nFace > 0xffff or nVert < 3 or nVert > 0xffff:
            print("    FAIL: Unexpected number of faces (%d) or verts (%d) for set %d" % (nFace, nVert, iset))
            return None
           
        # Find initial face address
        if iset == 0:
            print("    Finding face start address...")
            faceAddr = findFaceStartAddr(f, nFace, nVert)
            if faceAddr is None:
                print("      FAIL: Could not find face start address")
                return None
       
        faceSet.append({'addr': faceAddr, \
                        'faces': [], \
                        'nFace': nFace, \
                        'nVert': nVert})
       
        if meshArgs['debug']:
            print("    Face ID: %2d Address: %10s  #Faces: %5d  #Verts: %5d" % (iset, hex(faceAddr), nFace, nVert))
       
        # Grab faces
        posFace = faceSet[iset]['addr']
        for i in range(nFace):
            face = struct.unpack_from('3H', f, posFace)
            if max(face) + 1 > nVert:
                print("    FAIL: Could not read faces, vertex index (%d) higher than expected max (%d) on set %d" % (max(face)+1, nVert, len(faceSet)-1))
                return None
            elif i==0 and face != (0, 1, 2):
                print("    WARN: Expected start of faces to be (0,1,2), instead received (%d, %d, %d) for set %d" % (face[0], face[1], face[2],  i))
            faceSet[iset]['faces'].append(face)
            posFace += 6
        maxVert = max(max(x) for x in faceSet[iset]['faces'])+1
        if maxVert < nVert:
            print("    WARN: Max vert index (%d) less than nVert (%d) for set %d" % (maxVert, nVert, iset))
           
        faceAddr += 2*3*nFace
        pos += 27*4
        iset += 1
        block = struct.unpack_from('27I', f, pos)
    if not faceSet:
        print("    FAIL: Could not read face header block")
        return None
    return faceSet
       
   
def findFaceStartAddr(f, nFace, nVert):
   
    pos0 = meshArgs['faceStartAddr']
    match = f.find(firstFace, pos0)
    while match >= 0:
        if meshArgs['debug']:
            print("      Possible face start address: " + hex(match))
        pos = match
        iFail = False
        imax = 0
        for i in range(nFace):
            face = struct.unpack_from('3H', f, pos)
            if max(face) + 1 > nVert or max(face) > imax + 3:
                if meshArgs['debug']:
                    print("        Face values (face=%d, max=%d, prevMax=%d, nVert=%d) not consistent at address. Continuing search..." % (i, max(face), imax, nVert))
                iFail = True
                break;
            imax = max(imax, max(face))
            pos += 6
        if iFail:
            pos0 = pos
            match = f.find(firstFace, pos0)
        else:
            if meshArgs['debug']:
                print("      Found face start address: " + hex(match))
            return match
    return None

#------------------------------------------------------------------------------
def findVertAddresses(f, faceSet):
    # Required for the few files that don't have the same number of floats per
    # vertex in the vertex block data. Seems to work for everything
   
    headerCatch=struct.pack('2I', 12, int(faceSet[0]['nVert'])) 
    print("  Finding start of vertex header blocks...")
    offset = meshArgs['vertHeaderAddr']
    match = f.find(headerCatch, offset)
    while match >= 0:
        block = struct.unpack_from('16I', f, match)
        if block[14] == faceSet[0]['nVert']*4*3:
            break;
        match = f.find(headerCatch, match+1)
    if match < 0:
        print("    FAIL: Could not find start of header info")
        return None
    pos = match
    if meshArgs['debug']: print("    Start of header info: " + hex(pos))

    addr = [meshArgs['vertStartAddr']]
    iset = 0
    s = 0 # current position relative to last set address
    print("  Processing vertex header blocks...")
    while iset <= len(faceSet)-1:
        pos0 = pos
        # Check for equally sized sets
        imult = 1
        while (iset + imult) < len(faceSet) and faceSet[iset]['nVert'] == faceSet[iset+imult]['nVert']:
            imult += 1
        if meshArgs['debug'] and imult > 1:
            print("    Sets %d-%d have same # verticies. Assuming equal split" % (iset, iset+imult-1))
        (pos, s) = getHeaderBlockSize(f, pos, faceSet[iset]['nVert'])
        if s == 0:
            print("    FAIL: Could not read header info for set %d" % iset)
            return None
        s = int(s/imult)
        for i in range(imult):
            addr.append(addr[-1] + s)
            if meshArgs['debug']:
                print("    Set %d has %d floats per vertex (header info starts at %s)" \
                      % (iset, s/4/faceSet[iset]['nVert'], hex(pos0)))
            iset += 1
    return addr
   
#------------------------------------------------------------------------------
def getHeaderBlockSize(f, pos, nVert):
   
    vertSize = 0
    info = struct.unpack_from('16I', f, pos)
    if info[0] != 12:
        return (pos, vertSize)
    while info[1] == nVert:
        pos += 16*4
        vertSize += info[0]*info[1] # bytes of total vertex block
        info = struct.unpack_from('16I', f, pos)
    return (pos, vertSize)
   
#------------------------------------------------------------------------------   
def extractVertSets(f, faceSet, vertBlockAddr):
    # Pull vertex data
   
    vertSet = []
    iWarn = 0
    for iset in range(len(faceSet)):
        nVert = faceSet[iset]['nVert']
        pos = vertBlockAddr[iset]
        vertSet.append({'addr': pos, 'nVert': nVert, 'verts': []})
        if meshArgs['debug']:
            print("  Vert set %2d at %s" % (iset, hex(vertSet[iset]['addr'])))
        for iv in range(nVert):
            vert = struct.unpack_from('3f', f, pos)
            if meshArgs['showWarn'] and (max(map(abs,vert)) > meshArgs['maxVert']):
                iWarn += 1
                if iWarn <= meshArgs['maxWarns']:
                    print("  WARN: Vertex %d in set %d has large values! " \
                          % (iv, iset) + "(%.8f %.8f %.8f)" % vert)
                elif iWarn == meshArgs['maxWarns'] + 1:
                    print("  Additional warnings suppressed")
            vertSet[iset]['verts'].append(vert)
            pos += 3*4
    return vertSet

#------------------------------------------------------------------------------   
def extractUvSets(f, faceSet, vertBlockAddr):
    # Pull UV data
   
    uvSet = []
    iWarn = 0
   
    # Check if there's room for UV maps in first set
    nVert = faceSet[0]['nVert']
    if len(faceSet) == 1:
        nfpv = int((len(f)-1 - vertBlockAddr[0])/4/nVert)
    else:
        nfpv = (vertBlockAddr[1] - vertBlockAddr[0])/4/nVert
    if nfpv < 8:
        if  meshArgs['showWarn']:
            print("  NOTE: UV maps not present (%f floats per vert). Does a texture file exist?" % nfpv)
        return None
       
    for iset in range(len(faceSet)):
        nVert = faceSet[iset]['nVert']
        pos = vertBlockAddr[iset] + (3+3)*4*nVert
        uvSet.append({'addr': pos, 'nVert': nVert, 'uvs': []})
        if meshArgs['debug']:
            print("  UV set %2d at %s" % (iset, hex(uvSet[iset]['addr'])))
        for iv in range(nVert):
            uv = struct.unpack_from('2f', f, pos)
            if meshArgs['showWarn'] and \
            (max(uv) > meshArgs['uvBounds'][1] or min(uv) < meshArgs['uvBounds'][0]):
                iWarn += 1
                if iWarn <= meshArgs['maxWarns']:
                    print("  WARN: UV map %d in set %d is out of expected bounds! "\
                          % (iv, iset) + "(%.8f %.8f)" % uv)
                elif iWarn == meshArgs['maxWarns'] + 1:
                    print("  Additional warnings suppressed")
            uvSet[iset]['uvs'].append(uv)
            pos += 2*4
    if meshArgs['invertVertUV']:
        print("  Inverting vertical component of UV maps...")
        invertUv(uvSet)
    return uvSet

#------------------------------------------------------------------------------
def extractNormSets(f, faceSet, vertBlockAddr):
    # Pull normals data
   
    normSet = []
    iWarn = 0
    for iset in range(len(faceSet)):
        nVert = faceSet[iset]['nVert']
        pos = vertBlockAddr[iset] + nVert*4*(3)
        normSet.append({'addr': pos, 'nVert': nVert, 'norms': []})
        if meshArgs['debug']:
            print("  Normals set %2d at %s" % (iset, hex(normSet[iset]['addr'])))
        for iv in range(nVert):
            nrm = struct.unpack_from('3f', f, pos)
            if meshArgs['showWarn'] and (abs(l2Norm(nrm)-1.0) > meshArgs['normTol']):
                iWarn += 1
                if iWarn <= meshArgs['maxWarns']:
                    print("  WARN: Norm %d in set %d is outside tolerance! "\
                          %(iv, iset) + "(%.8f)" % l2Norm(nrm))
                elif iWarn == meshArgs['maxWarns'] + 1:
                    print("  Additional warnings suppressed")
            normSet[iset]['norms'].append(nrm)
            pos +=3*4
    return normSet

#------------------------------------------------------------------------------       
def invertUv(uvSet):
    # Invert UV map
   
    for iset in range(len(uvSet)):
        for iv in range(uvSet[iset]['nVert']):
            uvSet[iset]['uvs'][iv] = (uvSet[iset]['uvs'][iv][0], \
                                      1.0 - uvSet[iset]['uvs'][iv][1])

#------------------------------------------------------------------------------
def l2Norm(x):
    # Simple L2 norm
   
    return  sum(y**2 for y in x)**.5

#------------------------------------------------------------------------------
def writeObjFile(objFile, faceSet, vertSet, uvSet, normSet=None):
    # Write the .obj file 
   
    file = open(objFile, 'w')
   
    file.write("# %s\n" % os.path.basename(objFile))
    file.write("# Total vertices: %d\n" % sum(face['nVert'] for face in faceSet))
    file.write("# Total faces: %d\n" % sum(face['nFace'] for face in faceSet))
   
    file.write("#\n# Vertices\n")
    for iset in range(len(vertSet)):
        if iset == 0:
            faceSet[0]['offset'] = 1
        else:
            faceSet[iset]['offset'] = faceSet[iset-1]['offset'] + faceSet[iset-1]['nVert']
        file.write("# Starting Address: %s (%d vertices)\n" \
                   % (hex(vertSet[iset]['addr']), vertSet[iset]['nVert']))
        for iv in range(vertSet[iset]['nVert']):
            file.write("v %.8e %.8e %.8e\n" % vertSet[iset]['verts'][iv])
   
    if uvSet is not None:
        file.write("#\n# UV Maps\n")
        for iset in range(len(uvSet)):
            file.write("# Starting Address: %s (%d UV vertices)\n" \
                       % (hex(uvSet[iset]['addr']), uvSet[iset]['nVert']))
            for iv in range(uvSet[iset]['nVert']):
                file.write("vt %.8e %.8e\n" % uvSet[iset]['uvs'][iv])
           
    if normSet is not None:
        file.write("#\n# Normals\n")
        for iset in range(len(normSet)):
            file.write("# Starting Adress: %s (%d normals)\n" \
                       % (hex(normSet[iset]['addr']), normSet[iset]['nVert']))
            for iv in range(normSet[iset]['nVert']):
                file.write("vn %.8e %.8e %.8e\n" % normSet[iset]['norms'][iv])
       
    file.write("#\n# Face indices\n")
    for iset in range(len(faceSet)):
        file.write("# Starting Address: %s (%d faces, %d vertices)\n" \
                   % (hex(faceSet[iset]['addr']), faceSet[iset]['nFace'], \
                      faceSet[iset]['nVert']))
        file.write("g %s\n" % ("obj_" + str(iset)))
        offset = faceSet[iset]['offset']
        for i in range(faceSet[iset]['nFace']):
            face = faceSet[iset]['faces']
            face = tuple(j+offset for j in face)
            if uvSet is not None and normSet is None:
                file.write("f %d/%d %d/%d %d/%d\n" % \
                           (face[0], face[0], \
                            face[1],face[1], \
                            face[2], face[2]))
            elif uvSet is None and normSet is not None:
                file.write("f %d//%d %d//%d %d//%d\n" % \
                           (face[0], face[0], \
                            face[1],face[1], \
                            face[2], face[2]))
            elif uvSet is None and normSet is None:
                file.write("f %d %d %d\n" % (face[0], face[1], face[2]))
            else:
                file.write("f %d/%d/%d %d/%d/%d %d/%d/%d\n" % \
                           (face[0], face[0], face[0], \
                            face[1], face[1], face[1], \
                            face[2], face[2], face[2]))
    file.close()
   
#------------------------------------------------------------------------------
#------------------------------------------------------------------------------
# DDS functions

def extractDDS(phyreFile, ddsFile, **kwargs):
    # Main driver for DDS file extraction
   
    global ddsArgs, encode
   
    ddsArgs = parseKeywords(ddsArgs0, kwargs)
   
    print("EXTRACTDDS")
    if isinstance(ddsArgs['ddsStartAddr'], str):
        ddsArgs['ddsStartAddr'] = int(ddsArgs['ddsStartAddr'], 16)
   
    with open(phyreFile, 'rb') as myfile:
        f = myfile.read()

    if ddsArgs['encode'] is None:
        (ddsArgs['encode'], ddsArgs['ddsStartAddr']) = findEncoding(f)
        print("Encoding: " + ddsArgs['encode'])
        print("DDS start address: " + hex(ddsArgs['ddsStartAddr']))
    else:
        if ddsArgs['ddsStartAddr'] is not None:
            raise Exception("ddsStartAddr must be specified if encoding type is specified " \
            + "(Try 0xa68 for DXT or 0xa69 for ARGB8)")
        print("User supplied encoding: " + ddsArgs['encode'])
        print("User supplied start address: " + hex(ddsArgs['ddsStartAddr']))
    encode = encode0[ddsArgs['encode']]
           
    (width, height, mips) = getHeaderData(f)
    if (ddsArgs['width'] is None) != (ddsArgs['height'] is None): # biconditional and
        raise Exception('Width and height must be specified together')
    elif ddsArgs['width'] is None:
        (ddsArgs['width'], ddsArgs['height'])= (width, height)
        print("Extracted resolution: %dx%d" % (ddsArgs['width'], ddsArgs['height']))
    else:
        print("User provided resolution: %dx%d" % (ddsArgs['width'], ddsArgs['height']))
       
    if ddsArgs['mipMaps'] is None:
        print("Number of mip maps: %d" % mips)
        ddsArgs['mipMaps'] = mips
    else:
        print("User provided number of mip maps: %d" % ddsArgs['mipMaps'])

    header=buildHeader()
    with open(ddsFile, 'wb') as myfile:
        myfile.write(header + f[ddsArgs['ddsStartAddr']:])
    print("File written to: " + ddsFile)
   
#------------------------------------------------------------------------------
def findEncoding(dds_data):
    # Determine encoding by searching data structure
   
    for key, value in encode0.items():
        s = dds_data.find(str.encode(key))
        if s >= 0:
            startaddr = s + len(key) + 0x26
            return (key, startaddr)
    raise Exception('Encoding scheme could not be found')

#------------------------------------------------------------------------------
def getHeaderData(f):
    # Find resolution and number of mip maps
    import math

    match = f.find(b"PS3Data")
    if match<0:
        raise Exception("Could not find start of header data")
    pos = match+16*3
    mips = struct.unpack_from('I', f, pos)[0]
    pos += 16 
    (width, height) = struct.unpack_from('2I', f, pos)
    if (math.log2(width) % 1) != 0 :
        print("WARN: Width not a power of 2")
    elif max(width,height) > 4096:
        print("WARN: Large width or height")
    elif min(width,height) < 1:
        print("WARN: Small width or height")
    elif math.ceil(math.log2(max(width, height))) < mips:
        print("WARN: More mipmaps than expected")
    return(width, height, mips)
   
#------------------------------------------------------------------------------
def buildHeader(): 
    # build DDS header, assume DXT5
   
    def uf(x): return struct.pack('I', x)
   
    # dwFlags flags
    ddsd = {'caps': 0x1, 'height': 0x2, 'width': 0x4, 'pitch': 0x8, \
            'pixelFormat': 0x1000, 'mipMapCount': 0x20000, \
            'linearSize': 0x80000, 'depth': 0x800000
           }
   
    # dwCaps flags
    ddscaps = {'complex': 0x8, 'mipMap': 0x400000, 'texture': 0x1000}
   
    # dwCaps2 flags
#    ddscaps2 = {'cubeMap': 0x200, 'volume': 0x200000, \
#                'positiveX': 0x400,   'negativeX': 0x800, \
#                'positiveY': 0x1000,  'negativeY': 0x2000, \
#                'positiveZ': 0x4000,   'negativeZ': 0x8000               
#               }
       
    dwSize = uf(124)
    dwFlags0 = ddsd['caps'] + ddsd['height'] + ddsd['width'] \
                 + ddsd['pixelFormat'] + ddsd['mipMapCount']
    dwHeight = uf(ddsArgs['height'])
    dwWidth = uf(ddsArgs['width'])
    dwPitchOrLinearSize = uf(0) # gets set depending on encoding below
    dwDepth = uf(0)
    dwMipMapCount = uf(ddsArgs['mipMaps'])
    dwReserved1 = b''.join([uf(0) for i in range(11)])
    ddspf = buildDdsPixelFormat()
    dwCaps = uf(ddscaps['complex'] + ddscaps['mipMap'] + ddscaps['texture'])
    dwCaps2 = uf(0)
    dwCaps3 = uf(0)
    dwCaps4 = uf(0)
    dwReserved2 = uf(0)
   
    if ddsArgs['encode'][0:3] == 'DXT':
        dwPitchOrLinearSize = uf(max(1, int(((ddsArgs['width'] + 3)/4)))*(encode['bbp']*2))
        dwFlags = uf(dwFlags0 + ddsd['linearSize'])
    elif ddsArgs['encode'] == 'ARGB8':
        dwPitchOrLinearSize = uf(int((ddsArgs['width']*encode['bbp']+7)/8))
        dwFlags = uf(dwFlags0 +ddsd['pitch'])
    else:
        raise Exception("Unexpected encoding type: " + ddsArgs['encode'])
   
    return (b'DDS ' + dwSize + dwFlags + dwHeight + dwWidth \
            + dwPitchOrLinearSize + dwDepth + dwMipMapCount + dwReserved1 \
            + ddspf + dwCaps + dwCaps2 + dwCaps3 + dwCaps4 + dwReserved2 )
           
#------------------------------------------------------------------------------       
def buildDdsPixelFormat():
    # Build DDSPixelFormat structure for header (assume DXT5)
   
    def uf(x): return struct.pack('I', x)
   
    # dwFlags flags
    ddpf = {'alphaPixels': 0x1, 'alpha': 0x2, 'fourCC': 0x4, 'rgb': 0x40, \
            'yuv': 0x200, 'luminance': 0x20000
            }
   
    dwSize = uf(32)

    if ddsArgs['encode'][0:3] == 'DXT':
        dwFlags = uf(ddpf['fourCC'])
        dwFourCC = str.encode(ddsArgs['encode'])
        dwRGBBitCount = uf(0)
        dwRBitMask = uf(0)
        dwGBitMask = uf(0)
        dwBBitMask = uf(0)
        dwABitMask = uf(0)
    elif ddsArgs['encode'] == 'ARGB8':
        dwFlags = uf(ddpf['alphaPixels'] + ddpf['rgb'])
        dwFourCC = uf(0)
        dwRGBBitCount = uf(encode['bbp'])
        dwRBitMask = uf(0x00ff0000)
        dwGBitMask = uf(0x0000ff00)
        dwBBitMask = uf(0x000000ff)
        dwABitMask = uf(0xff000000)
    else:
        raise Exception("Unexpected encoding type: " + ddsArgs['encode'])
   
    return(dwSize + dwFlags + dwFourCC + dwRGBBitCount + dwRBitMask \
           + dwGBitMask + dwBBitMask + dwABitMask)



grab_meshes.py

Spoiler: show
# -*- coding: utf-8 -*-

import os
import shutil
from phyre import extractMesh, extractDDS
import sys

ffxBaseDir=r'C:\SteamLibrary\steamapps\common\FINAL FANTASY FFX&FFX-2 HD Remaster\data\FFX_Data_VBF\ffx_data\gamedata\ps3data\chr'
ffx2BaseDir=r'C:\SteamLibrary\steamapps\common\FINAL FANTASY FFX&FFX-2 HD Remaster\data\FFX2_Data_VBF\ffx-2_data\gamedata\ps3data\chr'
baseDir=[ffxBaseDir, ffx2BaseDir]

types={'pc':'c', 'npc':'n', 'mon':'m', 'obj':'f', 'skl':'k', 'sum':'s', 'wep':'w'}
ffx0=[1,2]
tps=['pc','npc','mon','obj','skl','sum','wep']


for ffx in ffx0:
    for tp in tps:
       
        if ffx==1:
            gamestr = "FFX"
        else:
            gamestr = "FFX2"
       
        logFile = gamestr + "_" + tp + r"_log.txt"
           
        f=open(logFile, 'w')
        stdout0 = sys.stdout
        sys.stdout = f
        for i in range(1000):
            cs = types[tp] + '%03d' % i
            thisDir = os.path.join(baseDir[ffx-1], tp, cs)
            if not os.path.exists(thisDir):
                continue
            print("\n\n\n\n")
            print("=====================================================================")
            print("Found " + thisDir)
            daeFile=os.path.join(thisDir,'mdl','d3d11',cs+r'.dae.phyre')
            ddsFile=os.path.join(thisDir,'tex','d3d11',cs+r'.dds.phyre')
            dumpDir = os.path.join(gamestr, tp, cs)
            if os.path.exists(dumpDir):
                shutil.rmtree(dumpDir)
            os.makedirs(dumpDir)
            objFile=os.path.join(dumpDir, cs+r'.obj')
            ddsFile2=os.path.join(dumpDir, cs+r'.dds')
           
            try:
                extractMesh(daeFile, objFile)
            except Exception as e:
                print ("Failed:" + repr(e))
           
            if os.path.exists(ddsFile):
                print("\n\n\n")
                try:
                    extractDDS(ddsFile, ddsFile2)
                except Exception as e:
                    print("Failed: " + repr(e))
            else:
                print("\n\n\nDDS file not found. Skipping")
               
        f.close()
        sys.stdout = stdout0
        print("Done with %s %s" % (gamestr, tp))



test_extraction.py
Spoiler: show
# -*- coding: utf-8 -*-

import phyre, importlib, os
importlib.reload(phyre)

# 1 or 2
ffx=1

# pc, npc, mon, obj, skl, sum, or wep
tp = 'pc'

# model number (no leading zeros)
num = 106

ffxBaseDir=r'C:\SteamLibrary\steamapps\common\FINAL FANTASY FFX&FFX-2 HD Remaster\data\FFX_Data_VBF\ffx_data\gamedata\ps3data\chr'
ffx2BaseDir=r'C:\SteamLibrary\steamapps\common\FINAL FANTASY FFX&FFX-2 HD Remaster\data\FFX2_Data_VBF\ffx-2_data\gamedata\ps3data\chr'
baseDir=[ffxBaseDir, ffx2BaseDir]



types={'pc':'c', 'npc':'n', 'mon':'m', 'obj':'f', 'skl':'k', 'sum':'s', 'wep':'w'}

file=baseDir[ffx-1]
cs = types[tp] + '%03d' % num
meshFile = os.path.join(file, tp, cs,'mdl','d3d11', cs + r'.dae.phyre')
ddsFile = os.path.join(file, tp, cs, 'tex', 'd3d11', cs + r'.dds.phyre')

outFile = r'mytest.obj'
outFile2 = r'mytest.dds'
#outFile = None
phyre.extractMesh(meshFile,outFile, debug=False)
print("\n")
if os.path.isfile(ddsFile):
    phyre.extractDDS(ddsFile, outFile2)
else:
    print("DDS file not found. Skipping")



The thread and the files got deleted but i managed to save the post

Quote
Not sure if there's still any interest in getting out the models for this game on PC, but I made a python script that can extract both the character meshes as objs as well as the DDS textures. I haven't managed to figure out the exact format of these phyre files, so instead the script searches for the faces, vertices, UV maps, and normals by looking for certain markers. While this isn't absolutely foolproof, it's now robust enough to process every character mesh in FFX/FFX-2 without complaining. I haven't looked at every single mesh or texture to verify it, but for every file I have checked, it works correctly (including all the main character low and high poly meshes).

The main script itself is phyre.py. You'll need python 3 to run it (I ran it on 3.5). The basics to run it look like:

Code:
from phyre import extractMesh, extractDDS
extractMesh(r'/path/to/file.dae.phyre', r'outputfile.obj')
extractDDS(r'/path/to/file.dds.phyre', r'outputfile.dds')


That's it. There are some additional keyword arguments you can supply, but they are debugging tools for the most part. The only one that might be of interest is includeNormals for extractMesh(). By default it doesn't export the normals since they're unnecessary, but if you want them just add includeNormals=True to extractMesh().

I've also included my two test wrapper scripts. test_extraction.py is just a simple wrapper for extracting a single model. grab_meshes.py will extract all the models in the chr folder for both games. It'll create about 2GB of data and create some log files from the output, and it takes a couple minutes to run. For both scripts, you'll need to change ffxBaseDir and ffx2BaseDir to your install location. They'll both dump everything into the current working directory.

There are a number of files that it can't process, but these don't appear to have any actual mesh data in them anyway. For FFX the files are c051, k002, k004, k007, k014, k015, k017, k201, k212, k214, k302, k303, k403 and k999. For FFX-2 they're k015, k201, k303, k403, and k999. The script will also give a warning for a random NPC (n174 in FFX, n119 in FFX-2) about the UV maps being out of bounds, but it's for a single tri, and that's actually how it's defined in the .dae.phyre file. The script will still process the file with the warning.

Finally, some caveats. This script is almost certainly only gonna work on the PC port of FFX/FFX-2 HD Remaster. It will definitely fail for the PS3 files due to the difference in endianness, and I'd be shocked if it worked on the PS4 version. I made it specifically for character meshes, and I haven't tested it out on anything outside of chr/. Finally, this is good for extracting meshes only. You're still SOL if you want to modify the meshes for in-game use.


What FFX needs right now is a way to mod the files without having to rebuild the entire 20 gb vbf file every time, that takes way too long and makes testing almost impossible.

Maybe someone can make some sort of data injector

If the community can come together we could see some amazing stuff, like swapping out the character models, seeing has the PS2 models are in the game files

nex86

  • Fast newbie
  • *
  • Posts: 7
  • Karma: 0
    • View Profile
Re: Kickstarting Final Fantasy X Modding
« Reply #1 on: 2017-10-28 20:11:50 »
good luck, I don't think many people are still interested in modding this game, other than texture mods.

What FFX needs right now is a way to mod the files without having to rebuild the entire 20 gb vbf file every time, that takes way too long and makes testing almost impossible.

I agree. But its hard to do if you don't have the sourcecode of the game to tell the ressourcemanager to load the files outside of the vbf file.

« Last Edit: 2017-10-28 20:14:23 by nex86 »

Cyberman

  • No life
  • *
  • Posts: 1573
  • Karma: 8
    • View Profile
Re: Kickstarting Final Fantasy X Modding
« Reply #2 on: 2017-11-07 23:48:01 »
Hey Good too see people beating on old games, I started with an FF10 save game and well a decade ago I got busy. Really busy. So now I have a bit of time and I want to finish some things.

Enough about that:
  • Good fortune I wish you well on your endeavor, the main thing is you aren't just doing it for everyone else, you are doing it because you want to.
  • The next thing to remember some of the "old" folks might be able to help you (doubtful but hey you never know).
  • Most importantly have fun with what you are doing, if you aren't enjoying yourself and feel obligated or discouraged because someone else said "don't" bother remember item 1. LOL

So the VBF file, if you know it's format and structure you likely can slice and dice it affectively. IE you can modify the file with wild abandon if you have a good idea of the format. However before wacking it to bits some things to consider:
  • Make a copy
  • see the first suggestion.
  • Work on the copy not the original, you can rename files at no FS penalty (IE rename the original to *.VBF.org or something) then rename the copy to *.VBF).
  • You want to make a tool for editing the VBF (as opposed to rebuilding it) that's my suggesttion a High Level method to start with.
Knowing the VBF structure etc. a simplified program to do the editing is probably necessary.
So lets say you wish to change X in the structure of the file X is substructure Y and part of Z which is a subsection of A.
The first thing your tool should check is 1 is it larger than the original X and if so does it change the VBF structure.
Depending on the VBF structure (IE it has clusters of file pointers internal to it, that's how I would do it anyhow) you may be able to just change the internal pointer and append data to the end of the VBF.
Anyhow that's all for now. I worked on an ISO generating tool a few years (centuries) back for UDF and ISO9660 format. That was, annoying but that's not your problem (it WAS mine). So your tool needs to treat what you call a FILE and a VFS (Virtual File System) and monkey with as necessary to change it to function. Depending on how FF10 loads data from the GIANT file (hey I should have that on my PS3 ... hmmm no must finish other problems, I mean projects).

Good fortune and enjoy!

Cyb