[FF7] PowerShell Scripted PC <-> Android PSX Save Sychronizer (via FTP)

[n1ckn4m3]

EDIT:  Re-titled thread, this is NOT for the just-released Android port of FFVII, it's for synchronizing between FPSE/epsxe and the PC version of FFVII.  if you're looking for the final script, it's here:  http://forums.qhimm.com/index.php?topic=17047.msg242571#msg242571 -- if you want to see me bumble through learning little things about the FFVII save format, read on!

Hey everyone, long time lurker, finally registered an account to ask this question:

I've got a GPD XD, and I use FPSE on it to play FFVII.  I also use the Steam copy of FFVII as well.  Black Chocobo is currently capable of switching between these file formats with no problems.

What I did was write a PowerShell script that connects to my GPD XD and checks to see whether the .MCD file on it for FFVII is newer than the local FF7 save00.ff7 file from the Steam copy, and depending on which is newer, it attempts to convert the file formats and then replace the older one with the newer one.

It's kind of a scripted way to effectively rsync my two save files, including the conversion.

The problem is that since Black Chocobo is a GUI app, I couldn't script using it to convert my ff7_pc.mcd Memory Card format file into a PC save00.ff7 file.  There is a tool that works on the pre-cloud enabled version of FFVII, and I was able to pipe input into it in order to automate the conversion -- but it doesn't generate the metadata.xml file, so FFVII's PC version just eats the save file and that's that.

So, I have two questions:

1)  Is there any current command line save converter for FF7 that supports the latest release including the metadata.xml editing

OR

2)  Is there a tool that will generate a working metadata.xml file if you point it at an a working save00.ff7 file that I could call from a script?  Alternately, is the Metadata.xml just the product of some set of md5sums that I could programmatically do with PowerShell (which I'm using for the rest of the script)?

I got so close before realizing that the save converter tool I was using wouldn't work because of the lack of metadata.xml -- everything else about the script works, it connects to the GPD XD, checks the time/date stamp, moves the files around, and even completes the conversion process -- but the GPD XD -> PC process can't yet be automated because of the Metadata.xml requirement.

Any help would be appreciated!

EDIT:  Since I'm not one to leave well enough alone, I did look and find that Black Chocobo uses the ff7tk libraries, which include the ability to convert between save file formats.  I'm not much of a programmer, more of a scripter -- but I'm mucking about getting the demo app to compile to see if I can cobble together some kind of converter in case one doesn't already exist.  I'm not extremely optimistic that I'll be successful, but we'll see.

2nd EDIT:  From what I can tell, the Metadata.XML is just looking for the md5sum of the save file with the savegame's unique ID appended to it.  Trouble is, I don't quite know precisely where to append my ID.  I stumbled across this post talking about how the Metadata.xml works and how it is generated: 

Where is the UserId? Look at the path of the saved game:
C:\Users\{user-name}\Documents\Square Enix\FINAL FANTASY VII\user_123456\save00.ff7
In that case, that would be 123456.

So, append the said UserId at the end of the saved game, and MD5-sum it. Voila!

But just cannot for the life of me figure out where in the file to append the unique ID.  I read the documentation about the ff7 save file format and found what I thought was the end of the file, but I can't generate the md5sum that exists already in my (working) metadata.xml.  I think if I could do that, I could then calculate the md5sum and stick in where it needed to go in the existing metadata.xml, but without knowing where precisely to put it (and whether to insert/overwrite), I'm kind of stuck.  So, I guess I now have 3 questions -- but any one of them being answered would do

Thanks!

3rd EDIT:  I'm a dumdum, I just need to append it to the actual end of the save file itself.  I think I can script the rest of this from here, will update if so and remove post.

Final EDIT:  Yep, I had just misunderstood.  Made a file with my unique ID in it, added some code to concatenate it and the save file, generated the md5 and then wrote it to the .XML.  Problem solved, one click synchronization between my gaming desktop and my portable emulation rig achieved.  If anyone wants the Powershell code, let me know.  It's ugly, but functional -- would work for any Android device running an FTP server and any emulator that supports .MCD format.  You likely need to be rooted though for the FTP server to have write access.

[sithlord48]

1.  you would have to write a cli front end for FF7Save (from ff7tk ) only a few functions would need to be exposed for the functionality your asking for. (save and open would be all you really need)
2. There is a metadata converted builtin to black chocobo if you use file_> create cloud folder you make a new cloud folder from any file BC can read.
3.It works like this .. take the data from the save then append in raw hex the user ID (in ff7tk i add it to the bytearray) then md5sum that.

4. nice to see you have it working . perhaps you should post your script here so others can learn from it.

[n1ckn4m3]

Thanks for the reply sithlord!  Here's my (ugly, but working) code.  Now that it's functional, it could use a lot of cleanup (which I'll be working through), but in case anyone wants to do something similarly, here's how I did it.  I can't stress enough, this is more POC than anything else.  It needs a whole bunch of clean-up and refactoring to make it 'nice'.  It's pretty much the sledgehammer approach at present.

Assumptions:
*  User has PSFTP Module installed from here: https://gallery.technet.microsoft.com/scriptcenter/PowerShell-FTP-Client-db6fe0cb
*  User has FF7Save.exe from here: http://www.zophar.net/utilities/genutil/ff7-savegame-converter.html
*  User has device that they want to play FF7 on that is accessible via FTP and uses .MCR format (in this case, a GPD XD with FPSE)
*  I statically assigned a DHCP address to my GPD XD and added it to my hosts file as "gpdxd" to make it easier, you can do it without that but you'll need to change the IP every time, which defeats the purpose of a one-click sync imo.
*  I'm using the first save slot.  Values in the script and a few other places would need to change if I was using a different slot.
*  You have created a file called md5salt.bin that contains only the hex representation of your User ID (the number after user_ in the profile folder location)
*  I'm using a folder I created called FF7 Save Sync that has all of the utilities, the .ps1 file, all the .cmd files and .txt files in it.  I picked the existing Square Enix folder because I knew I'd have write access to that 100% of the time without elevation.
*  You'll want to review the top variables and update to reflect your environment.  My main OS is on a smaller SSD so my My Documents libraries and such are migrated to another drive.  Since this is behavior outside the norm, you'll want to make special note when looking at the variables.

Bugs:
*  Since there's no way to change the modified date/time over FTP, there's no way I have figured to realistically have the timestamp replicated on the remote side.  As such, in any case that the local file is newer, the remote file will have the timestamp of the time the sync occurred, not the timestamp of the save file.  I'm looking into ways to fix this, but since I'm using FTP I don't think it's possible at present.

Sync-FF7Save.ps1

Code: [Select]

replaced, see below
local.cmd

Code: [Select]

replaced, see below
remote.cmd

Code: [Select]

replaced, see below
local.txt

Code: [Select]

replaced, see below
remote.txt

Code: [Select]

replaced, see below
This works in both directions, applies the userID to the end of the save file, MD5 sums it, and adds that to the metadata.XML.  Definitely open to constructive criticism, but do keep in mind this was definitely the first run proof of concept, I will definitely be streamlining things and making things cleaner.  I'm not pretending for a minute I'm a powershell guru, just functional enough to cobble this together

[n1ckn4m3]

Here's a cleaned up version of the script, automating the 'md5salt.bin' generation, cleaning variables up notably, and adding some error handling to the initial FTP connection.  Better, still pretty ugly and hacky, but on its way.  Will be adding the ability to sync different save slots as the next enhancement after some testing.

Code: [Select]

replaced with final version

[n1ckn4m3]

Once more with some bigger improvements:

* Profile ID extrapolated from FF7 Save Directory automatically
* Can change the save slot being synchronized, seems to work in my tests -- can't guarantee other save files in your .mcd will survive
* md5salt.bin Automatically generated at runtime from Profile ID
* local/remote.txt/.cmd generated at runtime (effectively making the script only require ff7save.exe + the icon dat and the PSFTP module to succeed), and only as needed
* Lots more here and there cleanup, a couple of minor bug fixes.

Again, who knows if this will be useful to anyone else, but it'd be a shame to have done all this work and not post it in case someone else has a similar need.

Known issues:
*  Still doesn't set the modified date on the uploaded FTP file, looks like that's a limitation of the PSFTP module.  I'm investigating other ways to fix that.

Code: [Select]

<#
.SYNOPSIS
    FF7 PC to Gaming Device Synchronizer
.DESCRIPTION
    Connects via FTP to a compatible gaming device that utilizes a common format for memory cards, compares the age of the
      targeted save file versus the age of the save file in the local save path and then synchronizes the two save files,
      including the conversion between file types and the modification of the metadata.xml file.
.NOTES
    Author: n1ckn4m3 - [email protected]
.DEPENDENCIES
    PSFTP Module by Michal Gajda: https://gallery.technet.microsoft.com/scriptcenter/PowerShell-FTP-Client-db6fe0cb
    FF7Save Converter: http://www.zophar.net/utilities/genutil/ff7-savegame-converter.html
.USAGE
    * Download and Import the PSFTP Module and ensure that Import-Module PSFTP successfully completes
    * Create a working directory for the script and copy the contents of the FF7Save.zip to that directory, as well as this script
    ** Make sure that the user account running the script has write access to the working directory you've chosen, else you'll need to elevate privileges for the script to work
    * Update the script variables to be accurate for your environment.  Key ones include $WorkingDir, $SaveSlot, $AndroidSave, $AndroidFTP, $AndroidDir, $UserName, and $Password
    * MAKE BACKUP COPIES OF YOUR SAVE FILES FIRST.  The script does this, but considering it replaces files, better safe than sorry.  I accept no liability if you overwrite
        saves or nuke a 400 hour perfect game, or if your HDD turns into a megalomanical robot trying to take over the world.  In short, use at your own risk.  Provided free with
        no warranty of fitness for any purpose, implied or otherwise.
#> 

# I tried doing this without this module and it was hacky, ugly, and stupid.  Thanks, guy who wrote this module.
Import-Module PSFTP

# Working Directory.  Change this to whatever you'd like, just make sure it has the FF7 Save converter in it, that it exists, and that it's writable by your user account.
$WorkingDir = ([environment]::getfolderpath(“mydocuments”)) + "\Square Enix\FF7 Save Sync\"

# General Script Variables
$SaveSlot = "00" # Use the filename's save slot ID, e.g., for save00.ff7, use 00 - NOT 1 - the script relies on this
$SaveDirs = Get-ChildItem ([environment]::getfolderpath(“mydocuments”))"Square Enix\Final Fantasy VII Steam\"
$LocalSave = $SaveDirs[1].FullName + "\save" + $SaveSlot + ".ff7"
$LocalBackup = $LocalSave + ".backup"
$WorkingLocalSave = $WorkingDir + "save" + $SaveSlot + ".ff7"
$LocalSaveFile = "save"+$SaveSlot+".ff7"

# Metadata.XML Specific Variables
$MD5Salt = $WorkingDir + "md5salt.bin"
$MetadataXML = $SaveDirs[1].FullName + "\Metadata.xml"
$MetadataBackup = $MetadataXML + ".backup"
$WorkingMetadataXML = $WorkingDir + "Metadata.xml"
$WorkingLocalMD5 = $WorkingDir + "save" + $SaveSlot + ".ff7.md5" 

# Metadata.XML Lookup - Save Slot to XML Line ID
If ($SaveSlot -eq "00"){ $XMLLine = 18 ; $SaveID = 1}
If ($SaveSlot -eq "01"){ $XMLLine = 36 ; $SaveID = 2}
If ($SaveSlot -eq "02"){ $XMLLine = 54 ; $SaveID = 3}
If ($SaveSlot -eq "03"){ $XMLLine = 72 ; $SaveID = 4}
If ($SaveSlot -eq "04"){ $XMLLine = 90 ; $SaveID = 5}
If ($SaveSlot -eq "05"){ $XMLLine = 108 ; $SaveID = 6}
If ($SaveSlot -eq "06"){ $XMLLine = 126 ; $SaveID = 7}
If ($SaveSlot -eq "07"){ $XMLLine = 144 ; $SaveID = 8}
If ($SaveSlot -eq "08"){ $XMLLine = 162 ; $SaveID = 9}
If ($SaveSlot -eq "09"){ $XMLLine = 180 ; $SaveID = 10}

# Android .MCD file locations and names, update to reflect yours.  Backup should be in the same directory to streamline but I suppose it would be fine anywhere.
$AndroidDir = "/mnt/external_sd/MEMCARDS/"
$AndroidSaveFile = "ff7_pc.mcd"
$AndroidSave = $AndroidDir + $AndroidSaveFile
$AndroidBackup = $AndroidDir + $AndroidSaveFile + ".backup"
$WorkingAndroidSave = $WorkingDir + $AndroidSaveFile
$WorkingAndroidBackup = $WorkingDir + $AndroidSaveFile + ".backup"

#FTP Support Variables, update to reflect your specifics.
$AndroidFTP="ftp://gpdxd:31337"
$UserName = "<username>"
$Password = "android" | ConvertTo-SecureString -asPlainText -Force
$Credential = New-Object System.Management.Automation.PSCredential($UserName,$Password)

# Set Save Sync Directory
Set-Location $WorkingDir

function Cleanup () {
  If (Test-Path $WorkingLocalSave){Remove-Item $WorkingLocalSave}
  If (Test-Path $WorkingAndroidSave){Remove-Item $WorkingAndroidSave}
  If (Test-Path $WorkingAndroidBackup){Remove-Item $WorkingAndroidBackup}
  If (Test-Path $WorkingMetadataXML){Remove-Item $WorkingMetadataXML}
  If (Test-Path $WorkingDir\local.txt){Remove-Item $WorkingDir\local.txt}
  If (Test-Path $WorkingDir\remote.txt){Remove-Item $WorkingDir\remote.txt}
  If (Test-Path $WorkingDir\local.cmd){Remove-Item $WorkingDir\local.cmd}
  If (Test-Path $WorkingDir\remote.cmd){Remove-Item $WorkingDir\remote.cmd}
}

# Delete working copies of saves if they exist due to a script failure or bad cleanup
Cleanup

# Copy Local Save to Save Sync Directory
Copy-Item $LocalSave $WorkingLocalSave -ErrorAction SilentlyContinue

#Initiate Connection to Android Device, break script if we can't connect.
Try {
  Set-FTPConnection -Credentials $Credential -Server $AndroidFTP -Session AndroidFF7SyncSession -UsePassive -ErrorAction Stop | Out-Null
}

Catch {
  Write-Host "No response from FTP server, check variables and network connectivity.  Aborting synchronization."
  # Clean up after ourselves, even if stuff goes south pretty early.
  Cleanup
  Break
}

# Since we know we can connect now, let's finish setting up the FTP session.
$FTPSession = Get-FTPConnection -Session AndroidFF7SyncSession

# Download Remote Save to Save Sync Directory and create working copy of backup
Get-FTPItem -Session $FTPSession -Path $AndroidSave -LocalPath $WorkingAndroidSave | Out-Null
Copy-Item $WorkingAndroidSave $WorkingAndroidBackup

# Check Timestamp of Remote Save vs. Local Save
$RemoteFullFile = $AndroidDir + $AndroidSaveFile
$RemoteTimestamp = [datetime](Get-FTPChildItem -Session $FTPSession -Path $RemoteFullFile).ModifiedDate
$LocalTimestamp = [datetime](Get-ItemProperty -Path $WorkingLocalSave -Name LastWriteTime).lastwritetime

# If Remote Save is newer, back up local save and replace
If ($RemoteTimestamp -gt $LocalTimestamp) {
  # Create remote.txt and .cmd to launch and control FF7 Converter
  $RemoteTXT="1",$AndroidSaveFile,$SaveID,"4",$LocalSaveFile,$SaveID,"1"
  $RemoteTXT | Set-Content -Path $WorkingDir\remote.txt
  $RemoteCMD = "ff7save < remote.txt"
  $RemoteCMD | Set-Content -Path $WorkingDir\remote.cmd

  # Remove the local backup, then convert the new save and copy it.
  If (Test-Path $LocalBackup){Remove-Item $LocalBackup}
  Copy-Item $WorkingLocalSave $LocalBackup -ErrorAction SilentlyContinue
  CMD /C "remote.cmd" | Out-Null # Convert .MCD format FF7 save to .FF7 save format save
  Remove-Item $LocalSave -ErrorAction SilentlyContinue
  Copy-Item $WorkingLocalSave $LocalSave
  Write-Host "Remote save was newer.  If one existed, local save has been backed up and replaced."
 
  # Generate MD5 "salt" (not really a salt, but whatever) for Metadata.XML generation later
  $ProfileID = $SaveDirs[1].Name
  $ProfileID = $ProfileID.Trim("user_")
  $ProfileIDArray = $ProfileID.ToCharArray();
  $Bytes = New-Object Byte[] 7
  $HEXArray=[int[]][char[]]$ProfileIDArray
  $Loop=0
  ForEach ($HEXLine in $HEXArray){
    $Bytes[$Loop]=$HEXLine
    $Loop = $Loop + 1
  }
  [IO.File]::WriteAllBytes($MD5Salt,$Bytes)

  # Metadata.XML support code
  $COPYCMD = "copy /b save" + $SaveSlot + ".ff7+md5salt.bin save" + $SaveSlot + ".ff7.md5"
  CMD /C $COPYCMD | Out-Null # Concatenate existing save file w/ md5 "salt" to allow signature generation
  $HashPath = "save" + $SaveSlot + ".ff7.md5"
  $MD5Hash = (Get-FileHash -Path $HashPath -Algorithm md5).Hash
  $PreppedHash = "    <signature>"+$MD5Hash.ToLower()+"</signature>"
  $MetadataContent = Get-Content -Path $MetadataXML
  $MetadataContent[$XMLLine] = $PreppedHash
  Copy-Item $MetadataXML $MetadataBackup
  $MetadataContent | Set-Content -Path $MetadataXML

  # Re-set proper modified time, as downloading it changed it to 'right now'.
  $ModifiedSave = Get-Item $LocalSave
  $ModifiedSave.LastWriteTime = $RemoteTimestamp
}

# If Local save is newer, back up remote save and replace
If ($LocalTimestamp -gt $RemoteTimestamp) {
  # Create local.txt and .cmd to launch and control FF7 Converter
  $LocalTXT="4",$LocalSaveFile,$SaveID,"1",$AndroidSaveFile,$SaveID,"1"
  $LocalTXT | Set-Content -Path $WorkingDir\local.txt
  $LocalCMD = "ff7save < local.txt"
  $LocalCMD | Set-Content -Path $WorkingDir\local.cmd

  # Create/replace backup save and convert new save to proper format, then upload
  Add-FTPItem -Session $FTPSession -Path $AndroidDir -LocalPath $WorkingAndroidBackup -Overwrite | Out-Null
  CMD /C "local.cmd" | Out-Null # Convert .ff7 format FF7 save to .MCD save format save
  Add-FTPItem -Session $FTPSession -Path $AndroidDir -LocalPath $WorkingAndroidSave -Overwrite | Out-Null
  Write-Host "Local save was newer.  Remote save has been backed up and replaced."
}

If ($LocalTimestamp -eq $RemoteTimestamp) {
  Write-Host "Timestamps the same, sync complete."
}

# Clean up after ourselves.
Cleanup