Before hex editing a scene file, you need to decompress it with scene reader (link is at the bottom of the wiki page) and recompress it when you've finished the editing (protip: close the hex editor window before recompressing with scene reader). Hojo and Proud Clod do the de- and recompressing themselves.
As for editing formations, I'd be careful; some enemies don't like being in certain positions. Not much research has been done with formations AFAIK, so I can't really give you any help.
gjoerulv gives some advice on creating enemies
here (post was made before Proud Clod came out). You may have to do some field editing as well; unfortunately that isn't my area of expertise. For the PC version, a program called meteor will do the job (you should be able to find it on the forums).
Unfortunately, I'm not sure what program would be best for the PSX version; I know that 7mimic can do some editing to field files, but I don't know whether that includes encounter and script edits.