You can't have modified the scene.bin too much for the look-up table to not work correctly. That's not the way it works. Info on the scene.bin look up table lacking in explanation on the wiki. I'll remedy that shortly. Here's how the whole thing works:
Each of the 256 "scenes" in the scene.bin contain four formations. The entire uncompressed scene is 7808 bytes (or something like that). When the individual scenes are compiled, they are compressed and sequentially stored in chunks of 8192 bytes.
Each of these chunks begins with a 32 byte header stating where the compressed scene is located within that chunk. So the first scene would start at 0x0000 within that chunk and the next might start at 0x01F2, the third at 0x0342, etc. The chunk ends when it gets to be 8192 bytes in size or contains 16 scenes.
The scene look-up table in the KERNEL.BIN indicates the first scene in a particular chunk. So the first chunk will always start with scene 0, the second chunk starts with scene C, etc. The game checks this when a formation is requested. From the formation requested it can get the particular scene that the data for said formation is in and it can easily find the scene by checking the SLUT (Scene Look-Up Table, SLUT was too awesome an acronym to NOT use
). If the game requests scene 15 it will find the last chunk whose first scene is less than that and search that compressed block.
ex.
Formation 134h requested (I don't know what that is, it's just an example). The scene that's contained in is ( 134h >> 2 ) = 4Dh. The SLUT in the example KERNEL.BIN I have starts like this:
00 09 10 16 1A 25 2C 34 3C 49 52 59 60 66 6A 72
It will find that the 49 is the greatest number in that list that is lower than 4D so it will search through chunk 0A of the scene.bin to find this formation. That happens to be the fifth scene in the chunk (4D - 49 = 5) so it will use the formation out of that scene. However, if that scene isn't really 4D (meaning the SLUT was off) it will look like the battle is out of place.
In conclusion (tl;dr for you more text-y types), based on the way the scenes are structured you CANNOT have done anything to them to make them suddenly not work. The scene.bin can be as small as 128K in size or as large as 512K in size (which is practically impossible). If it's in between those sizes (which it has to be) then you don't "have too much data".
There are at least three things that could be wrong:
1. Your SLUT is off and PrC is incorrectly reporting it as correct. Many people have used this feature and no one has recently said it's wrong, but that doesn't mean it's working in all cases. You can double-check it "manually" with a hex editor.
2. You screwed up something in the formation data and added an enemy you didn't mean to. This is likely the case if the background in the battle is what you want, but the enemy is out of place.
3. Your flevel is requesting the wrong formations.
I need more info to determine which one is the case. What does your SLUT look like?