*update*
okay I figured it out and I am currently recutting all the layers in a much smoother fashion. It may sound tedious but it is worth it.
here is an example of the drastic difference between smooth and unsmooth layers
http://www.mediafire.com/?cwm5kmz08tcf5xx__________________________________________________
I am going to be using a few pictures to help illustrate what I am trying to achieve here. I have already made some fairly decent progress after several experimental failures. Basically what I am trying to do is cut out the layers of my field upscale project much smoother than the originals. To help you understand what I mean take a look at these two images...
smoother
original
the reason why this is important is because when a model goes in front or behind a layer it will display the edge, and if that edge is "block"y then it is very unattractive, with smoothing it feels much more natural.
I am basically writing this thread to get both help and opinions on how I should do this ultimately. After a lot of testing, I came up with a very nice automated method for smoothing. I know I can smooth only the edges I need manually, but in an effort to save a ton of time and work I have tried to go automatic but this requires the entire layer to be smoothed. Fortunately I found a way to do this without making the layer too small or too big in comparison to the others. After working with different possibilities these three layers are what I came up with
Now I can assure you, even though you can't tell just by looking at them, all 3 of these layers are perfectly uniform to each other; that is ,when put together, they form the complete picture without any holes or overlaps. However when I actually implement them in game there is a different story to tell. This is where I am stuck. This is what is happening in game...
Important: This happens even if I leave the very bottom layer full, with no cuts (unlike the example above)
I have circled the problem areas. This is because the once 90 degree edges, as well as some other areas, are now smoothed and curved leaving a small area of open space which was once filled in that particular layer. Please keep in mind this open space is not there if you combine all 3 layers in Photoshop, only in game.
Now I think I know why this is happening, correct me if I am wrong but it seems in some way or another the game has predefined coordinates for where each layer should be. So for example if layer 3 is meant to be on top of all the other layers, then somewhere it is programed that layer 3 is on top and that it has exactly x height, x width, x dimensions. What I can surmise is that even if there is something there on the bottom layer to compensate for it, if a given area doesn't show up in the layer its defined to be in, it'll show up as blank in game, thus causing the small black areas. I know this probably isn't exactly right from a technical stand point but I hope I am at least on the right track.
It used to be a lot worse, for a while I had this weird huge line and if you walked in front of it, it would go on top of the character, so I have made a lot of progress but I seem to be stuck here. These little black areas are very tiny but very noticeable. I already have everything else figured out if I can just fix this somehow I can begin cutting smoother more attractive layers.
If there is no way to solve this then I will just have to work manually and it will take much longer, but I was wondering if it would be possible to modify Palmer or some other program, to set new coordinates and dimensions based on the layers being input and not from the original layers. Or if anyone knows other fixes that maybe I haven't thought of yet. I really need some help on this one guys. Thank you for your time!