Can you have a sprite appear above the background but below the window ?

On the GBC, yes. You might can do it on the other GBs as well but only with creative use of the LYC interrupt.