Hi,
I don't think that you need xfce. As you start midori in full screen , you don't need window manager anymore.
I think it's possible to start an X server with midori at boot time.
On Ubuntu it's possible to start a user session at boot time, if the user has an xinitrc script with just midori in it, I think it can be possible to have just midori on the screen.
This is just ideas not tested, but you know : "Be cool, Linux can do it! "