I think that the Poe Sisters are the ghosts of the inhabitants of the temple.

I'm not sure about the story behind it. Perhaps it was just a really old mansion that belonged to a rich family?