Okay, here's my idea: the Forest Temple was originally a very huge mansion, almost like a fortress, and was high up on a hill way
before the Lost Woods existed. It was probably a place of evil energy, and as one of the Guardians of Hyrule, the Deku Tree appeared in order to wipe out that evil. He did so by growing a large forest, the Kokiri Forest and the Lost Woods, which eventually covered the whole hill and oppressed the dark energy from the mansion. That place, dark and gloomy, became the Forest Temple and its entrance was broken; even more, the Lost Woods were to guarantee that no-one who wasn't supposed to would find the Temple.
Okay, that's a quick story I came up with and imagine to be accurate enough. Because in the Temple, there are paintings showing the mansion scene I described before, which kind of hints that this scenery is related to this place:
Granted - it could be Ganon's Castle, too. But the paintings don't disappear with Phantom Ganon and it's probably safe to assume that they have been there longer than PG, especially since you come near the Temple as a kid already.