A) Link doesn't take the Ocarina from Hyrule, and nothing he does is erased. That's a bit of a miss understanding on the Double Timeline.
B) The Oracle Series is, most likely, mere decades after
AlttP and yet Ganon manages to return.
C)
AoL is 3 years maximum after
LoZ. Still, Ganon's on a comeback.
D) Ganon doesn't want to "Access the Triforce", he has the ToP, the ToC is in eight pecies. He's just got the same world domination dream he's always had.
That being said, its a good thought. Ganon could be the cause of all the trouble in
TP, and it could be enforced in the same way as in
OOT. Of course, Nintendo has said the Ganon will be in this game in a different way then any game before. But that could mean anything...