There could be a scenario where a good person must break into the temple of time to save the life of another. Link is needed to pull the Master Sword, so he would have to "break in" with that good person. A hostage situation, or the stones/ocarina being somehow unattainable no matter what and ganondorf being unstoppable in wreaking havoc at the same time, are reasons enough.

But man is this random...

__________________
Time Travel | The Child Timeline | Fun Stuff