My opinion is this: The Gerudo became the Twili.
The "flashback" cutscenes, particularly of Ganondorf being sealed in the Twilight Realm, happened a LONG time ago, relative to the current game time.
If you read the LttP backstory (printed in the instruction manual), it states that Ganondorf Dragmire (did you know he had a last name? haha) and his band of Enchanted Thieves found an entrance to the Golden Land by accident, and Ganondorf fought with his own people in order to reach the Triforce; his wish transformed the Golden Land into the Dark World, a land of twilight.
Ganondorf then amassed an army and attacked Hyrule, which began the Imprisoning War. Ganondorf was eventually subdued by the Hylian Knights and sealed in the Sacred Realm, along with his remaining army, by the Sages. The cutscene shown in
TP is, I think, a portrayal of this original sealing, done at the end of the Imprisoning War.
The Gerudo (i.e. Enchanted Thieves) still loyal to Ganondorf would have been sealed away at the same time. But Ganon was weakened and went into hiding in the Dark World (i.e. Twilight Realm), leaving the Gerudo to rule themselves. They eventually became the Twili, a tribe of Twilight-dwelling magicians.
I haven't yet worked out all the kinks as to whether this is consistent with events in
OoT, though. Perhaps the Gerudo are not actually the original band of thieves, but the new recruits used in the Imprisoning War? Perhaps the Imprisoning War happened during the seven years of Link's entrapment in
OoT, and Link returned to end it. In which case the same sealing-away by the sages has been portrayed in two different ways.
Or perhaps the Imprisoning War happened long before
OoT, and
OoT represents simply another time that Ganon has returned. Overall, either Ganon has a habit of returning over the centuries, or there are several games which portray the same sequence of events from different perspectives.