I don't think Link was a native of Ordon, not only because he doesn't physically resemble anyone there. I wish the game had given him more development of his backstory like in
OoT, but in
TP nearly everyone but Midna gets shafted for backstory development.
Also, small, isolated settlements like Ordon tend have a very clear sense of "who's one of us" and "who's not one of us," especially when the population is small enough that everyone knows everyone else, and can trace their ancestry back generations. No mention is ever given of Link's family, so I can speculate that he may have been abandoned there. If so, he would no have no family connections or history, further emphasizing his "apartness." And as Sirius B pointed out way back, his house is not part of the village itself, but separate. I wonder if Link built his house himself, meaning that he set himself apart from everyone else, or if the villagers built his house there, meaning that they set Link apart from themselves.