To note Ganon, it did explain earlier in the game why the people could not see the "diamond around Hyrule Castle". In fact, the people are completely oblivious to anything that involves the "twilight realm".