The first time I played
OoT, the entire castle town enviorment scared the living crap outta me, at first I didn't notice them, so when they screamed I nearly fell out of my chair. Very creepy.
Once you get used the sound of their scream, and learn how to avoid them, they aren't too bad.
As for the ones that scared me the most, I'd say
OoT, to me it looked like their face was half chopped off.