I'm pretty sure I got stuck in the Tower of the Gods a couple of places, but other than that I don't remember finding any of the dungeons hard.

I'd only really condsider Dragon Roost Island, Forest Haven, Tower of the Gods, Earth Temple and Wind Temple dungeons though.