Oh I'd say it's quite the oposite, at least we see some of the Gerudo preforming magic. Hylians and Sheikah are just said to be gifted with magic, and I'm sure they are, but have we ever seen Sheikah magic? Have we seen Hylian magic? I'm not talking about magic items, I'm talking about wizzardy spells and the like.

Anyway, I think it's possible that the collossus represents Din, but we may never know it could also be a special deity to the Gerudo, someone representing the strength and independance of a woman, similar to Lilith.
And just because the Gorons worship Din that does not mean that the Gerudos can not worship her.
Think about the Sheikah, their ultimat goal is truth and wisdom. Wouldn't you excpect them to worship Nayru, but wait the Zoras allready do that.
The Kokiri, Zora and Gorons worshipping one of the golden goddesses is by the way far from proven.

What we can be relativly sure about is that the Desert collossus and the spirit temple are linked to the Gerudo.
The treasures found there carry the Gerudo insignia and the statue clearly features Gerudo fashion and jewelery.

On a side note: The Goddesse depictions in Arbiters ground most likley represent the same goddesse, since they feature the same snake attributes.
It should be noted that the statues in the Arbiters Ground carry flames in their hands. Another hint towards Din? Who knows...

__________________
(>' . ')>