The Clock Town shooting gallery was always more difficult for me. Mostly because, with the Swamp one, you have second chances to hit the targets that you miss. With the town one, not only are you unlikely to get a second chance, but you're also likely to hit a blue octorok which pretty much destroys your chance of getting a good score.
I don't think I've ever failed the swamp one, now that I think about it...
On another note, I'd say the hardest archery game in all of Zelda is actually the
PH one on Molida Island. I remember that taking hours for me.