I personally always found some of the puzzles annoying. As silly as it sounds, it is true. In TP, it was always those stupid statues to get the sword. 8I I always messed up at the very end of it and then I was like, "AW MAN!". There's some other thing about the series that has annoyed me, but it's still great as a whole. :)

Despite all of the memes etc, I do not find Navi or Tatl annoying at all.
Octoroks are pretty annoying however. Just leave me alone, please.
Oh and Water Temple in Skyward Sword.

In OOT the noises that the spiders make. Especially if you can't find the damn thing. Then the room is just a never ending chorus of 'ktcchtchtchtchtchtch'.

The wallmasters in pretty much all of the gba ones, how they could just fly through and over obstacles. You could be almost totally stuck and one of them could just be like, "Don't worry little man...you'll be safe at the entrance."

Oh and like-likes especially if the process of getting a new shield is annoying and/or time consuming.

Majora's Mask. I loved OoT, but I started playing MM and once it told me there was a time limit I didn't even touch the game again. I don't like time limits, even though I know you can just reset it back to the first day it still feels like too much pressure :(

Haha, yes. I hate time limits in games because it gets me so irritable.
I used to get so angry when I first got the game as a child, haha. Now the problem is gone as I progressed.

The Evil trains in Zelda: Spirit Tracks. Such a pain.

In the Twilight Princess there was this one sequence where you had to mimic the wolf howls and I just couldn't get it right. Thus it made me quit all together and now I don't even know where the game is.