When the gimmick is along the lines of silly voice/clothes etc it puts me right off. I tend to take the view that if a comedian has to dress up in order to make you laugh then his material probably isn't that funny in the first place. it took me quite a while to get past Harry Hill's clothes and annoying delivery and realise that some of his stuff was good.
I'm not talking about when a comedian adopts a specific character (although some of those can be irritating as well). Neither do I include catch phrases in this, as sometimes the humour of the catchphrase lies in how the comedian works his way around to it. I'm talking more about the in your face stuff where the implication is "observe the way I look/dress/sound - I must be funny".
If the performer doesn't have enough confidence to let his/her material go unsupported by a gimick then maybe its not good enough.
No-one's yet beaten Tommy Cooper's impressive gimmick of dying halfway through his act so dramatically that the audience thought it was part of the act, and laughed and applauded.
However if Joe Pasquale wants to try and upstage this stage death in some way, I'm sure we'll all be right behind him...