The Take That star confirmed last year that the 2013 show was his last.
Cowell told the Daily Mirror: "The truth is, and I am not being rude, I wouldn't hire him to run my record label.
"Essentially that is what you are doing on The X Factor. You have got to be wide-eyed and Cheryl (Cole), one of the main reasons she is coming back is she is good at that.
"She is the one who really spotted Cher Lloyd."
Asked if he and Barlow could work on the same panel, he said: "Gary was seen as the head judge and so am I.
"It is not so much what I say on the show as what I can do. I do believe I am the best person at spotting a potential star and making them work well on the show."
But he praised Louis Walsh, who previously said he was bowing out of the show and then changed his mind, telling the newspaper that he deserved a spot on the panel. He said he would like a singer, rather than a record executive, to take the last judging spot.
Cowell also defended lowering The X Factor's minimum age to 14, saying: "Can you imagine saying to Gary Barlow at 16 'You're too young to be a musician'? Give me a break."