Moffat said that Matt Smith, the former Doctor who became something of a sex symbol, and new incumbent Peter Capaldi were attractive, but only in an "odd way".
He told Radio Times magazine that one of the reasons that he picked Capaldi was that The Thick Of It star was odd-looking, while Smith is the "strangest looking man" from a certain angle.
"There is something about Peter's demeanour, his eyes, his attitude - he's tremendously bright and that comes out on screen.
"When you choose a Doctor, you want somebody who is utterly compelling, attractive in a very odd way.
"None of the Doctors are conventionally attractive, but they're all arresting. Handsome men don't quite suit."
He added: "Matt Smith's a young, good-looking bloke from one angle but is actually the strangest looking man from another.
"You need that oddity; you need somebody who is carved out of solid star...."
Moffat said: "I always thought Matt, while a very young man, had something of the demeanour of a much older man, whereas Peter is a man in his 50s but is terribly boyish and young at times. I like the Doctors to have mixed messages about what age they are - you can't really pin them down.
"The Doctors are all the same Doctor really, at the end of the day, but you can slide the faders up and down....
"Typically, Matt's method would do that, too: occasionally just turn cold and you'd think, 'You're not really a puppy are you?' Just like Peter Capaldi's Doctor will sometimes remind me he's a big kid at heart."
Cold Feet star Hermione Norris, Ben Miller, Tom Riley and Keeley Hawes are joining Doctor Who this year, with the show heading for Lanzarote.
Moffat said that writing the 50th anniversary episode of Doctor Who was "terrifying", adding: "I can remember sitting with my wife saying, 'I can't tell if it's good any more, it could be rubbish - I'll have to leave the country. I'll have to fake my own death.'"