HAVE I ever punched a man?” When he wants to think about a question Rob Brydon repeats it slowly back to you in that lovely rich Welsh accent of his, softened but not squashed by his years living in London.

It’s the voice of Uncle Bryn in Gavin & Stacey, of TV gameshows and even that P&O cruise TV ad.

“Have I ever punched a man?” Brydon pauses for a beat. And then another. “No,” he finally says. “No. I think I had a fight when I was a boy, but … No, no, I’ve never punched a man. It’s not in my nature.”

Maybe he thinks that’s a disappointing answer, so he quickly adds: “I do a bit of boxing training. I try and keep fit and part of my training is boxing training, which makes me think, were I ever to find myself in an altercation, I’d know what to do.

“But I suspect the reality would be a little messier.”

A Wednesday afternoon in Strawberry Hill in south-west London and all is calm. The sun is shining and Brydon has spent the morning in the garden. Here is Brydonus Domesticus. Husband – at one point in our conversation his wife calls him to ask him

to take something out of the freezer; he politely tells her it will have to wait – and father of five; three from his first marriage and two children who are 10 and six from his second.

“Until you have children you don’t know how deeply you can love,” he tells me. “I think the love for a child can be overwhelming. Which is not to say I’m Mr Lovey-Dovey. I’m as exasperated and defeated by it as anyone. But it’s another level of love.”

Of course it’s Brydonus Professionalus that the rest of us know. Later tonight he is performing at a private function. Tomorrow he is filming more episodes of the TV panel show he hosts, Would I Lie to You? (now in its 11th series). And right now he has retreated to his office on the top floor of his house to talk to me.

The reason? His latest film, Swimming with Men, which is tomorrow night’s closing night gala choice for this year’s Edinburgh International Film Festival.

Swimming with Men is the story of Eric, a middle-aged accountant (played by Brydon) who has never punched anyone (hence the question above) and is nurturing a midlife crisis. To combat it, he joins a male synchronised swimming team featuring the likes of Daniel Mays and Thomas Turgoose, coached by Charlotte Riley. It’s a deliberate attempt to make a feelgood movie. Swimming with Men is this year’s The

Full Monty (or so its makers will be hoping).

From the title, I tell him, I had imagined Brokeback Mountain in Water, but that’s not quite what I got. “That’s a shame,” he says laughing. “That’s what we were aiming for.”

Admit it, Rob, you only did it so you could spend the whole day in a pair of Speedos.

“I’m not sure about that. That was quite challenging, the physical aspect of it. We did two weeks of boot camp in the water. Three hours a day. Shooting scenes in the pool, it doesn’t take long being in and out of the water for your body temperature to drop. We were all getting freezing cold and you’d be competing with the other guys to be last into the water and first out.”

And then there’s the sucking in your belly for every shot, I say. No, he doesn’t think that happened too much.

“I’m certainly guilty of holding my stomach in real life. I waver between thinking: ‘Oh, no, let it all hang out. Show a human being’. And then occasionally vanity would creep up on you and you’d think: ‘Hold it in’.”

Swimming with Men aims to be a crowd-pleaser and I think that’s what Brydon tries to be too. Yes, you can find dark, dysfunctional sitcoms such as Marion and Geoff or Human Remains on his CV. And, yes, the ongoing mockumentary

series The Trip, in which he appears alongside Steve Coogan, is all needle and snark.

But, really, Brydon is just at home – maybe more so – in his more mainstream ventures. I think it’s fair to say he is not afraid of the “light entertainment” label. He has little of the comedy snobbery some of us bring to the table.

“I remember talking to Steve when we were doing one of the Trips,” he begins when I bring it up. “He had older siblings, which I think can make a difference. I didn’t. And I remember him saying that he didn’t go to see Grease at the cinema because he felt or knew that it was for ‘the masses’.

“I had no conception of there being such a thing as ‘the masses’ at that age. In fact, it was years before I had a conception of that. I’ve always liked many different types of culture. I’ve always liked Bruce Forsyth.”

Brydon admires Gore Vidal, too, he points out. But he is not ashamed of admiring Bruce and Des O’Connor and Frankie Howerd and Bob Monkhouse and Terry Wogan. He sees nothing wrong with Saturday night prime-time TV or West End glitz.

And he’s had the odd moment himself. “I’ve been very lucky and I’ve done shows at the Palladium with Neil Diamond, sung with Tom Jones at Wembley Arena.”

In short, cult status is not something he has ever aspired to.

He's not one for being difficult on set either. “I like to get on with people, I like to be easy to work with. I see no value in being dark and moody and mysterious and enigmatic. I’m more of a ‘Let’s get on with it and then go home’ type of a person. I like to be quite focused on a set.

“I’m probably not as jokey as you might think. I’m not a big corpser. I’m not a big one for breaking up in the middle of a scene in howls of laughter. I get a bit miffed by that.”

Did he ever want to be dark and mysterious? “Yeah, I think all actors when they’re younger think they’re going to be Al Pacino. But I think all you can be is who you are. You can only find success if you are true to yourself rather than aping somebody else.”

Anyway, he says, sometimes the choices you make have nothing to do with what is cool or what might be successful or how it will help your career.

“Certainly, for me, the choices I make now are as much to do with family and my home life. So I don’t want to go away for months on end, for example. I just don’t want to not see the children.”

“That’s a big part of it, which I think people fail to understand. They assume you exist on some weird plane where there is nothing else going on.

“I’m not defined by my work. It’s not the biggest thing in my life, whereas I think there are many for whom it is. It’s still their thing.”

Was it ever that for you, Rob? “Yes, yes. But then I seem to be somebody who will tick something off …”

He reaches for an example. “I grew up being very into American culture. People like Johnny Carson and David Letterman and I had an ambition to go and do that when I was younger.

“But what I’ve found is I’ve done a little bit over there and I’m kind of sated by it now, you know?

“I did one Jimmy Fallon show about five years ago. I had that experience of going to NBC, going up in the lift, meeting Jimmy Fallon, who was adorable, and going on the show. And I thought: ‘OK, I don’t need to do that again’.”

Ask Brydon about childhood and he might start talking about the smell of leather. “A few years ago, my wife bought me a leather wallet for my birthday and I remember opening it up and she was quite alarmed as I plunged my nose into it because the smell was identical to the satchel I had on my first day of proper school. It brought back all those memories.”

It’s the noise of woodpigeons that takes me back, I tell him. “Yes!” he almost shouts. “There was a school I was at and one class was two storeys up and [outside] there was a beautiful tree; thick branches, verdant leaves and that …”

He starts to mimic the thick-throated noise of Welsh woodpigeons.

“And, of course, it would have been an ordinary day but, in my mind, it was filmic. It’s cinematic and beautifully lit. It’s interesting how these things play out in the mind.”

Brydon grew up in south Wales, went to the same comprehensive school as his fellow Gavin & Stacey star Ruth Jones and always wanted to entertain. “I would be the one in the playground making the others laugh.”

And yet it would take him a long time before he managed to parlay that love affair into a career. He wanted to be an actor. He did stand-up. But all that led to was a job as a radio presenter and, at one stage in his twenties, presenting the Shopping Channel.

Were you any good at it, Rob?

“I was. I think I was one of the best they had. This was in about 1988-89 and it was in the very early days of Sky television. For me it was a way of getting to London. With hindsight it probably hamstrung me for a bit because you’re certainly not going to get taken seriously as an actor if you’ve been on the Shopping Channel.

“But in the longer run I learned a lot from it. I learned about telling and energy. It gave me a lot of time on camera. It gave me a lot of time reading autocue.

“The past is another country and you think: ‘What was I doing?’ But I was very young, naïve. My father was a salesman. That was maybe something I would have done if I hadn’t gone into what I’m doing. I slightly cringe when it’s brought up, but there it is.

“I did all sorts of things. I did that. I did a lot of pretty mediocre stuff. In many ways it’s a miracle I’ve done what I’ve done after that start.”

Brydon was 35 when Marion and Geoff and Human Remains suddenly gave him some career traction. “I was beginning to lose faith in ever making it just before that.”

That late start may have staved off any midlife crisis, he thinks. “I’m 53 now. Many things started to improve for me around my midlife period. One of the things in Swimming with Men that I try to convey is Eric’s awkwardness of joining a new group of people that he wasn’t familiar with, which I think a lot of people would feel. I’m told that for men their social circles diminish as they get older.

“Being an actor and being me, it’s actually very different. Because I’m constantly going into new groups of people with different jobs and constantly picking up new friends, some of whom become real friends, some of whom become acquaintances, colleagues. It’s very different for me. Personally, I find that kind of thing easy.”

That said, I wonder if you can trace Brydon’s willingness to try his hand at everything back to that period before success. Does he still see the shadow of

the wolf at the door out of the corner of his eye?

No, he says, but only through force of will. “I don’t allow myself to think of that in the same way that I don’t allow myself to think of nerves or stagefright. I won’t allow it. But sometimes it creeps in. It does knock at the door.

“You have to say: ‘No, no, no, you’re not coming in here’. You have to be on your toes and not allow it because my career is over if I lose my confidence, if I lose my self-belief.”

Swimming with Men closes the Edinburgh International Film Festival tomorrow night. It then goes on general release on Friday.

In Person: Rob Brydon is at the Edinburgh Filmhouse tonight at 6.20pm