Duffy is at his eighth club as a manager or coach, and few in the Scottish game have had such a rich or testing experience.
"Just look at the chairmen - or chairpersons - I've worked for," he says, taking a deep breath.
"Wait for it: Ron Dixon, the Marr Brothers, Ken Bates, Vladimir Romanov, Delia Smith, Milan Mandaric, Giovanni di Stefano."
The one Duffy remembers most vividly, and even affectionately, is Mad Vlad. Working at Hearts as director of football in 2006, he can still recall the blood draining from the face of the coach, Valdas Ivanauskas, any time Romanov's number showed up on his phone.
"Valdas used to turn white at the very thought of confronting Romanov, but I treated him differently," Duffy says. "Vlad was a maverick. He did things off the cuff. He looked very serious, almost a Bond-like baddie with that cold, steely look, but he was quirky. I found him humorous and likeable.
"Football can be very clichéd: guys fitting precisely into their boxes. So guys like Romanov, slagging off the SFA, the SPL, our referees, are refreshing to me. It adds colour to things."
Having said that, Duffy looks at Hearts now and knows the damage that was done. "I've no illusions about Romanov," he adds. "He ran Hearts at a huge financial risk and it has cost them. I was there during that period and saw it first hand: the club adhered to no known budget whatsoever so far as I could see."
Having coached at Hearts and Chelsea in recent years, these days Duffy is engaged in an altogether different battle: just keeping once financially crippled Clyde afloat and, if there's any chance, hauling them up out of the lowest tier of Scottish football. So far, so good; the Bully Wee are top of the league, and face Stranraer today in the William Hill Scottish Cup.
It is breadline football management. Duffy scavenges the junior ranks for promising youngsters, such as Kevin Watt, whom he signed from Bo'ness United, and Stefan McCluskey, plucked from Shotts Bon Accord. Add to that other kids freed by professional clubs - like Iain Gray, once of St Mirren, and Gavin Brown, once at Kilmarnock - and you have a feel for his working tools.
"Without doubt this is the toughest job I've ever had in football," Duffy says. "Initially I went into Clyde for a few months [in February 2011] but the board asked me to stay.
My phone wasn't red-hot offering me a load of jobs, so I thought, 'okay, I enjoy work, I love football, I'm here for the challenge'. What we did back then was put down a philosophy for the longer term. First and foremost, we had to reduce the debt drastically. The costs were crippling the club.
"This season I've have three players who are on £100 a week and that is our top whack. The rest are all on between £30 and £75 a week. We don't do expenses and we don't do pre-match meals; the guys have to do their own thing, bring in sandwiches, whatever. And we only pay a bonus after every fourth win, however long that takes. If a guy kicks a ball over a fence, I say, 'go and get it . . . we need that ball. I sell raffle tickets for the club. So do the players. We do sponsored walks and such. It's a totally different challenge. But, then again, it's what I love doing, working with players and trying to educate them, trying to make them better."
Duffy is acutely aware of the realities facing a manager today. "The average tenure in Britain now is less than 15 months," he says. "That means most managers think, 'wait a minute, I've got round about a year to get things right'.
"This was the single biggest reason I decided to stay at Clyde: because the board were totally adamant that the only way this could work was to have patience, to give everybody time to do the job."
Back in 1996, Duffy whose career was almost ended by a knee injury when he was just 28, was among the hottest properties in Scottish football management. Famously, one lunchtime, he arrived by helicopter at Easter Road, flown in as the new Hibernian manager following Alex Miller's long reign.
"Failing at Hibs remains one of my biggest regrets," he says. "I was just 36 when I got that job and I made too many mistakes, too many misjudgments. I look at that club today, with the experience I now have, and think of all the things I would do differently.
"I rushed everything. I tried to change it all. I have a tremendous respect Alex Miller but he had been there 10 years and some things had to change. But it was difficult. 'No, we don't do that, we do this,' people kept saying. I was in a hurry. I went in there with a sledgehammer and it was the wrong approach."
Duffy admits he still wonders what might have been but for his ruined knee. He says: "I'm still curious
to know what level I could have gone to in top-class football. Terry Venables told me recently he was thinking of buying me [at the time of the injury].
"I was in the Guinness Book of Records: Britain's youngest manager at 29 with Falkirk. Today my knee is murder. It's in bits. I went to see a surgeon and he told me categorically, 'Jim, you need a new knee. Your knee is completely knackered'.
So the day is coming when they'll have to open it up. I'm philosophical about everything, though. I don't cry. What is done is done. I love football."