YOU might remember his "very brief brush with pop stardom" courtesy of Kiss Me, which - at the third time of asking - peaked at number four in the UK singles chart 30 years ago. 

You might remember him with Alex James of Blur in the short-lived Me Me Me (their one single, Hanging Around, breached the top 20 in 1996). But you might not know much about the extraordinary pastoral folk pop Stephen Duffy has released on and off since 1987 in the guise of The Lilac Time.

Which, if true, is a shame. We will get to The Lilac Time in due course but first there is a tale to tell, "the longest story ever told", he laughs down the line from the home he shares with his wife and colleague Claire, their daughter and their two dogs in Cornwall. "It's impossible - I say I'm going to do an interview and then I have to try to remember all of it."

The arc of Duffy's narrative breaks with tradition, in as much as it begins at the top: co-founding Duran Duran almost as soon as he'd stepped through the doors of Birmingham Polytechnic in his hometown. "The first person I spoke to was John Taylor so art was a thing of the past virtually immediately," he recalls. "But that was the thing. Keith Richards, John Lennon. Go to art college, start a band, steal the PA. We did our first gig in the lecture theatre, a beautiful old Victorian building. Obviously they've knocked it down - it's Birmingham."

Three gigs later and Duffy walked, leaving co-founders Taylor and Nick Rhodes to their own devices (and a vastly inferior frontman, but let's not go there). "It would've been fun to have done more but they were ambitious in a way I wasn't," he concedes. "I went off and started another band then completely caved in and did Kiss Me because when all your friends have been on Top of the Pops it's like, 'I've got to do something here.' The Beat, Dexy's Midnight Runners, The Specials - every time you turned on the TV there was somebody you knew."

Success wasn't quite as instantaneous as that sounds, however. "You thought: 'I'll put out a record and I'll be on Top of the Pops.' But Kiss Me was around for three years before it was on Top of the Pops. I kept on signing with different record companies and saying: 'Can I move on now, please?' And they'd say: 'No. Record another version of Kiss Me.' But it worked in the end."

Duffy's instinctive distaste for the easy option kicked in without delay. "As soon as it was a hit I thought: 'I can do what I like now' and started The Lilac Time. And Virgin, the record company, said it didn't want anything to do with it."

Taking their name from a line in River Man by Nick Drake, Duffy, his brother Nick and their friend Michael Weston found another label and embarked a journey shot through with a bloody-mindedness that persists to this day. "Halfway through the 1980s it was if I'd spent all the time pretending the 1980s weren't happening, escaping from the big drum sounds and the synthetic thing, trying to make small, intimate acoustic music," he says. "When we started it was only 12 or 13 years since Nick Drake had died and many people thought we were insane, playing the acoustic guitar. Now there's a greater appreciation of it. We were only 30 years too soon." Duffy, now 54, laughs at the thought.

The music industry at the time was anathema to the folk-obsessed adventurers, who billeted themselves in an 18th-century farmhouse in Herefordshire with no mod cons. "We were signed to PolyGram, this huge international record company, and they kept saying: 'Go to America and let's record some drivetime hits.' And we were in this farmhouse without a phone, without any heating. We'd walk across the fields to the pub to use the phone box outside."

The music business now is unrecognisable to its 1980s incarnation, Duffy says. "Even unsuccessful record companies seemed to have millions and millions of pounds knocking around. When you were at school they said: 'Whatever you do, don't go into music because there's no future.' Then you went into it and there was all this money sloshing around. Now, when they're actively encouraging people to go into it - you can do degrees in the music business and there's all these Opportunity Knocks TV shows - there's no place for these people to go." Kiss Me, he says, "made something like £12,000 in radio plays in a month, and it went on for ages - the money kept coming in". On top of royalties, he says, the 1980s was the era of the colossal advance. "You could buy a house with it."

The Lilac Time's attempt to maintain a bucolic lifestyle with one foot in a resolutely Londoncentric industry ended after four albums in 1991. Eight years passed wherein Duffy released a triptych of solo albums, collaborating with Alex James ("I always knew he was a Tory") and Nigel Kennedy along the way, before The Lilac Time regrouped. "By then we were on an independent label and we were like: if we get back together now we can do it as we wanted to do it. And we've pretty much stuck to that."

Besides co-writing and co-producing Robbie Williams' album Intensive Care in 2005, an experience of which he is proud but which "confirmed I wouldn't have been comfortable with the rigours and repetition of success", Duffy has masterminded a further four Lilac Time albums since their reformation, with a glorious fifth, No Sad Songs, ending eight years of public silence a week tomorrow. "There was a fair bit of time where I thought: 'We'll make music for ourselves,'" he says, "but then it got good and that terrible moment comes where you think: 'I have to share this with the world. It would be criminal to keep this to myself!'

"The other reason I felt obliged to share this is that it is a lot happier than the last record. When I hear Keep Going now I think: 'My god, he's depressed.' So No Sad Songs is like the happy ending. We all like a happy ending, don't we?"

No Sad Songs is released on April 6 on Tapete.