MID-AFTERNOON on the south bank of the Thames, the sky is an Orwellian shade - storm-swollen, gun-metal grey, and bearing down on the heads of the millions going about their business. Up there in the London Eye they're right in the thick of it, squinting between trickling raindrops at those jutting icons of power and money, Big Ben and the Gherkin'. If they were to peer a little closer at the street, they might see a man who has spent almost 25 years opposing, in song and deed, the brute excesses of capitalism and the state. From this distance he looks like a Lowry matchstick man in blue jacket, grey trousers and check shirt. His hair is silver, his heart gold. His nose is as pointy an emblem of the Eighties as Morrissey's quiff and Madonna's conical bra, and he has a splendid track record of poking it where it doesn't belong. Ladies and gentlemen, it's Billy Bragg.

I meet him in a dim Italian restaurant in a railway arch beneath Waterloo Station. It seems an appropriate venue. His mother's family are from the Amalfi coast, south of Naples, and Bragg's songs have something of the elegiac urban romance of The Kinks' Waterloo Sunset.

He eats pizza, drinks coffee and talks with his mouth full. He will be 49 next month, and like other one-time hate-figures on the Left - Peter Tatchell, Tony Benn - has been around for so long that in these less ideological times even his natural enemies regard him with something approaching affection. The Times described him as a national treasure ("I wish that came with funding," he sniffs) and even David Cameron has admitted A New England is one of his favourite tunes.

"He's such a cop-out," says Bragg. "He's the sort of person who likes my love songs, but doesn't like my politics."

Bragg has travelled up by train from his home in Dorset. He has a busy day. There's this interview, then he is going on David Starkey's discussion show ("The researcher described the other guest as an alleged Mossad agent, so I'd better be f**king careful"), and later he will make a public appearance to talk about The Progressive Patriot, his first book. He is still getting used to the idea of being an author. At a couple of parties thrown by his publisher, he found most of the other writers regarded him with suspicion. "I wanted to say to them, I did write my own book! I'm not like Kerry Katona!'" Perhaps surprisingly, he got on very well with Andy McNab, SAS member turned trashy novelist. "Lovely bloke!"

Bragg's book is partly an exploration of national identity, inspired by his anguish at the success of the British National Party in the council elections in his home town of Barking, and by the London terror attacks. "What all the political parties have failed to do, to the detriment of the current challenges we face because of the threat of jihadist suicide bombers, is to define Britishness in some way that is easily communicated to people," he says. "We are asking these people coming into our midst to assimilate. But to what? What part of Britishness should they cleave to? Do we want them to come out of Friday prayers and go and get pissed in the street?"

For me, the more interesting parts of his book are the autobiographical passages. Bragg was born in Barking in 1957 and his accent is pure estuary English. When he asks me what I think of his book he says, "Izzi' abou' Inglishness or izzi' abou' summing else? Wochoo fink?"

He understands his life as a series of key moments. One of these came in 1970, when he was 12, on a school coach trip to Holland. Sitting at the back of the bus on a car ferry, he and his classmates spotted an adjacent coach full of girls around their age. There was a lot of waving and smiling, and Stephen Bragg - as he was then - felt for the first time an attraction to the opposite sex.

But the ferry crossing only took a few minutes and soon the girls were gone. Bragg was left feeling a great sense of loss, which he was frustratingly unable to articulate. Just then, however, a song came on the radio - The Boxer by Simon and Garfunkel. He'd never heard it before, but it seemed to meet his mood, and his eyes filled with tears. He had just discovered "the fabulous nexus" between pop music and human desire, and he would never be the same.

"Making that connection with music was very important to me," he recalls. "It did give me access to a set of feelings and ultimately to an emotional language. These were feelings that I couldn't speak to anybody about. I didn't have an older sibling. I just had my parents, and they weren't really interested in listening to me, aged 12, gibbering away about Dutch schoolgirls I'd met on a bus."

It's interesting that the first time he ever felt positive feelings of attraction they were immediately followed by a sense of a chance lost. It set a pattern for most of his adult romances - his wife Juliet has said that before they got together he had one disastrous relationship after another - and a mood for his songs.

"Yeah, listen to The Saturday Boy, it's all still there," he agrees. "The most wonderful love is unrequited. All my favourite love songs are unrequited love songs. Back To The Old House by The Smiths is probably the finest example. Tracks Of My Tears. Long Distance Love by Little Feat. That, to me, is the best kind of love. It was that feeling of possibility snatched away, that was the feeling I wanted to get. It was that longing. It wasn't actually those girls. It wasn't lust; I found longing a much stronger sense. It wasn't until later when I worked out there was a place in Barking where you could buy second-hand copies of Penthouse that I really got into exploring lust, though maybe your readers on Sunday don't want to know about that. But that longing informs so much of my work and it does stem back to that moment."

Does that mean it has been harder to write songs since getting happily married? "No, I'm still writing love songs about the relationship I'm in with my missus. I don't think it's about happiness. You can long for many things. You can long for a better world."

Is it possible that he has in the past sabotaged his own relationships in order to give himself material for songs? He shakes his head. "The awful truth is that by the time I was 25 and started making Billy Bragg records I had already had so many f**ked up relationships I really didn't need to self-sabotage. Although, Workers' Playtime is my great break-up album. A lot of it - Must I Paint You A Picture, She's Got A New Spell, Life With The Lions - they are all about a single relationship which really was mostly tailspin. It wasn't much of a relationship. It was mostly falling away to the ground. But I didn't do that on purpose in order to be able to write."

That 1988 album was written about Mary Bollingbroke. After listening to the record for the first time she threw it off Hammersmith Bridge. "Was it worth going through the break-up in order to create the album?" I ask.

"Well, if I tell you how great it feels to sing Must I Paint You A Picture," Bragg smiles. "I love that song. It's a great song. So, it's worth experiencing things, yeah. If you can come away from an experience with some insight, or a nugget of self-knowledge which you can then impart to other people, it's always worth it. Even some of the horrible things that might happen to you, some tragedy, if you can salvage something you can communicate to other people that might help them. That's what my songs do, so people tell me.

"A song like Tank Park Salute would be a prime example. It's about the death of my father. A number of people have told me how that has helped them. I'm not talking about people whose fathers died when they were 18 like me. I'm talking about guys my age who are now facing up to that, and Tank Park Salute is somehow connecting with them."

Dennis Bragg, who worked in a warehouse, died of lung cancer in 1976 at the age of 52. Tank Park Salute, released in 1991, marked the first time Bragg had ever allowed himself to think deeply about that event. It's arguably his best song and is certainly among his most moving. "What's it like to perform it in concert?" I ask.

"It's hard sometimes. It's hard," he says. "In May we played in Barking and I knew my mum and my brother would be in the audience, and I was so conscious of that that I actually had to write the lyrics down. I know the lyrics inside out, but I had to write them on a piece of paper just in case. While singing it I was thinking of mum and I know, because my brother told me, that she was crying all the way through it."

The year 1977 was transforming for Bragg. He saw The Clash and became a supporter of Rock Against Racism. I ask how his father's death only a few months before had fed his involvement with punk and politics.

"In a number of ways actually. My experience with having a close family member dying of a terminal disease in the house is that everything stops. It feels like everything stops. You can't make plans, you can't look forward, you can't live every day as something new because you've got this constant weight on you. So dad dying in October '76, well not only did he die but I was 18 as well. At the start of it I'd been 16 and now I was 18.

"You know, we tidied all his things away, and I tidied all my kid things away out of my room as well, and then I walked out the front door and - there was punk. This was the modern world; this was what was going to happen next. So punk carried me away from the experience instead of leaving me there to try and deal with the enormity of it there and then. Punk took me away from it. I was still living in the same house, but punk had a velocity to it, an urgency, a sense of purpose, and I desperately needed that at the time."

This subject has interested him, and he carries on, following a train of thought, to talk about Riff Raff, his band of that time. "Being in a band, even though we were only playing the back rooms, it was a real clear cut-off point. That was where my childhood ended. Maybe other people's childhoods end by shades. I don't know. I'll have to watch my son to see that. But for me it was a closed door. That was the past, that was the family I was in, and now I wasn't in my family any more. My mum and my brother and me became three individuals. It wasn't great. We didn't talk to each other about my dad's death for a while. It was quite difficult. I left home in '77 and my mum was left to pick up the pieces on her own.

"I regret that. I'm sorry that happened, but I just didn't have the words then. I never spoke to anybody about it ever until I wrote Tank Park Salute. The idea of discussing it with a stranger like we are now? I just wouldn't even go there. In fact I stopped getting drunk because I did it when I was drunk. I would start thinking about it and talking about it when I got drunk. So I stopped doing that. Just not wanting to deal with it."

He says that his mother still finds it difficult to talk about what happened, and that he and his brother don't talk about it at all. I thank him for being so open today. He's not done though.

"It saved my life," he says. "Punk probably saved my life. It had been such a shock. Before that I had actually been bumming around France. I had left my job and gone to France. I was 17 and going to go and see the world. I was halfway round France when I got the phone call to say my dad had gone into hospital, and I came home. So it had caused my plans for growing up to be aborted. Once that was over, punk helped me to reconnect with what was going on."

How did it save your life, though? That's a very strong thing to say. "If I had sat around and felt sorry for myself I would have sunk into all sorts of unfortunate options, growing up in suburbia in the Seventies."

Drugs and alcohol? "Yeah. Instead there was something about punk that seemed to eschew that, and particularly eschew self-pity."

That's interesting. He dealt with bereavement by putting his faith in music, similar to the way that many people find comfort in religious belief. "Hmm. My mum. She had been brought up a Roman Catholic when my dad got ill and subsequently died she found great strength in her faith. I respect that."

Are you a Christian? "Yeah, I am. I've been wearing this for years." He fishes a chunky St Christopher medal out of his shirt. "That might be because I'm from Essex though." He chuckles at this reference to bling.

Bragg is no great believer in organised religion. He feels closest to God when he is among nature, walking his dog Buster (as in Prince Buster, the ska star) in the woods or on the beach near his home. Coastal Dorset has many iron and bronze age sites, and as Bragg is a person who has always been sensitive to the feeling of the past being part of the present, he finds it very meaningful and enriching to live there, part of the generations piled up like sedimentary layers. His book is subtitled A Search For Belonging, and he seems to have spent his whole life on that quest.

In his mid-teens he was really very withdrawn, spending a lot of time alone in his room with his Simon and Garfunkel songs, frightened of bullies and uncomfortable around girls. He felt unloved and unable to communicate, although he dreamed of being a singer-songwriter. "Well, I was a soft boy, y'know. I was in the school football team, but I was more bookish. I was that kind of guy. Stephen Bragg, when I think of him, he didn't have the confidence to do what I do."

You think of Stephen as being separate from Billy? "I do. I think of him as a different person, yes. Somewhere in the cleansing fire of punk, I reinvented myself. For the better, I think. He was someone who lacked that urge to change the world. He would have rather preferred the world to go away from him."

His eventual involvement in the miners' strike of 1984 was thus an extremely positive and exciting personal development. It's well known that it was his experience of travelling the country playing benefits for the miners that first politicised Bragg (he hadn't even voted in the 1979 election), but on a more basic level it made him feel part of a community, a family, not so alone. He describes it now as a "homecoming".

He had made an earlier and less successful attempt to find an alternative family - although he probably wouldn't put it quite like that - when he joined the army in 1981. He says now that he was trying to find a way to measure himself against his father, Dennis Bragg, having served with the Royal Tank Regiment in India during the Forties. At the time, though, it felt more like a reaction to the failure of Riff Raff and the end of punk.

"Everything I'd committed my life to had come to nothing," he says. "The band I was in had broken up. The Clash were making shit triple albums. I'm not listening to Sandanista, don't give me that shit. What's the most outrageous thing I can do now?' And joining the army was it. I wouldn't have to look anyone in the eye any more and see that I'd failed."

He enlisted in the Royal Armoured Corps and was packed off to Yorkshire for training. "Of course, as soon as I got there I started writing more songs, better songs. It was hugely inspirational and I thought to myself, F**king hell, now I'm going to have to get out again, and I'm really going to have to do this properly.'"

Buying himself out after three months, he was determined to make one final attempt at a successful career in music. "I think that's why I ended up doing it solo. I wanted to either do it right or be consumed in failure. As raw as possible. It was almost like a one-man kamikaze charge at the idea of being a singer-songwriter. Shot or bust!"

Happily, it worked out. He released his first album in 1983 - Life's A Riot With Spy Vs Spy. It sold well and he followed up with increasingly political material such as the 1985 folk song Between The Wars (a hymn to working-class values, which saw him on the same edition of Top Of The Pops as Phil Collins doing Easy Lover) and the classic 1986 album Talking With The Taxman About Poetry. The shy kid from Barking had become a star and it suited him. He admits to having a sizeable ego just like any other performer.

"I'm not a great one for doubt," he says. "In some ways I'm just a glorified busker. I'm here and I'm trying to get your attention. I'm not claiming that it's great art or I'm singing perfectly or playing wonderfully. It can be a bit hit or miss sometimes. But my bottom line is I'm communicating. I feel I've got something I want to say, and I'm confident enough to believe that if I can say it the right way round then it will get your attention."

Even though he was very engaged with the times in which he was living - becoming, for instance, a key player in Red Wedge, Labour's attempt to use pop to court the youth vote ahead of the 1987 election - Bragg's music never sounded like the Eighties, and so it hasn't really dated. It still sounds quite edgy, and has found a new generation of fans. Hard-Fi perform Levi Stubbs' Tears during their concerts, and the up-and-coming Jamie T has covered A New England.

In any case, Bragg continues to make music. Two box-sets were released this year, and there will be a new album in 2007. It would be wrong to think he has mellowed. His son Jack was born in 1993, an event which served to harden his commitment to social justice. "For someone like me, with the politics I have, becoming a parent makes you realise why you are doing all this shit, what it's for. It's no longer an abstract idea."

Whether writing about terrorism, or reform of the House of Lords, or campaigning to get his local MP, Oliver Letwin, voted out of office, or whether he is simply trying to delineate the complex workings of the human heart within a simple pop song, Bragg refuses to shirk from what he sees as the most pressing issues of the day. You might call him a singer-wrong righter, but to explain himself he prefers to reach for a metaphor from his army past.

"I don't try to stir up a fight," he says, "but I always try to march towards the sound of gunfire."

Billy Bragg plays the City Hall, Glasgow, on December 4; Perth Concert Hall on December 6; and the Lemon Tree, Aberdeen, on December 7. On December 5 he will be talking and signing books at Waterstone's, Sauchiehall Street, Glasgow, 6.30pm. Tickets £3, redeemable against the book. The Progressive Patriot is published by Bantam, £17.99