BILLY Bragg believes you have to be an optimist to be a person of the left. That may explain why he re-joined the Labour Party last year, following Jeremy Corbyn's election as leader. "I felt I'd been singing about this stuff all these years and now somebody who represents that has been elected. I really should take part now because otherwise what am I doing? I'm just exploiting it. I have to get involved."

And so he became a party member down in sleepy Dorset where he lives and where the Tories have been in power since the 19th century. He was part of the post-Corbyn surge. "All of a sudden we have 600 members where we used to have 200 members and it's still rising in our constituency Labour Party."

It's a Tuesday morning and Bragg is talking politics. It's what he does, what he's always done. When we finish chatting he's got to speak to a journalist from BBC History magazine. There was a time – back when he first came to prominence in the early 1980s with his foghorn voice and songs of unrequited love – when the only magazines he would have been speaking to were the NME and Melody Maker. But the world and Bragg have moved on.

And they keep moving. Something he said in an interview a couple of weeks after Brexit was already out of date by the following day. "The whole premise of what I was saying had changed," he says. "And if you’re a writer of topical songs, trying to keep up with the changes is just ridiculous."

Yet on he goes, trying to find the positive in the political. Bragg is by nature a glass-half-full kind of guy. As Labour goes through its dark night of the soul, as Scotland considers its position in the United Kingdom (yet again) and Brexit tears up the economic and social settlement, he is constitutionally inclined to see the positives. These are challenging times, he says; daunting times. But political uncertainty also offers opportunities.

"It seems to me that the idea of a managerialist political consensus is breaking down," he argues. "The Republican Party has lost control of its political agenda. Donald Trump has run off with it. The same has happened with Jeremy Corbyn and the Labour Party.

"And both the Democrats and the Conservatives have clearly shown themselves to be afraid of their members. The Conservatives refused to let Andrea Leadsom run. They more or less put her in a room with a bottle of whisky and a revolver and said, 'Work it out for yourself'.

"The politics we're used to that just promise everybody that we'll manage the economy and deliver growth is clearly not working with people. Everyone knows that only delivers for the one per cent.

"So how do mainstream parties respond to that? I think that's the challenge. It's a particularly existential challenge for the Labour Party but it will be for the Conservatives as well, I think. A majority of its MPs are Remainers and now they have got to deliver Brexit. As soon as it becomes clear that we can't have our cake and eat it, the pressure to remain, particularly from big business, is going to pull the seams of the Conservative Party apart as well. So the next election isn't a done deal."

A couple of days after we speak the latest polls show the Tories have a 14-point lead over Labour. There are optimists and then there are optimists, Billy …

Of course Bragg has never been only a political songwriter. Some of us first discovered him when he was as well known for songs about the politics of romance as the politics of Thatcherism. For every song about the Falklands War (Island Of No Return) there were two about love and romance (if I get to choose I'd opt for the thwarted school disco romance of The Saturday Boy and the none-more-English regret of St Swithin's Day).

It's possible that it will be Bragg the bruised romantic who is front and centre at the Edinburgh International Book Festival this evening as he discusses the printed collection of his lyrics A Lover Sings (actually, that's another good one).

He says he still feels connected to the young man he was, the one, he notes in the book's annotations, who was getting through relationships "with the speed and commitment of a flat stone skimming over a lake" and when the only woman "banging on about getting married" was his mum.

These days he is living a life of domestic contentment with his wife Juliet near Chesil Beach. (I take it that it's Juliet who answers the phone with a cheery "Bragg Central".)

The family – Bragg, Juliet and son Jack, moved down at the start of the century in an effort to recalibrate his work/life balance. "Our son was six years old and being in London I was just available for too many things so moving 150 miles south and west meant my family took priority."

Reading Bragg's early lyrics, you realise that the world he once wrote about – where working-class blokes are trying to take on board ideas of sexual equality and that very 1980s idea of the new man and political correctness – has long gone, as has the sexist, racist culture that surrounded it. "And hurray for that," he says.

It's a habitual tic to reduce 1980s pop to Duran Duran, Spandau Ballet and the New Romantics, as Dominic Sandbrook did the other week on his BBC documentary on the decade. But Bragg was an example of the decade's alternative voices, politically and culturally. "Politics and pregnancy are debated as we empty our glasses," as Bragg sings on Greetings To The New Brunette.

You could argue that if Thatcherism won the political battles back then, the alternative narrative of the 1980s won the culture wars. We've travelled from Clause 28 to same-sex marriage, after all. "Well, I think we became grown-ups," Bragg suggests, "and those ideas became the consensus ideas."

What has also changed, he suggests, is the role played by music in the culture. "Music has lost its vanguard role really. Although we didn't think about it at the time in those terms it kind of was our only social medium. It was how we spoke to one another and how we identified. All the things you do on Facebook now, we did through music."

Have we lost something then? "I don't think we've lost something. But all those young people who would have been so frustrated that nobody listened to them that they would pick up an instrument and learn to play it and then write songs because they're so desperate to get their voices heard, can now use Twitter and Facebook. Make a film on their phone. There's lots of ways to take part in the discussion now."

As a result, he suggests, being a musician no longer automatically qualifies you as progressive. "Back then, it was taken for granted that if you were a musician you had progressive views. It wasn't just people like me who talked about politics. Everybody had to be able to express the views of the counterculture. Very few musicians now in their early days would be putting forward their views on the world. If anything, they would be shying away from it. So in that sense music has lost its edge a bit. I think it's gone back to being about entertainment, 'how big is your light show?' ... and all of that."

What's also disappeared, Bragg says, is the idea of a generation gap. "My son doesn't think any great music has been made in the 21st century. He's 22. He just buys vinyl. He believes he's getting music straight from the source.

"I can see that but if I'd grown up with the sort of music my dad liked it would have been a few George Formby songs. So that's really changed. Our culture has seeped into the mainstream and through that, lost some of its potency."

Then again, while some things change … "The really strange thing is, some of the political songs become relevant again," Bragg points out. He reminds me of a recent news story that revealed how back in the day Thatcher's then press officer Bernard Ingham, concerned that CND was on the rise, asked Buckingham Palace to release some new pictures of then toddler Prince William. One generation on …

"I have a song that goes, 'It says here/This year's prince is born'. Do these things change?" he asks. "I sing songs written 79, 80 years ago: 'I ain't got no home in this world any more.' The housing crisis in Britain is as bad as it's been for 30 years. Topical songs don't always remain stuck in their time. They move on and people pick them up and use them for different things.

"You can turn around and see a newspaper headline and think, 'I've got a song that fits that,' and pull it off the shelf. Ostensibly, Between The Wars is a song about the 1930s and the 1980s, but in the context of Brexit the line 'sweet moderation/heart of this nation', kind of brings that into play again."

Bragg has previously admitted that the 1984-5 miners' strike was a major source of his political education (it's worth remembering he'd joined the army before he became a singer). And now it's hard for him to separate the personal from the political.

His forthcoming album, Shine A Light, is a collection of railroad songs written in collaboration with Joe Henry. The duo spent 65 hours travelling by train between Chicago and Los Angeles, jumping out at stations during changeovers to record a song. Even when discussing it, Bragg jumps tracks to the political.

"There are more railroad songs than any other type of songs about travelling and the train is often a metaphor," he says. "It was to shine a light on why that might be. But also to look at the idea of the railroad – particularly in the United States of America – as a viable form of transport for the future. It's a lot less environmentally destructive than road or air."

Inevitably we return to the fallout from Brexit again. What bothers me, I tell him, is that whatever victories we might have made in the culture wars may be under threat again. "There are those people who cast their vote against immigration now feel legitimised to start lashing out and talking to people about 'going home' in inverted commas," he agrees.

"I think the problem is because everybody knew what Remain meant and it was pretty boring: b***ering on with the European Union. The idea that we will just carry on with the status quo wasn't a good pitch.

The Brexit campaigners were allowed to paint anything around what Brexit meant."

Boris Johnson's famous quote about being "pro cake and pro eating it" could, says Bragg, have been the Leave campaign slogan: "'You will have your cake and you will eat it.' Now people are waking up to the reality of it. We're in a difficult situation. This is what happens if you try to govern by plebiscite."

Well indeed. Depressingly, though, the Tory Party has adjusted more quickly to the post-Brexit reality than Labour, who are in the mire of a leadership battle between two candidates, neither of whom is up to the job, in many people's view.

"I think Labour were already in their own existential crisis," says Bragg. "They were leaking votes to Ukip from the late years of Tony Blair. Then losing nearly all their MPs in Scotland to the SNP. These things predate Corbyn."

Is a split inevitable? "No, I don't think so. What you might call the big figures in Labour haven't even got the courage to stand against Corbyn so they definitely won't have the courage to form their own party. Owen Smith is the best they can do. It's embarrassing for the Parliamentary Labour Party. They send out someone who is not only unknown, he's basically another technocrat.

"Everyone knows that Corbyn is going to win. They should have the election next week and get on with it because at the moment we're just wasting time when we could be really starting to work out how we're going to resolve this situation. The Labour Party has moved to the left."

Scotland, of course, has moved in a different direction. Labour no longer speaks for many people here if the recent Scottish election is to be believed.

"That shows what happens if you fall into the trap of tribalism. I think the Labour Party in Scotland has been riven with tribalism for a while. What we need now is parties that can make common causes with other parties. Elements within the Labour Party are already talking about that. To bring about the real fundamental changes we need, to bring proportional representation to Westminster, to make sure everybody's vote counts. It was first-past-the-post excluding people from being represented that made the anger build up towards Brexit."

Scotland could always choose independence. That would be a pretty fundamental change. Yet though Bragg wrote during the 2014 referendum campaign that Scots should "vote yes to independence and set us all free", he is not being drawn on the possibility this time around. "That would be an ecumenical matter," he suggests. Father Ted would be proud of him.

We end where we began. With hope. "I don't know where Brexit is going to end up," he says. "I don't know where the Labour Party is going to end up. I don't know where the American election is going to end up. These are challenging times. But they're also times of change and opportunity for putting forward positive ideas, It's down to all of us who believe in those ideas to engage.

"I'm optimistic. I keep my cynicism on a very short lead and try not to feed it, hard though it is at times.

Billy Bragg is still hoping to make the great leap forward even now.

Billy Bragg talks to Vic Galloway at the Edinburgh International Book Festival tonight at 8.15pm

A Lover Sings: Selected Lyrics is published by Faber, £14.99