Terence Davies has a new film out and another in post-production. At the age of 70 it appears as if the director – possibly Britain's finest living director (it's a toss-up between him and Nic Roeg, I'd suggest) – is speeding up. "Yes," he says, as we sit down together, "I might even be prolific. I might have to take medicine to stop it."

He laughs at the thought. But then he knows the truth. That in the years since his first full-length film Distant Voices, Still Lives in 1988 he has spent more years not making movies than making them. His has been a career that all too often has been one of quiet frustration. Just seven films in 27 years.

His new film, an adaptation of Lewis Grassic Gibbons' Sunset Song is a case study in Daviesian exasperation. He first talked to producer Bob Last about making it back in 2000, the year his adaptation of Edith Wharton's House of Mirth was released. He finished the script in 2003, was told that it didn't have legs by the UK Film Council and that was more or less it. For years.

The story of his cinematic life you might say. Indeed, between House of Mirth and his adaptation of Terence Rattigan's play The Deep Blue Sea with Rachel Weisz in 2011 he managed just the one film –Of Time and the City, a glorious, very personal and deeply partisan documentary about growing up in Liverpool in the years after the war, made from found footage.

But now here we are. Here is Edinburgh's Caledonian Hotel. Davies sits, stately and plump, full of passion and good humour. Around the corner, one of his stars Kevin Guthrie is being interviewed. Agyness Dene, the model turned actor who turns in a very creditable take on Chris is somewhere not far away. And Davies? In between bemoaning modern life and the pernicious impact of television on the very syntax of cinema, he is telling me about how much he loves Grassic Gibbon's book.

Why Scotland loves Sunset Song 

It's clear he loves Sunset Song. "You can't read that last page without being in tears," he tells. "It's about the nature of forgiving and forgiving all suffering. That sets us apart from the animals. We can forgive and we can hope and we can despair."

He discovered the book via the 1971 BBC adaptation starring Vivien Heilbron, and he worries that Scots will feel he hasn't done the book justice. For what it's worth I think he has for the most part. Even if he was limited by the lack of money (he can't recall how much the budget was but he knows it wasn't enough), the result is a decent, handsome film, with fine performances and moments where Davies constructs glorious tracking and crane shots that remind you that cinema is also an art form. It's not his best work, but it is full of resonant moments (there's a wartime sequence that will scour your heart).

Sunset Song has all the things you'd expect from a Davies film, pain and heartache and the sumptuous slow thrill of camera movement. This is his USP. At one point we end up talking at length about the purpose of camera movements in cinema. He tells me that tracking shots should always be moments of transformation in the film; that between the beginning and ending of the track our understanding of the situation should have been transformed. "An awful lot of tracks are simply to get people across the rooms. That's not interesting."

There's been a method to his films since the very beginning. You can see it in the three austere short films he made in the seventies now known as the Terence Davies trilogy. My first encounter with Davies was with the last of the three, Death and Transfiguration, which mixed up the last moments of a man's life (played by the late Wilfred Brambell, aka Steptoe), gay desire and church ritual. The result was itself ritualistic and disturbing.

"A lot of people don't like my work," Davies concedes. "They think the tracks are far too slow and the films are very pompous and solemn and lifeless. People do think that."

He does have something of an Eeyoreish outlook in life. Give him the chance and he'll rail against modern technology (he has a mobile but he has only one number on it, he says), our obsession with money and the body beautiful. ("Yes, he's got a great physique and she's very beautiful, but what else did your mother buy you for Christmas? There has to be something more than that.")

He is a man of a certain age who is consciously – and perhaps a little theatrically – rejecting the modern world. Indeed, he holds a particular torch for the cinema of his childhood, those films of the 1940s and 1950s when, he says, "there was a glow about cinema both in America and here, and it's not been matched since. That glow is gone. I wish I had it."

At his most depressive he will even wonder aloud if all those years trying to make films with a glow have been worthwhile. "There are times when I think 'oh, what's been the bloody point? They want big blockbusters. They want big names.' That's what people want because that's what they go to see.

"No one may remember mine. What does worry me is if they're something kept alive by academics. I mean who reads Finnegan's Wake now? It's incomprehensible. Ulysses is pretty hard going as well although the Molly Bloom sequence is one of the greatest achievements in literature. That for me would be a second death if they were only kept alive by academics saying how wonderful they were because in the end you want ordinary people to respond to them. And I'm an acquired taste. I just am. The number one film critic in America after seeing the trilogy said these films make Ingmar Bergman look like Jerry Lewis which is a wonderful insult. It's almost a compliment."

But then, he admits, he doesn't really have a choice. "If I can see it and hear it then I do it," he says of his approach." If I can't there's no point because I'm not a jobbing director. I wish I were, simply in terms of my bank balance. But I can't do something that I don't love."

The writer Frank Cottrell Boyce once wrote of Davies that "he shoots working-class people as if they've got souls." And despite his plummy accent, his distaste for pop culture, his love of the composer Anton Bruckner, he is himself of working class stock. Over the years his past has been transformed into cinema. His early films drew specifically on his own back story of growing up in post-war Liverpool, the gay Catholic son of an abusive father.

Indeed, watching Peter Mullan as the oppressive father figure in Sunset Song it's impossible not to map the character's violence onto Davies' own father. "Well, I think there will always be that element of … not antipathy, but a kind of wariness of men especially when they're in a powerful position," Davies admits when I bring it up, "because my own father was psychotic and you can't get away from that.

"But I can't get away from other things either. I can't get away from shooting people at windows and going up and down stairs and doors being opened. If you open a room and there's no one in it. There's the ghost of all the people that used to be there, so I'm rather obsessed with all that."

He does this a lot in our short time together. Time and again in our conversation he slides from the coal black, slate-sharp hardness of his personal experience into the light of his cinematic life, his other life.

In a way his early films were acts of transformation, reclaiming bitter experience for light and music (Distant Voices, Still Lives has one of the most terrifying father figures in cinematic history). Davies' own father died before his youngest son's seventh birthday. Now that Davies is 70, I wonder, can he finally look back and begin to have any understanding of the man who both created him and abused him? No, Davies brusquely says. His father is and was simply incomprehensible to him. "He was just very violent. You were tyrannised from the moment you woke up to the moment you went to bed."

For the last two years of his father's life Davies slept beside him. "He'd say, 'if you move again I'll f****** kill you.' I didn't move for hours. I was so frightened. He was very brutal. I can't forget that kind of brutality. There's never any excuse for it. Ever. And why you should have children and then abuse them … I just think it's really wicked. It's truly evil so I don't feel anything other than I'm glad he's dead and I'm glad my mother had 40 years of a life then because she deserved it."

Once again he jumps from the personal to the cinematic. "Just as I don't understand violence in cinema. I had all this when I was a kid. It's not cool, it's not funky. It's unpleasant. It's not funny, it's not entertaining, it's horrible. And when they appear to drool over it; that is beyond me. I don't see what anybody sees in that. I just don't. If it's got physical violence I'm not going. I'd sooner stay at home because it doesn't tell you anything except the obvious."

He has been marked for life by his earliest years, he says. He doesn't lose his own temper often, he says, because he doesn't like to. "I wouldn't actually hit anybody but I'm ferocious. I'm not afraid of anybody. I wouldn't care if it was Dwayne Johnson. I'd wipe the floor with him. So, yes, enormously damaged. As indeed most of my family is. I got off lightly."

You wonder, though. He has chosen a largely solitary existence. We talk about gay life then and now. Would he rather have been born later when his sexuality wasn't a criminal act (as it was, shamefully, in England up until 1967; and until 1980 in Scotland)? He uses the question to imagine growing up earlier and being an adult when the films he loved so much were being made.

"But then, of course, I wouldn't have been allowed to make films a) because I came from the wrong background; people of working-class backgrounds did not go into films. And b) the subjects I choose just wouldn't get made.

"I'm glad I'm not starting now because I think it's immensely difficult despite all the different ways you can make films. How do you make your first film? How do you learn your craft? I don't know where people learn the grammar. A lot of films you see it's clear they don't know the grammar."

But we now live in a world were gay marriage is allowed and welcomed, I persist. Isn't that something to be welcomed? "I have no idea. I wouldn't do it. It wouldn't be something that I'd want. I've lived on my own too long. It gets lonely sometimes but at the same time if you've been very, very busy I don't then want to go through the palaver of 'well, what kind of a day have you had?' I just want to lie down, light my fire and put something nauseatingly sentimental on my DVD and wallow. That's all I want."

He once told an interviewer that he has been celibate for decades. His sexuality has not been his drive, it would seem. "This obsession with sex all the time," he says. "Why are people so obsessed with it? I can't even imagine group sex because after it's over you'd wonder who to avoid."

Are there upsides to celibacy, Terence? "No, there isn't. But the alternative for me is much worse. I saw it in my own family, people being possessive and I couldn't be doing with all that. And I do think sex muddies the waters. I think it does. Jealousy, possession. I can't be doing with that because I don't think that's love. I think you can love without there being any sex. You love someone because you love them."

Maybe cinema is sublimated desire for you, Terence, I suggest. "Well, perhaps it is. But there is also the 19th-century politician who said of sex 'the pleasure is momentary, the position ridiculous and the results damnable.'"

For his passion then look to his films. He will soon complete his next film, about the life of the poet Emily Dickenson, with Sex and the City's Cynthia Nixon. Hopefully there will be more.

Are you a different person on a film set than in real life, Terence? "Well I do get very worked up. I do get very passionate about it, because I think it's thrilling. It's almost embarrassing really. But [on Sunset Song] very often I was unable to say cut. I was so moved."

What is this? This is love, I guess.

Sunset Song goes on general release on Friday.