When Javier Cercas was a teenager, he used to look at the local hoodlums in his home town with a mixture of fear and admiration.

"I saw them all the time, they were everywhere. I was scared of them, of course, because they were dangerous, but I admired them because they were free, they went with beautiful girls, they had money." These swaggering youths were small-time criminals - sometimes worse - but they had an allure for well-brought-up middle-class boys like him. And in the late 1970s, after General Franco's death, they flourished, enjoying a short-lived but glamorous notoriety.

Cercas was 13 when, in 1975, Franco died and Spain began its transformation from dictatorship to wealthy democracy. Now one of Spain's most acclaimed novelists, he vividly recalls these years, which were both exhilarating and terrifying. Softly spoken, his English fluent and colloquial, Cercas is by turns earnest and humorous. Even at a distance of a thousand miles and more, his laugh is infectious.

"When people tell me there's no difference, democracy is like dictatorship in disguise, I simply think that is bull****, totally bull****. Dictatorship is dictatorship. When you have no freedom, when you are living in that regime, you feel also like s***. So I remember perfectly the smell of that time, of no freedom."

Cercas was born in 1962 in the province of Extremadura, near the Portuguese border. Even for those with good jobs, life was hard, and when he was four his parents moved to Catalonia. "I'm from the south, which is very typical of Spain, because in the 50s and 60s a lot of people who came to Catalonia and the Basque country are from the south, because the south was extremely poor. My father wasn't so poor, he was a vet, so at home we could eat and things like that, but in the south people felt there was no future, not even for a veterinarian."

The place they settled in was Gerona, a small town north of Barcelona, where today Cercas is professor at the university, though he teaches only occasionally. It is also where Outlaws, his evocative and mellifluous new novel is set. When we speak, however, he is not in Barcelona, as I had been led to expect, but in his country house near the French border, in the Costa Brava, with his wife and son. One can sense the heat. It is mid-evening, and from the windows of his white-walled study on the first floor, whose floor he tells me is strewn with papers and books as he finishes his new novel, he can look across his garden, and its trees and swimming pool. He has been living here rather than the city for the past few months, writing. "It's quite nice," he says, understatedly.

Cercas's circumstances are very different now from that of his childhood, a complicated era he can't consign to history. Indeed, he appears especially keen to write about it because most people, he says, have forgotten it. After Franco's death, he says, which is when Outlaws begins, "the sense was of danger. A mixture of danger and hope. I was a kid, but I know that at home my parents were a bit scared. What was going to happen?" But as well as uncertainty and fear, there was also anticipation: "a sense of, this is new, this is democracy, what is going to happen? I was a child, an adolescent, these feelings were really vivid."

Into the space left in dictatorship's wake the delinquents came to the fore, and it is the story of one imaginary gang leader, Zarco, that Outlaws tells. An effortlessly readable but layered work, it is told mainly through the voice of a middle-class boy who is drawn into Zarco's gang, lured by love for a girl he believes is Zarco's girlfriend. He will hold that belief for 30 years, as the story unspools, and with it Zarco's unhappy fate.

"I would say the book talks about something very peculiar of the Spanish moment," says Cercas, "and also something very universal. I mean, precisely at that moment, 1976-1978, it's a moment when there were more adolescents, it was called the baby boom. I am one of the baby boom boys. There was a lot of young boys like that, really poor boys of the outskirts of the cities, boys without roots, because most of them came from the south, from little villages, without education. They had no hope so they took arms, they organised themselves in bands, and they began doing what Zarco's gang does.

"The most important thing is that they captured the imagination of the whole country. They became myths, they became heroes, there was at that moment a very strong subculture that created these myths. Some of the most famous films in Spanish history are films about these boys. The main characters were these same boys, the real delinquents. The media was full of stories about these kids."

Yet the fascination these youngsters held was fleeting. Why was it so ephemeral? He sighs, with obvious regret. "Well, most important, the country changed very fast. In 85-87, with Spain getting into the European Union, the economic change was really fast, and democracy really entered into the country. But the main reason was that most of these kids died. Died because of the violence, of course, died because of heroin, which was the war of my generation, and died because of Aids."

The mythology that grew around these doomed gangs was, he believes, symptomatic of Spanish society in that unsettled period. "It is eloquent on this mixture of fear and hope we had at that time. This is the same feeling we had for those boys ... In a sense, the book is trying to see what was in these boys, to deconstruct this myth, which has by the way been totally forgotten. It's not in the history books. Right now there is a book by a historian saying it's being re-examined because of my work."

Cercas hates being called a historical novelist, because he dislikes historical novels, yet his fiction in recent years has been strongly influenced by the need to explore his country's past, dirty though it is. In so doing, he has become one of Spain's most popular and respected writers, winning international as well as Spanish acclaim.

Making a living as a teacher and a columnist for El Pais - which job he retains - he began by writing satirical and comic novels, such as The Motive and The Tenant. Then he wrote the book that would dramatically change his career. Soldiers Of Salamis (2001) was the tale of a real-life Falangist writer who, during the Spanish Civil War, was spared from death by a Republican soldier. Cercas's handling of the complexities of that conflict was so subtle and affecting, the book became a bestseller. The historian Antony Beever wrote that the author demonstrated an "emotional intelligence that quite often historians who rely on documents are incapable of, and he never corrupts history. He's not putting words into the mouths of historical characters." More simply, Mario Vargas Llosa called it "magnificent".

Cercas followed this with Anatomy Of A Moment, a quasi-factual account of the famous failed coup d'etat in 1981, when Lieutenant Colonel Tejero and his Civil Guards took over Parliament. Only three men remained standing when Tejero fired his pistol, one of whom, the Falangist prime minister Adolfo Suarez, became the central figure of the book. By focussing on Suarez, Cercas was trying to understand his late father's Francoist outlook, which as a young leftist had riled him. As he once admitted, the book was perhaps a way of saying he was sorry.

"He was very modest, Catholic, totally devoted to his family - not as ambitious as I am. He had five children and he wanted to bring them up. I understand I'm not better than him. I thought I was, and now I don't."

In both books, Cercas has addressed issues the Spanish would have preferred to forget. In so doing, however, he reignited debate, and for some it has allowed a fresh, and healing exploration of old wounds. Since he is clearly drawn to the difficult or the controversial, does he write about such subjects partly out of a sense of obligation to his country's past?

"No. I don't work out of duty. I work out of very selfish interests. Meaning obsessions. Things that are obsessive to me, that are important to me, that are in my guts. Of course, that doesn't mean I don't believe in the importance a book can have for society. I am an old-fashioned man, you know, I believe literature can change the world, can change the perception of the world the reader has. That's the main goal of serious, great literature. And like other writers I want to write serious, great literature. The result is another thing. But I strongly believe you can only transform the reader and write great books if you write out of personal interest, out of your guts."

While recent history is the hook for much of his work, it is not historical fiction as most of us understand the term. On this subject he is eloquent, as if he has often had to explain what he does.

"I write on the past but I write on the past as present ... We live in a sort of dictatorship of the present. You agree with that? Great! In the sense that people think the present can be explained, can be understood, only in the present. People think the past lives in the shelves of the libraries, in the archives, that it has no relationship with the present.

"I think this is a result of the enormous power, the overwhelming power of the media today, because they want to be fast, we talk about what is happening now and now and now, present, present, present. Of course the media have things that are extremely good, but this is not very good. This idea that only with the present can we understand the present is totally false. The past is a part of the present. The past is a dimension of the present. Without the past it is impossible to understand the present. The past is not even past, as Faulkner said. I don't remember exactly the quote, but he says it's not even past. It's happening now."

The book that came after his breakthrough, however, was entirely different, turning the lens back on himself. The Speed Of Light, published in 2005, was about a novelist whose life is destroyed by unexpected fame. Was it written to exorcise Cercas's own fears after his sudden rise to literary stardom? He sounds serious, as well he might. "When I published Soldiers Of Salamis I was totally unknown as a writer. Only my mother read my books, and some friends. It was normal, I wasn't complaining about that."

What scared him about suddenly being well-known, he says, was the fear of "becoming simply stupid. Fame is nothing. It's okay if you can spend your whole time doing what you want to do, which in my case is writing, which is wonderful, and having readers. But the rest is bull****, the rest is nothing. To become famous is simply ridiculous, being a public figure is simply ridiculous."

He gives a mischievous laugh. "I always remember a friend of Umberto Eco saying to him, 'Umberto, every time I don't see you on television, I feel you are more intelligent.' I think this is a way of losing time. Every time people are not appearing in the media, I feel for them more respect ...

"You should remember that great writers have been killed by success. I mean Salinger for instance, or Juan Rulfo, one of the greatest writers in the Spanish language, Mexican, he was on the verge of stopping writing. And I was on the verge of stopping writing."

He takes a deep breath and one can picture him looking out of his window, at the darkened garden, before he replies. "That book was a form of saying, success is nothing and I am going to go on writing, because it's what I love."

Outlaws is published by Bloomsbury, £16.99. Javier Cercas is at the Edinburgh International Book Festival on August 17.