This week, the BBC Scottish Symphony Orchestra opens its season under outgoing chief conductor Donald Runnicles with one of the most savagely personal and philosophically thorny works in musical history: Mahler’s incomplete final symphony, his Tenth.

There’s no avoiding autobiographical dirt when it comes to the Tenth — the piece emerged out of a domestic crisis so traumatic that it set Mahler’s entire life on the brink. Sometimes the depth of its musical heritage gets eclipsed by that salacious melodrama, but the piece is the culmination of two decades of blazing symphonic experimentation through which Mahler exploded the form and synthesised a new amalgam of the philosophical and the personal, the elemental and the archetypical. Back in the late 1880s he had opened the door with his First Symphony and in came the whole world: in came street tunes and klezmer bands, blazing expressionism and dizzy elation, earthy brawn and satirical bite. And, of course, the stormy existential narratives of his own personal life, writ large across that epic canvass and the eight complete symphonies that were to follow.

When he reached his Ninth, Mahler was terrified. Bruckner died while writing his Ninth Symphony and Beethoven died just after writing his. “It seems that the Ninth is a limit,” Schoenberg wrote. “He who wants to go beyond it must pass away. It seems as if something might be imparted to us in the Tenth which we ought not yet to know, for which we are not ready. Those who have written a Ninth stood too close to the hereafter."

Mahler gives us a glimpse of that hereafter, but Schoenberg was right: the Tenth is so desperately, explicitly intimate that it sometimes feels as though we ought not to know. He drafted it in the summer of 1910 at his home in the Dolomites. At the end of July, he opened a letter that he wasn’t supposed to read — a love note to his wife Alma from the architect Walter Gropius. The two had been having a secret affair and the news sent Mahler spinning. He went to consult Sigmund Freud, who sent him home with a diagnosis of the Oedipal complex.

The last three movements of the symphony are annotated with gallingly candid quotations: “Have mercy”, or “You alone know what it means”. The biblical “Oh God, why hast thou forsaken me!” appears in the central movement; later comes the fervid cry — “To live for you! To die for you!”. In the final bars Mahler simply writes the name “Almschi”. Alma herself later described these markings as “outcries and ejaculations addressed to me”.

Turns out that Mahler had reason to be superstitious over his symphonic numerology. He never completed the draft of the Tenth, but died in 1911 with five movements in various stages of development. The first is nearly fully orchestrated, the second only half. The third — initially titled “Purgatorio or Inferno”, with the second word later crossed out — has only 28 fully orchestrated bars, while the fourth and fifth movements contain only a few notes on instrumentation.

Some believe the piece should have been left to rest at that. Certain conductors (though increasingly few) still refuse to perform anything other than the notes Mahler himself wrote. The philosopher Theodor Adorno argued the Tenth should remain the preserve of the cognoscenti: “precisely someone who senses the extraordinary scope of the conception of the Tenth ought to do without adaptations and performances. The case is similar with sketches of unfinished pictures by masters: anyone who understands them and can visualise how they might have been completed would prefer to file them away and contemplate them privately, rather than hang them all on the wall."

For others, the drafts were too tantalising. The first and third movements were performed in Vienna in 1924 in a completion by the composer Ernst Krenek, and after the Second World War work began on more extensive reconstructions of the whole thing. The most important version came about almost by accident. In 1960, a diligent British musicologist and broadcaster called Deryck Cooke was asked to prepare a pamphlet on Mahler to mark the centenary of his birth. Cooke didn’t feel able to write about the Tenth without looking at the manuscript, and when he looked at the manuscript it began to dawn on him quite how feasible a full transcription might be.

“There was a blueprint for basically every bar,” explains the composer Colin Matthews, who was 13 when Cooke’s modestly self-described “performing version for the draft of the Tenth Symphony” was first broadcast by the BBC at the end of 1960. “I became completely obsessed by it in an embarrassingly adolescent way. I obtained a facsimile of the original manuscript from the library and began making my own transcription. I have no doubt that working on the score in that detail, getting to know it inside out, was what made me into a composer.”

Alma was suspicious of Cooke’s performing version and forbid performances — until she actually heard it in 1963, that is, at which point she lifted her ban. When Cooke was preparing for its Proms debut the following year, young Matthews plucked up the courage and got in touch. “With the presumption of a schoolboy, I told Deryck which parts of his version could be improved. Instead of telling me not to be so rude, he wrote back and suggested we had a meeting.”

Matthews would spend much of the next decade working with Cooke on the Tenth, painstakingly revising the orchestration of every page. “It’s extraordinarily private stuff,” he describes. “It’s the most autobiographical symphony Mahler wrote. It’s also critical that we hear it. People often think of the Ninth as his last, ending in such utterly bleak despair, but actually the Tenth was his final word. It comes out the other side of that darkness into a love song, full of redemption.”

Critical that we hear it, but Matthews also stresses (as Cooke always did) how much we should remember the work’s motley provenance. “Mahler wrote the draft incredibly quickly. It took him just eight weeks, not to mention the fact he was simultaneously preparing for the first performance of his Eighth Symphony and dealing with the potential breakup of his marriage. Making that trip to see Freud must have given him a fair bit to think about, too. Basically this draft is an outpouring of early plans for the symphony.”

Who can say what would have changed had Mahler lived long enough? The Tenth surges with near-irredeemable despair, a volatility bordering on madness. Its harmonic language skirts a new realm somewhere between tonality and atonality. How could anyone anticipate what he would have done in terms of its orchestration, its formal development? Mahler was a chronic tinkerer, still making changes to his First Symphony for a performance he conducted in New York in 1909. Or take the differences between the initial and final drafts of the Ninth; the central movement, ultimately the very emotional heart of the work, was only sketched in the margins of the early manuscript.

“I do still get plenty questions about the Tenth,” says Matthews, and he laughs when I suggest he has become de facto portal to Mahler himself. “As long as we remember what we’re listening to — that Mahler was in such a very turbulent state when he wrote it, that a huge amount would have changed, that the idea of ever being able to predict Mahler’s thoughts is laughable — as long as we remember all of that, then I believe Deryck’s version stands up strong.”

Donald Runnicles conducts the BBC Scottish Symphony Orchestra in Mahler’s Tenth Symphony at City Halls, Glasgow, on Thursday and the Usher Hall, Edinburgh, on Sunday. Colin Matthews introduces the work with a preconcert talk at City Halls