NEWS in from the heartland of British contemporary music: James Dillon has declared himself an instinctive musician. “I count on something happening that is not deliberate,” the Scottish composer told an audience in Yorkshire the day after his latest major work, Tanz/haus, opened the 40th edition of Huddersfield Contemporary Music Festival.

If Dillon had spoken these words just 24 hours earlier, the statement would have come as a much wider curveball. The 67-year-old Glaswegian is known as a fantastically cerebral composer, a maker of arcane musical universes that layer up lost centuries of visual, philosophical and linguistic references. His works are impressive, unapologetic, full of complicated intrigue, but they are hardly easy access. Now here he was, revealing with quiet geniality and a new bushy beard that there is gut – gut! – behind all that erudition and obscurity. “The dialectic between the ear and the imagination is a very mixed up business,” he admitted.

I remember the composer James MacMillan once telling me that he started voting Conservative because he had “grown up”. It seems Dillon’s ageing process is having a more compassionate effect, though the implacable composer portrait in the festival programme didn’t forewarn of any major temperamental changes. (Imagine a photo so wilfully underlit that it’s more murk than face). I’ve long admired the uncompromising intellectual conviction of Dillon’s music, the depth of his engagement with, say, the art of Rogier van der Weydan, or some forgotten medieval Latin text, or the poststructural feminism of Julia Kristeva. But I’ve also wondered whether he purposefully cultivates an aura of inaccessibility.

Six years ago, before the Huddersfield premiere of his Oslo Triptych (that particular score circled around quotations from Virgil, Coleridge and Apollinaire) he admitted to there being an element of provocation in the extreme technical challenges he writes into his music. I’m talking about the kind of challenges that have made him infamous, and not always loved, among the instrumentalists who have to actually play the notes. “It’s about demanding concentration and engagement,” Dillon told me back then. “Players are so highly trained these days that they easily drift into a comfort zone.” He might as well have put a health warning at the top of all his scores: there will be no comfort zones here.

And so his latest work, the expansive 40-minute Tanz/haus, came as a revelation for its gentle spaces, its tangibility, its moments of outright warmth and melancholia and sweet humour. The instrumentation is a nine-piece ensemble that includes electric guitar, clarinet and accordion – a beautifully mellow combination which at times verged on languid surf-rock or slow-fi lounge jazz or 1960s psych-folk. The title is a typical play on etymology: ‘tanz’ is the German word for dance, with traces back to the Frisian meaning to ‘tremble’ or ‘quiver’, while the word ‘haus’ suggests, if you trace its Indo-European roots, something hidden or secret which Dillon related to “the way I conceal technical things that are going on in the work”. The title, he went on to explain, becomes a forcefield for itself.

Bass hooks surface then re-submerge; sometimes the momentum feels nervy and abstract, drifting and obstinate, sometimes all trills and shivers. Dillon still plays his mercurial trick whereby he lets us glimpse a beguiling musical character then slips it behind a gauze made up of multiple other sounds, but now those retreats feel nostalgic rather than pointed or guarded, and the places in between seem to accept the human mess that exists between order and disorder. Dillon’s writing has probably had moments of instinct all along; what seems new is that he’s less draconian about censoring himself, or simply more willing to embrace what he calls “the mixed up business”. I found the results captivating and unexpectedly moving, but you can judge for yourself: Tanz/haus was broadcast on BBC Radio 3’s Hear and Now on 18 November, and it’s available online for another three weeks.

The premiere was delivered with superb focus and intensity by Scotland’s top contemporary music ensemble, Red Note, who seem incapable of being fazed by anything and who add their own wit and bright spirit to whatever they tackle. Later the same night they performed again, this time taking on a sort of obstacle course around deconstructed accordion-type-things. The piece was called Aeolian, conceived by Norwegian composer Maja SK Ratkje and installation artist Kathy Hinde and involving bellows of many shapes and sizes, tubes with whirly propellors, balloons that inflated and deflated, wind chimes and kazoos. What it lacked in musical substance was made up for in low-key playfulness, and the audience, surely among the toughest in the UK, cracked genuinely discernible smiles.

It’s probably a sign of maturity to not take yourself too seriously, at least not all the time. At 40, HCMF is ageing well, and the programme this year – the 12th under director Graham McKenzie – was a happily-balanced thing with proper breathing room around the edges. The heavyweights remained at its core: the Dillons and the Brian Ferneyhoughs (whose dense and elusive 45-minute piece Umbrations was played by the fearsome joint force of the Arditti Quartet and Ensemble Modern). The festival also made plenty of space for the pristine, quiet expanses of Linda Catlin Smith and Jurg Frey, composers leading a current contemporary music trend for the slow and the gentle.

On various other ends of various other spectrums: the excellent Kit Downes cranked brazen exploratory sounds from the organ of St Paul’s Hall ahead of his forthcoming debut solo release on ECM; Berlin ensemble Zeitkratzer rollicked through loose, grainy instrumental versions of Lou Reed’s avant-garde noise album Metal Machine Music and early Kraftwerk tracks; a stunningly poised Pauline Oliveros piece for string quartet and tape was given a rare outing in an industrial mill building, which was also where the intrepid Scottish improvisor Lauren Sarah Hayes wielded her deft little analogue/digital sound-making handset. Folk singer Sam Amidon teamed up with a terrific bunch of jazzers for a brave, roving set of fractured bluegrass, and the festival closed with strobe lights and techno sub-bass via Alexander Schubert’s Subramodal Parser – a full-sensory nod to hedonism, escapism and Berlin club culture.

Many of the artists I just mentioned reference experimental rock or jazz or electronica or whatever more than they do classical traditions. Maybe the point is those genre definitions don’t really matter any more, or that they are too mixed up to mean much. Mixed up is how we listen these days. And as Dillon demonstrated, mixed up is really no bad thing.