One commentator tagged him "Manna from Havana".

Innumerable, often racy column inches have measured out his stature, both as a dancer and as a sex symbol. "Superstar" might as well be his first name, although "legendary" runs it a close second.

There are so many labels attached to Carlos Acosta, you find yourself wondering if he'll ever be able to jump free of them – his ability to jump with seemingly effortless elegance, mercurial speed and elevated height being a distinctive trait his worldwide fans dote on. But Acosta himself proves humorously philosophical about how he's become a magnet for the breathless adulation usually reserved for film stars and pop idols, and rarely accorded to male ballet dancers.

He'll joke about it, saying he doesn't go to the supermarket thinking "here I come, Carlos the Cuban sex missile" and expecting other shoppers to swoon in the aisles. But as he prepares to bring his own production, On Before, to Edinburgh – the only Scottish stop on the tour – he does have concerns that the labels, and most especially the audience expectations that go with them, might prove a stumbling block to what he hopes to achieve with a programme that's radically different from the classical work he has excelled in since his teens.

Acosta turns 40 this year, and he's aware of what time, and the rigours of classical technique, do to a dancer's body. Pain becomes a constant partner in the rehearsal studio, even if the on-stage performance shows no sign of aches and strains. Acosta is still delivering the thrilling blend of physical prowess and expressive artistry that saw him badged as the new Nureyev soon after he joined the Royal Ballet in 1998. If anything, his familiarity with the roles that cause box-office meltdowns – and the likes of Colas in La Fille Mal Gardee, Prince Siegfried in Swan Lake and Albrecht in Giselle never fail to attract long queues – has deepened the emotional palette that shades meaningful character into the steps.

So why, when audiences worldwide clamour to see him in the classics, has he taken time out to go in what seems the opposite direction with On Before? Indeed, why opt to task a balletically trained body with movement vocabularies that twist and turn with decidedly contemporary inflections? Sitting opposite me, somehow lending easy grace to the laid-back look of jeans and T-shirt, Acosta nods at this line of questioning as if he'd gone in search of the answers long before the routine of press interviews asked him for quotable explanations. His starting point lies in the necessity of finding new challenges.

"As an artist, I need to try out different things," he begins. "You have to evolve, embrace new experiences, learn from them in unfamiliar, unexpected ways. Then take back those different ideas, different tools into what you already know. I take what I discover back into dance because dance is a part of who I am, a part of what shaped me as a person. To learn, to evolve – and still feel, as I do, that there could be even more - I see that as a kind of freedom that can keep you from repeating yourself. And as an artist, that's one of the hardest challenges. Bringing freshness to what you already know almost too well, so that you don't let it slip into becoming boring."

He grins. We both know that "boring" is not a word that springs to mind when he or his work are under review. He's not deploying any "false modesty" tactics either. He freely admits to being ambitious, maybe even driven to achieve. And his roster of achievements is not only impressive, in dance-y terms it's atypical. Some high-profile dancers do shift over into choreography. And some fetch up in print, usually in a memoir that mixes personal reminiscences with morsels of back-stage gossip. A few will find themselves on-screen, more or less being themselves on film or TV. Acosta has done all that, and more.

Ten years ago, he wrote and choreographed the semi-autobiographical show Tocororo, which went on to break box-office records at Sadlers Wells in London. Three years later, his autobiography, No Way Home, was published and, with it, the pithy-poignant details of a hard-pressed early life in one of Havana's poorest quarters where running with the local street gangs might have been his lot if his father hadn't dragooned him into dancing.

Something of that personal backdrop – the intense vitality of Cuban culture, but also the harsh realities of surviving below the breadline – crept into The Day Of The Flowers, the feature film that marked his acting debut on-screen. If the film itself (given its world premiere at the 2012 Edinburgh International Film Festival) had a lukewarm reception, Acosta impressed as the tourist guide caught up in the emotional turmoils of two Scottish sisters in search of family secrets in Cuba. One reviewer homed in on his "commandingly feline poise" before adding "hopefully he'll find scripts more tailored to his skills and worthier of his talents".

Well, perhaps Acosta will pen those himself. Not such a far-fetched notion – his first novel, Pig's Foot, has already made it on to Waterstones Eleven, the bookseller's list of "debut literary stars of 2013". Acosta fans will have to wait until the autumn before they can read it for themselves: publication is scheduled for October. Given his other commitments – he's still appearing as a guest artist with the Royal Ballet and companies abroad while masterminding On Before – you do wonder how and when he finds the time to write more than Post-it notes. It transpires that he's spent the past four years creating Pig's Foot during breaks between rehearsals or during performances. And no, it isn't a fictionalised account of a ballet dancer's life. It's a caringly researched saga of one Cuban family that traces their – and the country's – history, back through five generations.

"When you spend so many years abroad," he says, "you have to keep remembering where it is you come from. It's important not to lose your true sense of identity, and I'm reliving this through my characters. Even though it's a fiction, it has brought me closer to my homeland and its history, made me understand not just the complexities of that place but also of its people. And whatever the social issues are in that history, you are still always looking into human nature."

Which, in a way, brings us back to On Before and the nature of audience expectations. "Ohhh...!" It's part-sigh, part-laugh. "I know that there are people who just want to see me in the tights – yellow tights for Colas, white tights for Siegfried or Romeo... And they won't see any of that in On Before. There are nine separate elements in this show with some new and some existing choreographies that I think are strong, but they are contemporary, even minimalistic in form. There are only two people dancing: me and Zenaida [Yanowsky]. No big ensembles – but there is a choir at one point.

"For me, it has a sort of narrative. I chose the pieces to create a dance journey that I think is moving and that I think offers something about love, death, maybe loneliness. But really, I think it is for people to come with open minds and make their own stories out of what they see. And part of what they see is a different kind of Carlos Acosta. Not the Acosta of double cabrioles or tours en l'air – but an Acosta who is asking his body, and his mind, to meet new challenges."

Among those challenges is Russell Maliphant's Two, an eight-minute solo that roots the dancer to a spot, or rather a small square of light centre-stage. Acosta instinctively flexes his shoulders as he recalls the rehearsals that tutored him into holding that initial stillness before sending his upper torso into nuanced convulsings that seem to chafe against some invisible forcefield. Elsewhere, On Before showcases works by choreographers who are not household names among ballet fans but whose influence on the contemporary scene is crucial to its evolution. He returns to that concept with a quiet intensity. "It is absolutely about evolving – myself as a dancer; the art-form in the 21st century; the audience, if they are willing to make that journey with me. I don't ever want anyone to think I have turned my back on classical ballet. But I won't always be that Colas, or that Prince. It's time to find the other dances in me."

On Before is at Festival Theatre, Edinburgh on April 26 and 27, www.edtheatres.com/festival