Some nights, Ellis Norrell dreams about Britannia.

He dreams he's back on board the ship, below decks, somewhere out at sea. "We're always at sea in the dreams," he says, "and it's normally rough conditions." He dreams about the yacht, he says, because of the length of time he spent working on her – 35 years. Sometimes, with others who served on Britannia, he visits the vessel at her berth down at Leith. "It's quite emotional sometimes to go back," he says. "It brings back memories. I lost my wife Grace six years ago and there's a strong bond between me and Britannia. Britannia was my second love."

Most of the yachtsmen who served on Britannia seem to feel the same way as Norrell, although his love is particularly deep because he worked on her for so long: from her early days until the late 1980s. Mention Britannia's decommissioning, though – on December 11, 1997 at the hands of then prime minister Tony Blair – and you'll get frustration, resignation, but also the hope that perhaps one day, despite everything that's happened, despite the holes that have been cut out in her hull to let the tourists in, Britannia might sail again. The Duke of Edinburgh certainly thinks so. He says the ship could have gone on for another 50 years.

For the past 15 years, Britannia has been in Leith, berthed a little ignominiously next to an ugly shopping centre and this Tuesday there will be celebrations of her 60th anniversary including special guests and fireworks.

There are many who see the vessel as an expensive, antediluvian symbol of royalty or an imperial past, but once you get on board it's hard not to become intrigued and surprised. For 44 years after her launch in 1953, this Clyde-built vessel was royalty-on-the-waves, the Queen's HQ round the world. The ship did more than one million miles and visited more than 600 ports in 135 countries.

And yet inside one of the most famous ship in the world, parts of it look distinctly unroyal. There is no bling, nothing vulgar. The Queen and the Duke of Edinburgh choose all the furniture and fabric and, although there is a piano where Noel Coward and Princess Diana once played, the main living area would probably remind you of your granny's living room. The Queen's bedroom is also tiny and twee.

Britannia's last captain, Anthony Morrow, tells me none of this is surprising. Britannia was a house more than anything. The Queen herself said Britannia was the one place where she could truly relax. "It was a home for the royal family," says Morrow. "They could go to Australia and then when the Queen came on board, she's in her home again."

For much of the time, it was a home for the 270-strong crew too, though conditions down below were basic and cramped. Until the 1970s, most of the staff slept on hammocks. All your belongings were contained in a cupboard no bigger than a suitcase. You would also often be away from your family and friends for anything up to six months. The only contact you had with them was by letter. No phone calls.

Ellis Norrell loved it, though. "In the early days, ships went abroad for that length of time," he says. "I couldn't call my wife but I adjusted, and coming back to her was like going on honeymoon again. It was good for my relationship with my wife."

Norrell, who is 79 now and lives in Portsmouth, clearly remembers his feelings when he first heard there were jobs going on Britannia. He joined the navy in 1949 and when a call when out in 1953 for volunteers to join Britannia on her first tour out of John Brown's Shipyard the following year, at first he thought he wasn't right for it. It was his father, who served on the V&A yacht, Britannia's predecessor, who encouraged him to go for it.

"My dad said, 'Put in for it, it's a good number.' I said, 'I drink, I smoke and I swear – I doubt I'd get on board the royal yacht.'"

In the end, though, he changed his mind, applied, and made a good impression. In 1953, he was flown back from Singapore, where was serving on a minesweeper, for a second interview. He got the job and was on board for her first voyage from Clydebank on Thursday, April 16, 1953.

His first post was naval seaman. It wasn't glamorous. His duties were dishwashing and dealing with the rubbish, and his patch was the giant incinerator below decks. If Britannia was in the Tropics, the heat in that room could become very hard to bear. The conditions in the living quarters were just as squeezed. "It was cramped in the old days, before the refit, but we got used to it. I was practically brought up with it. The bunks were fairly luxurious compared to what we had before, although we had to reduce the complement to accommodate everybody because you can get more people in hammocks than you can in bunks."

In all, there were around 270 in the crew at any one time, made up of 220 yachtsmen, 24 officers and, on royal tours, a Royal Marine Band of 26. If the royals were on board, the crew would expand even further to include up to 45 extra staff including 25 servants. The entire operation was run along naval lines and all orders had to be given by hand signal to preserve the tranquil atmosphere. Despite all this, says Norrell, the atmosphere was often informal.

Norrell himself was eventually promoted to the boat deck after several months on rubbish duty. His typical day would start at 7am when he was required to wash down the decks, then it was breakfast at 8am. The shift would finish at noon although there would be duty watches for the remainder of the day. For the crew, there was a roughly 50:50 split between work and free time.

Later, Norrell was promoted to coxswain, a job he did for 14 years, and as such was responsible for the crew's appearance. The rules on this were strict and down in the laundry there are various devices, including hat presses and shirt shapers, to keep uniforms spic and span. "When we were in plain clothes," says Norrell, "we had to wear a tie which was to stop us wearing T-shirts – we also weren't allowed to wear jeans. In the first years, we had to wear uniform all the time, wherever we were, but then the Irish problem happened and we could have been targeted so we wore plain clothes. If the Queen was on board, we wore uniform all the time. But there wasn't a formal atmosphere on board; it was quite informal."

It was friendly too, he says, although for most of the time the socialising on board was divided according to rank. There are several bars on board, some of which look like a typical British pub, and the officers' mess where the top ranks relaxed. There is also a huge state dining room decorated with some of the gifts the Queen received on state visits.

Over his years on Britannia, Norrell got to know the Queen and the royal family, particularly when he was promoted from coxswain to driver of the royal barge. He says the family were particularly relaxed when Britannia was in Scotland. "Any trip to the Western Isles was always informal – that was brilliant. The royals loved it there."

His relationship with the royals was always a fairly distant one, though. "You nod your head and say Ma'am," he says. "You kept your distance and showed respect obviously." Perhaps understandably, he has no time for republicanism and can empathise with the Queen's affection for Britannia. On board the ship, all the clocks are permanently set to one minute past three to mark the time the Queen stepped off her for the last time. "I know the Queen doesn't want to visit Britannia because it would be too emotional for her," says Norrell, "The Duke is the same."

The Britannia's last captain, Commodore Anthony Morrow, has a slightly less sentimental approach and has been back to the vessel several times since he led her final voyage in 1997. Morrow served on Britannia three times: once in the 1960s, once in the 1970s and then finally as Commodore of Royal Yachts for the final two years of Britannia's working life.

He is bullish in her defence and grumpily dismisses any idea that Britannia had become an anachronism. "She could be doing her work now just as efficiently," he says, "and could have been a major part of the diamond and golden jubilee – there were all manner of opportunities for the royal yacht's role to continue well on throughout the Queen's reign and even beyond.

"I think everybody was disappointed when she was decommissioned but it was a decision that was made at the time and we had to accept that, regrettable as it is. We have lost this great ship and great organisation. There's nothing to replace it at the moment, but don't forget: there's always a chance to have another one."

Both Norrell and Morrow claim, as defenders of royal yachts often do, that she earned her keep, that wherever she was, she was working on her dual roles as royal yacht but also as promoter of British industry and commerce. Occasionally, she would also perform other duties – in 1986 for instance, she rescued more than 1000 refugees from war in Yemen. "I don't think it was inevitable that Britannia would go because she was earning money," says Norrell. "I thought it was the wrong decision to decommission her – the hull is fine and still is."

As to whether Britannia could ever sail again, both men believe she could. Morrow says a survey last year showed the hull was sound for another 20 years.

Whether it is likely that she will ever sail again is another matter. There is a lot of affection for the ship, but she makes money as a tourist attraction (around 250,000 people visit her every year). Norrell finds it hard when he goes to see her. He remembers how hard it was to leave Britannia, which he eventually did in 1988 when he retired.

"I stayed because I was frightened of leaving," he says, "Britannia became a comfort zone for me." She is still his second love. n

For more details on the events being held to mark the anniversary visit royalyachtbritannia.co.uk.