It will irk him for the next three-and-a-half years. Such is the territory that comes with possessing an Olympic silver medal.
Not that the Helensburgh sailor's achievement at London 2012, where he finished second to the 2011 and 2012 world champions in the 470 two-man dinghy event with Rochdale's Stuart Bithell, should be glossed over.
Standing on an Olympic podium was easily the proudest moment of the Scot's life. But it is just that, well, there is an impatience about Patience. Rio 2016 is already on the horizon and there is a determination to turn that silver into gold.
Well before the Olympic celebrations died down – and the 26-year-old admits they extended to several months – Patience and Bithell had to come to the most difficult decision of their careers. A partnership seemingly made in heaven (within two weeks of coming together in 2009 they won a silver medal at the World Championships), was dead in the water.
Patience had colourfully stressed that the pair would be together for the next Olympics "providing we don't die in a plane crash", but was forced to revise that thought. It was only days after they clambered down from the podium last summer that they discovered the light winds around Rio for the next Games meant the pair would likely be too heavy to make a realistic bid for gold.
"It was a no-brainer factually but it was very difficult," said Patience, who had been rivals with Bithell in the 420 class before they teamed up. "Ten or 12 days after the Games I was sitting down with Stephen Park, the Olympic manager, and Stuart in the knowledge that this was a chat about moving forward. We were just getting some guidance about how we might go about making a decision and we talked at great length about what it would take to win in Rio. Weight and height are a big deal in our sport and we looked at the stats and Stuart and I knew we couldn't do it together.
"We didn't make the decision there and then but decided to get more facts and look at it a bit more but I think we knew in our hearts the whole time what the conclusion would be. It was hard. He's one of my best mates and what a ride we've had together. In these two-man boats you are 'married' and effectively live together. I spent more time with him than anyone else in the world and we were on a similar wavelength.
"We could've stayed together on the boat. We're quite marketable lads, we could have found ourselves some lovely sponsorship and milked the next four years. We might have even dominated the circuit rather than just being one of the best but, in the one week it actually mattered to us, we'd probably have been beaten because of the weight disadvantage.
"It wouldn't make you a bad guy to choose that route but that's not why we do it. We do it to win and we made the hard choice and the right choice. At the top of any sport, it's the marginal moments and inches and millimetres that make the difference and in this case it was just a few kilograms. I'm not saying it would have been impossible but we knew we would always be at a disadvantage and know that four years on, if we didn't win we'd never forgive ourselves. We weren't prepared for that to be the case."
Weymouth-based Patience has started working with a new partner but his identity will not be revealed until next month's Boat Show. It is still very early days with the Scot only back in a boat with any purpose since early last month, but the 2013 World Championships are looming in La Rochelle in France in August.
However, this year was not going to be results-based, he said, as everything was geared towards Brazil in 2016 with the aim of bringing home a gold medal.
"My life has changed forever [since the Olympics]," said Patience, who will be in Largs next weekend attending the Royal Yachting Association Scotland's Big Weekend. "It comes down to the fact that there's life before your first Olympic medal and life after it. I'm sure with a gold medal, it would be different again. None of us do this to come second, we all do it to win and so, in four years' time, if all goes well and I can win a gold medal, I'll be saying that there's life before gold and life after."
Patience first took to the water at Rhu Marina aged seven, after being encouraged by his parents, and he has been living, breathing and dreaming every day of his life for the last 15 years of competing at the Olympics, but London turned out to be a bittersweet experience.
"To have finally gone to the Games, finally to have ticked that massive life goal off, and come so close to the ultimate goal, is a strange experience. Not winning was an anti-climax," said Patience, who was the first Scot named in the Great Britain team for London 2012.
"You are totally overwhelmed by the Olympic experience and so proud of yourself. There's a huge sense of relief when it's over but, on the back of that, comes – and I can only talk from a silver medallist point of view – the "what if?" moments. That's really the downer that you experience. And you experience the downer of adrenaline; you've been working your butt off for so long and there is a come down.
"But I have absolutely zero regrets. I can look back and say we did it right. I'm not saying we didn't make mistakes, of course we did. If we'd made no mistakes, we'd have won every race and made history.
"But certainly there are some great learning points to take forward and things I would do differently. At the Olympic Games, you can't say you are going to win as that is not in your control. We went to the Games and had a personal best, we'd never scored a regatta that low in our lives. We'd never raced that well. The Australians [Mat Belcher and Malcolm Page] just did it better and I have to take my hat off to them.
"I went there and was ready for the fight, because that's what sport is, it's an absolute fight to the death. It's a battlefield but it was just the most wonderful regatta I'd ever had. It went so smoothly. We made history with the Australians in that we had the lowest scoring Olympics in the 470 ever. Our points were low enough to win by miles any gold medal in any previous Olympics."
Patience has now moved on and there is little time to dwell on his success or on what-might-have-beens.
"You see the likes of [cyclist] Chris Hoy on the podium with yet another gold medal; he's been there and done it so many times and yet he's still in tears and still has the emotion running through him. It made me think when I watched him, and when you see the likes of Roger Federer winning Wimbledon again, that maybe this really doesn't sink in until you retire. While you are still a sportsman, it's just never good enough. You aspire to reach perfection all the time and you're always on that treadmill so you never get time to relax."
Retirement is certainly not in Patience's plans any time soon, so Greenpeace, whom he says he would like to work for if not sailing, will have to wait for his services.
"When I do retire, hopefully I will have many memories to look back on and not just the Olympic silver medal," he said.