When Rae was out on the course with a friend at their local club, the very mention of Green was enough to disrupt his game. A former Rangers player and lifelong fan, Rae was never comfortable with the way Green took control using grandstanding and gruff charisma and then left behind a club with financial concerns.
The consequences of Green's reign - after the consortium he led bought the business and assets of Rangers Football Club plc in administration - are being felt now. They are worse than Rae feared. Graham Wallace, the current chief executive, is conducting a review of the entire business, and cost cutting is required before fresh funding is sought to invest in the football operation and to try to prepare it for a return to the top division.
In a sense, Rae's view has been vindicated, although that brings him no satisfaction. He used to vent his frustrations during five-a-side matches with friends, although it affected his golf game more significantly. His anecdote is flippant but it reveals the extent of his ire. "I used to play in a medal with my mates," Rae said. "My best mate's a Celtic man and he used to get me on the third or fourth hole and say, 'they're going to put statues up of Charles Green', and my head would go. I used to post terrible scores, because I used to start arguing with him on the course. He was loving it.
"[Green's] legacy is a shambles. The only thing he did was see a really good opportunity. There were allegations he was involved with [Craig] Whyte initially and he turned him over as well. Unfortunately he has walked away with a fortune.
"He never sat right with me. He came out with good soundbites but there was never any direction, never any plan in place. I remember his behaviour at Brechin [when Green stood and clapped as Rangers fans chanted against the SFA]. I couldn't believe what he was doing.
"I consider Rangers to have been class throughout my time and he wasn't representing the club in the manner that role should demand. There was no doubt he was in it for himself. It was a mad ego trip for him. One thing about football fans is that they want to believe in the people in charge. They want to believe that they are going to take the club forward. Unfortunately, this guy has duped everyone."
Rae does, though, have faith in Wallace to address the problems that Rangers now face. Sources at the club have dismissed rumours that Wallace was on the verge of leaving, and the chief executive insists he is fully committed to the job. He also maintains the club are not on the verge of administration.
At some stage, Wallace will present his plan to the board and approval or otherwise will be a pivotal decision since a chief executive needs to know he has the full backing of his board. Rae is confident, though, that Wallace can steer the club on to safer ground.
"The previous CEOs have made wild statements and nothing's ever been backed up," Rae said. "This guy's silence, his taking stock of the whole situation, bodes well for us. He seems to have a plan. He'll assess it in time. This is what has been lacking going back to Martin Bain's days. When you look at all the departments within Rangers, there was no guidance, no direction, everybody was on their own.
"This guy seems to have some sort of direction which he wants to take things in. I wouldn't like to see him go. There are other elements of the board that some people are not quite sure of, from a fans' point of view, and that is worrying."
Rae has sympathy for the Rangers manager, Ally McCoist, who has had to work under a succession of difficult circumstances and, on some occasions, moments of crisis. As a former manager of Dundee, Rae understands the demands of the job even in conventional situations.
He would like to return to coaching or management as soon as possible, though, despite the challenges and despite having just lost his job as Paul Ince's assistant at Blackpool, via a text message from the chairman, Karl Oysten. Ince has been replaced by Barry Ferguson, the former Rangers midfielder, who was part of Ince's squad. "To be sacked by text was classless," Rae said. "We never quite got the support I felt we deserved, having taken the club so far. We'd had the best start in the club's history, after having kept them up [last season]."
Rae hopes Dundee's new manager, Paul Hartley, will be given more time in the job than he and Ince had at Bloomfield Road. "Paul is a decent appointment but I was a bit perplexed to see another Dundee manager sacked, especially when they are joint top of the league," Rae said. "That's six managers in eight years. At some point, they need to change that cycle and get a bit of continuity. Dundee are a good club and they gave me great support for a few years. It would make sense to give Paul a bit of time."