Michael Russell called on Kirk Ramsay, chairman of Stow College in Glasgow, to step down after a recording was made of a meeting last month to discuss controversial reforms in the sector.
Mr Ramsay was asked to explain his actions to the Education Secretary last Tuesday at the Scottish Parliament.
According to Mr Ramsay, Mr Russell said he was outraged at the recording and said he wanted the chairman to resign. Mr Ramsay subsequently apologised.
But Mr Russell has since issued a letter to others who were in attendance at the meeting to express his strong disapproval of Mr Ramsay's actions, and to alert them to the fact a recording had been made.
Last night, Mr Ramsay said he was disappointed the minister had used his position in such a way to put pressure on him to resign.
In his letter, seen by The Herald, Mr Russell questions Mr Ramsay's suitability for a post on the board of Stow. Mr Russell's letter states: "It appears that an unauthorised audio recording was made of the event, without my knowledge or the courtesy of notifying those in attendance [by the chairman of Stow College, Kirk Ramsay].
"I'm informed that this recording has now been distributed by Mr Ramsay.
"I am afraid I do not regard Mr Ramsay's actions as consistent in any way with the protocol expected at such an event or of the standards I expect [of the chairman of a board of management of] any college."
Mr Russell said his conduct has caused him to question his confidence in Mr Ramsay in his capacity as chairman of Stow College and he had told him this. He added it is unprecedented that he should find it necessary to bring to their attention his disappointment concerning the conduct of the chairman of the governing body of a publicly funded institution. "I do so with deep regret," he said.
However, last night Mr Ramsay hit back with his own letter to college leaders expressing his disappointment"at the actions of the Education Secretary and accusing him of acting on the basis of partial facts.
He writes: "Mr Russell was making an important announcement as to college reforms and his comments were extremely significant in forming the future of the college sector, of which we are all part.
"Rather than take detailed notes of Mr Russell's speech, I recorded his comments. This recording was solely for my own use and for others who could not attend the meeting. It was no different to someone who takes a verbatim note of what was said.
"Indeed, the recording of the speech ensures accuracy and that the comments are properly understood."
The closed-doors meeting was seen as an opportunity for college leaders to speak candidly to Mr Russell about ongoing restructuring, which has led to a wave of mergers across the country.
It was controversial because some within the sector do not agree with the reforms and they are happening at a difficult time financially, with savage cuts to teaching budgets of £73 million, a reduction in student numbers and more than 1000 job losses. Colleges have also reported significant unmet demand, with thousands of names on college waiting lists.
But it subsequently emerged Mr Ramsay, former chief executive of the Glasgow Science Centre, had recorded the meeting and passed the tape to others in the sector not in attendance.
Mr Ramsay said: "I was surprised that Mr Russell had taken such exception to this, given his role as a public servant and his position in introducing the fundamental reforms to the sector.
"I did explain that I had shared the recording with a small number of people directly involved in the implementation of the college reforms to help ensure their clarity of what he wished to develop. No distribution took place beyond that limited group nor was that ever intended.
"I am extremely disappointed that Mr Russell has used his position to seek to exert such control, influence and power both privately, but also so publicly.
"I find his comments about me disturbing given the foregoing. It is regrettable that he has chosen to seek my removal in such a fashion, particularly given the difficult period of change we are all experiencing and the work we... are doing voluntarily, driven only by passion for the work of the college sector."
Stow College has previously courted controversy by pulling out of a merger which saw a number of Glasgow colleges join forces to become the City of Glasgow College.
Despite pressure from the Scottish Funding Council to join the merger, Stow decided to stay as a single institution, although though it is now considering merging with other colleges in Glasgow.