Stow College chairman Kirk Ramsay said it was "outside the law" for a minister to interfere with his organisation and insisted he had done nothing wrong.
He added that other individuals at the meeting, to discuss restructuring in the further education (FE) sector, had also taped proceedings. The sector is facing savage cuts of £73 million to teaching budgets, a reduction in student numbers and more than 1000 job losses.
Russell called for Ramsay to stand down after learning the chairman had recorded the meeting he had addressed last month with senior college figures. Ramsay then circulated the digital pen recording to fellow board members and other college chairmen.
An angry Russell summoned Ramsay to a meeting last Wednesday to quiz him about the tape.
He followed this with a letter to others who were in attendance at the initial meeting, in which he criticised Ramsay, saying: "His conduct has caused me to question my confidence in Mr Ramsay in his capacity as chair of Stow College."
However, in an interview with the Sunday Herald, Ramsay said the decision to tape the meeting was "not untoward at all". He said: "Any of us are perfectly entitled, in those circumstances, to make a record of a meeting, by whichever way we choose."
He said there were between 80 and 100 people present, adding: "The reason for me wanting to ensure I had a good record of what he was saying was [that] there was some critical information he was providing that I, as a chair, would have to take into consideration in strategic planning."
He described his session with Russell on Wednesday as an "ambush meeting", adding: "He immediately launched into an angry attack, in which he expressed his outrage ... He then said quite plainly that 'if I had the power to remove the chair of the board of a college, I would remove you immediately. You must consider your position, I expect you to resign'."
Ramsay then quoted Russell as saying "if you don't do that [quit], I will write to all the people who were at the meeting, and tell them what you did. If you do resign, I won't write to them."
The chairman said of Russell's approach: "That sounds like coercion to me."
Ramsay said of the college-sector meeting: "I noticed a number of people using iPhones and iPpads in various ways to record the proceedings."
The chairman also criticised Russell's intervention: "The key issue at hand here really is that Mike Russell is a Cabinet Secretary. He's actually precluded from interfering in the business of the board of an independent public body. It is outside of the law for a minister to do that."
Liberal Democrat MSP Liam McArthur said: "Mr Russell is exploiting this honest error for his own political ends. He is infuriated that someone has had the temerity to ask questions about the Scottish Government's reforms and now wants to remove them from their position by any means."
A spokesperson for Russell said: "Mr Ramsay did not request permission to record any of the participants in the meeting. He used a device which was not obviously a recording device and did not indicate to the meeting that he was undertaking a recording.
"The Cabinet Secretary has made it clear he has no power to dismiss Mr Ramsay but it was his clear duty to indicate to him that his behaviour fell well short of that expected from a publicly funded College chairman. The Cabinet Secretary has no dispute with the board of Stow College but only with Mr Ramsay, in whom he has lost all trust."