Ministers should have no legal power to hire and fire senior college officials in the wake of the spy-pen row, Labour has demanded.

The party's call came at the end of a turbulent week for Scotland's college sector after claims Education Secretary Michael Russell had "bullied" Kirk Ramsay out as chairman of Stow College in Glasgow for making a recording of a ministerial meeting.

The Government has powers to remove college board members if the affairs of the institution have been mismanaged. Under new legislation, that right will continue – and additional powers will be introduced to allow ministers to make appointments to the chairs of powerful regional college boards.

Hugh Henry, education spokesman for Scottish Labour, said the proposals should be dropped.

Mr Ramsay accused Mr Russell of an abuse of power after resigning on Tuesday, after Mr Russell urged him to quit following the row over an audio recording of a meeting with Mr Russell and 80 college officials using a Smartpen.

Mr Russell will also write to all colleges informing them of Mr Ramsay's actions suggesting he was not fit to hold the post.

Mr Henry said: "If Michael Russell won't do the decent thing and resign we need at least to make sure the Education Secretary's power to bully and intimidate is curbed.

"The SNP Government has plans to give Mr Russell the power to hire and fire college chairs. That is at the heart of his power to bully and intimidate college leaders and must not happen. The power must be dropped from the SNP's legislative plans."

Mr Russell's spokesman described the latest attack from Mr Henry as "petty point-scoring".

"This is pathetic – after making 10 different resignation calls for just about every member of the Scottish Cabinet, Labour can't even stick to their latest demand for more than 24 hours. The First Minister put the record straight and it is time for Labour to accept that and start focusing on the priorities of the people of Scotland instead of petty point-scoring.

"The Scottish Government is supporting our young people at college, helping them into work through opportunities for all and working to improve further education across Scotland with new college buildings, better courses and increased support for students."

Meanwhile, the Educational Institute of Scotland called on all Scotland's politicians to refocus their attention on addressing the real issuesfacing the college sector.

Larry Flanagan, general secretary, said: "Scotland's colleges are facing very serious and very real problems and Scotland's politicians should be focusing on addressing these issues rather than engaging in party-political point-scoring."