Scott Kilday made the accusation at an employment tribunal in Glasgow where he is claiming unfair dismissal against Miss Mone's MJM International.
The 35-year-old also said his relationship with Miss Mone had latterly become "tense".
Mr Kilday was giving evidence for a second day at the hearing in Glasgow.
Ms Mone lost control of MJM International to MAS Holdings in after following the break up of her marriage. She now has a 49% stake.
The hearing had earlier heard how Mr Kilday left the firm in East Kilbride, South Lanarkshire, earlier this year after discovering his office had been bugged.
It heard a listening device had been hidden in a plant pot on the authorisation of company chairman Eliaz Poleg.
Mr Kilday, the former operations director, had earlier been asked to resign by Miss Mone pending another company, MAS Holdings, buying into the firm.
The hearing was yesterday told he had "trust issues" with Miss Mone.
Tribunal panel member Margaret Nelson asked: "What was your relationship by that time? She did not appear to be too bothered about you leaving the company."
Mr Kilday replied: "I suppose it was pretty tense. The fact was I had been promised so many things by her and they did not materialise."
The hearing was previously told how problems had arisen in the company after Miss Mone split with her husband and business partner Michael in December 2011.
Mr Kilday said: "Michelle is a very difficult person to work with.
"When it was Michael and Michelle, Michael was able to control her a wee bit better, but latterly that was also proving more difficult."
It emerged Mr Kilday now works for Michael Mone's Pendulum Apparel firm.
Under cross-examination, lawyer Alice Stobart for MJM asked Mr Kilday: "Mr Poleg's evidence will be that he was concerned you were maybe going to join Michael because he knew Michael was setting up this business.
"He asked 'were you going to work with Michael?' and at the time you said no. Do you recall that conversation?" Mr Kilday: "Yes."
Later, the man who authorised the bugging said he thought Mr Kilday was part of "Team Michael."
The hearing heard there was an acrimonious break-up between Ms Mone and her husband which resulted in him walking out and setting up his own company.
Mr Poleg, who heads the firm that bought over Michelle Mone's company MJM in February this year, said Kilday told him that there were "two camps".
He added that Mr Kilday told him: "I belong to Michael Mone's camp. I don't think I can work with Michelle."
The tribunal has heard from lawyers representing both sides that it was agreed that Mr Kilday's office was bugged for "business strategic reasons".
Mr Kilday has told the tribunal he decided to quit after finding the bug in an artificial flower.
This was contradicted by Mr Poleg who said Mr Kilday had already intimated his resignation.
MJM contests the unfair dismissal claim and the hearing continues.