The businessman did not pay tax and national insurance for his former housekeepers, who are on trial accused of stealing items worth thousands of pounds from Castle Grant, and salary payments made to them were often late and came from different companies, Inverness Sheriff Court heard.
Jane Hagan, 51, and her husband Terence Horan deny the charge against them.
In evidence yesterday, Ms Hagan said their former employer instructed them to have items put in storage when it became clear he was several months behind in mortgage payments and there was a danger the property, near Grantown-on-Spey, would be repossessed.
Ms Hagan, giving evidence, spoke of the stress she and her husband were under when nine threats were made against Mr Whyte and his castle. One was a threat to firebomb the property, but Ms Hagan told the court her ex-boss refused to install fire prevention equipment.
She said: "He did not seem concerned and had a cavalier attitude about it.
"An insurance company came to the castle concerned about its fire precautions and its condition. The electrics had been badly installed and the sprinkler system was redundant.
"Fire experts came and gave us a proposal to replace it. But Mr Whyte did nothing about it. The Fire Service also came and were horrified to see no fire precautions where the family and the children stayed.
"We got new extinguishers put in but he didn't pay the bill and they took them back. We bought two of them ourselves for our own safety.
"He was pretty distant and liked to remain private. I would do the cooking and cleaning and order the food and wine. He wanted the castle run like a five-star hotel.
"After a while we had problems getting supplies because he wasn't paying the suppliers."
Ms Hagan said Mr Whyte would give excuses for not providing a contract of employment.
"We got two wage slips in eight months," she said. "They stated he had paid our tax and national insurance. After we were sacked, we contacted HMRC and were told that nothing had ever been paid during our employment.
"Sometimes we were paid five weeks late and often from different sources. Once we were paid from a company called Tixway, another time from Liberty Corporation and a couple of times by his father."
Ms Hagan also spoke about the Bank of Scotland phoning the castle to have a message passed on to Mr Whyte of the seriousness of the situation surrounding the mortgage on the castle.
She said: "It was several months behind and it was clear there was a danger the castle would be repossessed.
"He told us he wasn't going to come back to the castle and we were instructed to have items put in storage."
The items she detailed, including a quad bike and clay pigeon shoot trap, are among those listed as allegedly stolen from Mr Whyte. Ms Hagan added that he had told the couple to store items of expensive jewellery and costly gold and bejewelled pens which had been in Whyte's study and he claimed were stolen by them.
"He wanted things removed that might be seized," she said.
Mr Horan, 54, also gave evidence and insisted an instruction to store items, including farm machinery, came from Mr Whyte himself. He said: "I was then told to collect his personal belongings from his office and keep them in a safe place. Jane and I did a supermarket sweep putting stuff in bins and plastic bags and stored it in our personal quarters until we went on holiday to Hawaii.
"We then put it in our storage container until we got back. He was concerned about people entering the castle and taking it.
"While we were on holiday he sent us an email to tell us we were sacked. We were shocked and had no idea why."
The trial continues on Monday.