Castle Grant has failed to attract a buyer since being put on the market three months ago.
It was thought to have been snapped up less than a fortnight after going on the market but the sale fell through.
Offers over £1 million are now being asked for the property near Grantown-on-Spey, down from £1.1m.
Whyte, whose reign at Ibrox saw the club plunged into liquidation, bought the castle for £720,000 in 2006.
The Bank of Scotland raised a legal action against him at Inverness Sheriff Court last year after he fell into arrears with the £7,000-a-month mortgage payments.