The quartet, all shareholders in the Fraserburgh-based Klondyke Fishing Company, breached European regulations to land tonnes of herring and mackerel between January 2002 and February 2005.
They were eventually trapped by Operation Trawler, which has already resulted in a string of successful prosecutions in Scotland with millions of pounds levied in fines and criminal earning confiscations.
Detectives said the greed of Andrew Tait, 50, William Tait, 50, Robert Tait, 45, and Peter Tait,44 – all members of the same extended family – was staggering and should not be forgotten.
Judge Lord Turnbull ordered Andrew Tait and Robert Tait, both of Fraserburgh, and Peter Tait, of Aberdeen, to hand over £40,000 each with William Tait, also of Fraserburgh, ordered to pay £35,000.
Fresh Catch Ltd, also of Fraserburgh, was fined £160,000 for handling quantities of illegal fish between January 2002 and February 2005.
Earlier, the court heard the lengths the accused had gone to evade detection by fish inspectors. Scales and computer equipment had been set to give false readings and another fish processor, Alexander Buchan, of Peterhead, had installed a secret pipeline to allow fish to be brought into the plant without being weighed.
The fines come in addition to the £720,000 demanded by prosecutors after a successful action against the four under Proceeds of Crime legislation last month.
More than 20 other skippers and at least two other processing plants have been prosecuted following the work of Operation Trawler, with the Taits and Fresh Catch the last to be taken through the courts.
Detective Superintendent Gordon Gibson, of Grampian Police, said: "Having led this investigation for more than four years, it has been extremely complex and is a prime example of how various agencies working closely together have been highly successful in bringing no fewer than 31 individuals and three large companies to justice.
"Fines totalling nearly £1.75m have been handed out by the courts and more than £8.1m has been seized under the proceeds of crime over the course of this investigation and bear testament to my view.
"The greed shown by these men and the respective com-panies has been absolutely staggering, at a level never seen before in the UK and that should not be forgotten."
The four men pled guilty to breaking European fishing regulations during a hearing at the High Court in Edinburgh in June 2012.
On Wednesday, defence QC Murdo MacLeod said the Taits and the firm were sorry for carrying out the scam.
Mr MacLeod added: "These men are involved in an industry that is dangerous and full of risk. They place themselves in danger on a regular basis. The accused were participating in an activity that was widespread throughout the entire industry.
"That, of course, is not a mitigatory factor. But it provides the court with an insight into why the accused participated in the activity.
"Since stopping the activity, the four men have found they no longer have the worry or the stress they had when they were participating in criminal conduct."
Mr MacLeod also asked the court to restrict the fine that Fresh Catch would have to pay, saying the company was a major employer in Fraserburgh and it was currently operating in difficult economic circumstances.
Lord Turnbull then ordered that the men had six months to pay their fines. He gave Fresh Catch 18 months to pay its fines.
The Crown Office's head of the Serious and Organised Crime Division and the Proceeds of Crime Champion, Lindsey Miller said: "These accused may not have been involved in drug dealing or prostitution but make no mistake that they were involved in significant and serious organised criminality."