Richard Lochhead drew attention to the controversial "discards" practice as EU ministers prepare for a round of talks in Brussels.
Under the EU's Common Fisheries Policy, skippers are forced to throw away some fish to meet strict quota rules.
Discards of North Sea cod, haddock and whiting alone could amount to about £350 million by 2022, according to the Scottish Government.
Mr Lochhead said: "The disgraceful discarding of marketable fish dead overboard is often enforced on our fishermen by the flawed Common Fisheries Policy. That's why over many frustrating years I have been calling for action to stop this economic and ecological madness. I am pleased that at long last Europe is now getting down to business and seriously discussing how to tackle the madness of discards.
"If Europe doesn't get serious about discards and get radical, with regulations that are effective and enforceable, then we could face the prospect of EU fleets dumping up to a billion pounds worth of a valuable food resource overboard in the next decade."
He warned against a blanket ban, calling for fishermen to be able to land more of what they catch in return for taking less fish from the sea overall.
Bertie Armstrong, chief executive of the Scottish Fishermen's Federation, backed the call.
He said: "Scots Fisheries Secretary Richard Lochhead has laid out the main message, that a graduated, area-based approach is the only one with a chance of success.
"I am sure that this message will be supported by other European Fisheries ministers."
He called for the wider use of selective fishing gear, adding: "That will produce solutions that will avoid perverse outcomes reminiscent of the butter mountains and wine lakes of yesteryear."