Researchers in Glasgow looked at the EU's decision to stop "discards", a controversial policy designed to regulate strict quotas on how much fish the fleet can land.
The changes, being brought in from next year, are designed to improve fish stocks.
However, Professor Mike Heath, of Strathclyde University, said: "Wildlife everywhere capitalises on waste from human activity, and discarded fish are food for a wide range of seabirds, marine mammals, seabed animals and other fish. Therefore, banning discards of fish could have unintended effects on the ecosystem."
The practice of throwing dead fish back into the sea was changed after public outcry, reflected in television shows such as Hugh's Fish Fight, presented by Hugh Fearnley-Whittingstall.
In 2009, Scottish vessels were forced by the EU's Common Fisheries Policy to discard almost 28,000 tonnes of fish, about a quarter of the white fish catch, valued at £33 million.
The Strathclyde team said MPs were debating the problem of discards as far back as the 1890s.
The university's study shows that landing the currently discarded fish can have an adverse effect on seabirds and marine mammals, without improving fish stocks.
Changing fishing practices so that unwanted fish are no longer captured had dramatic effects that affected the entire ecosystem, with major benefits for birds, mammals, and fish stocks, according to the research.
Dr Robin Cook, who worked on the study, said: "Our results highlight the importance of considering the broader ecosystem consequences of fishery management. Policy changes to reduce discards affect the food web and, without careful consideration, may dissipate or negate intended benefits.
"Inflating landing quotas to accommodate the entire catch is an inadequate solution with few conservation benefits. On the other hand, the effective reductions in harvest rates resulting from changes in fishing practices to eliminate the capture of unwanted fish can deliver conservation benefits, especially in heavily exploited systems.
"These ecological effects need to be considered alongside the practical, societal and economic issues in developing a sustainable policy."
The study is to be published in the journal Nature Communications.
Lang Bank, director of environment group WWF Scotland, said: "This study reveals one of the many challenges of trying to support coastal communities, deliver sustainable fisheries and help the wider marine environment recover.
"However, it should be remembered the Common Fisheries Policy is only one of a number of measures that exist with a aim of delivering healthier seas. Therefore, we believe that by working with the industry, government and others, it will be possible to find effective solutions that will deliver benefits both for conservation and for those who depend upon our seas for a living.
"Those solutions are likely to require changes in industry behaviour and gear use so that non-target fish and other marine life are avoided in the first instance. Studies such as this one will be helpful to all those involved in delivering more fish and healthy seas."