IR35 is meant to stop the abuse of personal service companies to disguise employees as freelancers to gain better tax treatment.
But critics say it harms small firms set up for legitimate reasons by contractors such as IT professionals on short-term contracts.
In a new report a House of Lords Select Committee said HM Revenue and Customs (HMRC) had failed to provide a sound basis to claim £550 million of its revenue was at risk if IR35 was abolished.
HMRC "need to do more to demonstrate that the revenue protection they claim for the IR35 legislation outweighs the costs it imposes", said the report.
"There are many reasons for the use of personal service companies, including the possibility of reducing tax and national insurance bills. The Government's anti-avoidance legislation, often referred to as IR35, is complex and raises its own problems," said committee chairwoman Baroness Noakes.
An HMRC spokesman said personal services firms are used for genuine commercial purposes but the report recognises some are used for tax and national insurance avoidance.