Sir James Paice, who was an environment minister until 2012, said plenty of agricultural employers paid their workers well and housed them in good accommodation.
But Sir James said the only reason there was job displacement in the sector was because "British workers are just not prepared to do that work".
Turning to Home Secretary Theresa May during Home Office questions, the former employment minister said: "Would you accept that there are countless employers in the food and farming sector who do not use illegal gangmasters and instead who pay good pay and provide good accommodation for their workers?
"And that where there is job displacement it is because British workers are just not prepared to do that work rather than taking illegal migrants somehow on the cheap?"
Mrs May replied: "I absolutely accept that there have been, and particularly you will be aware of the seasonal agricultural workers scheme, which was a very particular scheme which did indeed ensure that people were brought across to do work in the agricultural sector.
"But, as the Agriculture Minister made clear, we do need to look at ensuring that people here in the UK are able to take on the jobs that are available to them."