He said an influx of low-skilled migrant workers will make life harder for Britons facing a cost of living squeeze and he pledged action to close a legal loophole using agency staff to exploit cheap foreign labour.
His comments came as Humza Yousaf, Scotland's external affairs minister, warned the use of "inflamed" rhetoric from Westminster politicians risks ramping up tensions over immigration.
"Rhetoric and hyperbole describing floods or invasions of immigrants coming to the UK in the last week does nothing but potentially heighten any tensions that may exist. Politicians and governments have a duty to act responsibly in respect to immigration," he said.
"We don't want at all to be kowtowing to an agenda which is being driven by UKIP. Politicians of any hue, of any government on these isles, have to be extraordinarily careful of the language they use and make sure they is it responsibly."
He added: "We have to accept if UK citizens can live and benefit from services in Spain and France and across Europe, then of course Europeans will be able to benefit from services and so on in the UK. People should be judged on the contribution they make and new Scots of Polish, Pakistani, Irish or any descent should be welcomed on that basis."
Mr Miliband said: "When millions of workers already have low pay and poor job security in Britain and we add high levels of low-skilled migration mostly from within the EU, some benefit but some lose out. It isn't prejudiced to believe that."
The CBI defended the "perfectly legal" use of agency staff. Policy director Katja Hall said: "The flexible labour market in this country has saved jobs and kept our economy going during tough times. Undermining this flexibility would put the very system which has kept unemployment down at risk."