Some 140,000 people who were born in Romania and Bulgaria were employed in the UK between January and March, the Office for National Statistics (ONS) said, a drop of 4,000 when compared with the 144,000 employed in the last three months of 2013.
Employment curbs were lifted for citizens of the so-called A2 countries on January 1, prompting warnings of a looming surge of immigration from the likes of the UK Independence Party (Ukip).
After the figures were published, Ukip leader Nigel Farage was attacked for "scaremongering" in the run up to the transitional controls being lifted.
Barbara Roche, former MP and chair of cross-party campaign group Migration Matters, said: "After all of Ukip's prejudice and scaremongering over Romanian and Bulgarian migration, today's figures suggest that the numbers of their citizens in Britain have actually fallen by 4,000 since transitional controls were lifted at the start of the year. This is after Ukip claimed that over 5,000 would be arriving each week, every week, for several years."
A range of polarised reports emerged in the run-up to the lifting of controls, including a predicted surge in pickpocketing, mugging, beggars on the streets and rioting.
Other reports suggested citizens from the two eastern European countries would attempt to sell their babies when they arrived in Britain.
Claims of fully-booked flights and coaches from Bucharest and Sofia at the turn of the year were incorrect and retracted.
Keith Vaz MP, chairman of the Home Affairs Select Committee, said: "These figures are unsurprising. The Committee viewed for itself how the supposed flood of immigration from Romania and Bulgaria was little more than a trickle. It would appear now that many may have actually left the UK."
He added: "By not understanding the likely levels of immigration we risk increasing the poisonous rhetoric and prejudice which leads to the destruction of all rational debate. We must not have an immigration arms race."
However, between the final quarter of last year and first three months of this year, the number of employed citizens from the so-called A8 countries, which includes Poland, rose by 74,000 to 802,000.
Similarly, the number of Indians employed in the UK rose by 16,000 to 443,000 and the number of Australians and New Zealanders increased by 3,000 to 132,000.
Conservative MP Anna Soubry later admitted the Government could miss its target to reduce net migration to the tens of thousands by the time of the general election next year.
The Defence Minister said: "At the moment we don't seem to be on course."
But migration experts urged caution when approaching today's figures as it is "too early" to know the long-term implications of lifting the restrictions.
Dr Carlos Vargas-Silva, senior researcher at the Migration Observatory at the University of Oxford, said: "The changes that we see in today's data cannot really be seen as a result of the end of controls on the employment opportunities of A2 migrants.
"It is important that we see this data for what it is - an important first step in understanding how A2 migration is changing - but until we have complete data for 2014 it is impossible to achieve any definite conclusions about the impact of the end of restrictions."
Against a backdrop of growing concerns of an influx of Romanians and Bulgarians, and a surge in popularity for the UK Independence Party (Ukip), Prime Minister David Cameron rushed through new measures at the end of last year to ensure EU migrants will be unable to claim out-of-work benefits for their first three months in the UK.
In addition, those found begging or sleeping rough could be deported and barred from re-entry for 12 months unless they can show they have a proper reason to be in the UK, such as holding a job.
Other proposals previously announced in the Government's Immigration Bill will see migrant access to the NHS restricted, while landlords, employers, bankers and DVLA staff will be expected to take part in checks for illegal immigrants under tough reforms.
An extension of the NHS charging regime was announced, which will see overseas visitors and migrants charged for accident and emergency treatment in England.
Migrants will also have to pay for primary care services such as minor surgery carried out by GPs, while prescription charges will be extended.
Ukip said the ONS figures reveal an extra 292,000 workers born outside the UK took up employment over a year, including 168,000 more EU workers.
Mr Farage said: "These numbers represent another huge increase in the number of foreign workers coming into Britain. Far from controlling immigration, this Government has shown it has absolutely no control over Britain's borders and no intention of putting the British people first."