Lord Fink backs away from Miliband threats: I've taken vanilla measures to reduce my liabilities

The former Tory treasurer branded Mr Miliband's jibe "untrue and defamatory" and challenged him to step out from the cover of parliamentary privilege so he could take legal action - something Labour sources said Mr Miliband would do as he launches the party's education policy in a speech in north London later today.

But in an interview with the Evening Standard, Lord Fink acknowledged that if Mr Miliband accused him of "ordinary tax avoidance" he would be unable to sue.

The hedge fund multimillionaire said: "I don't even want to sue Ed Miliband. In my life I have been libelled a few dozen times and I have never sued anybody, even for some comments that were quite outrageous.

"If he simply uses the words 'Lord Fink did ordinary tax avoidance' then no, I couldn't sue him. But if he made the statement 'dodgy' about my bank account, that was potentially libellous. That was the issue I took exception to.

"I also took exception to him saying I had questions to answer. In fact, whenever anyone has put questions to me I have answered them."

Lord Fink told the Standard: "The expression tax avoidance is so wide that everyone does tax avoidance at some level."

He said he had rejected expert advice that he could save a fortune by adopting "aggressive" avoidance measures, but added: "I chose the mildest end of the spectrum that I was advised on. What I did ... was at the vanilla, bland, end of the spectrum."

Lord Fink said he "used the opportunity ... to set up some simple family trusts" while on a four-year posting to Switzerland.

"Really what I was trying to do was, not like a living will, but to allocate a very small shareholding to each of my children so they could pay deposits on houses in London one day after we returned," he said. "There was nothing complex, and they weren't aggressive tax planning.

"My family and I paid tax on all the dividends, both in Switzerland and the UK. They were done because my children were under 18 and I wanted them to have something to help them make their way in the wider world."

Lord Fink was among nine Conservative donors listed as clients of HSBC's Swiss subsidiary in leaked files, according to The Guardian. The newspaper said the accounts were held legally for a wide variety of reasons, and made no allegation of wrongdoing against those listed.

Of more than 6,000 British names understood to have been passed to HM Revenue & Customs in 2010, UK tax authorities say they have pursued around 1,100, recovering £135 million in unpaid tax, fines and interest. Just one prosecution has been brought and around 130 cases remain outstanding.

During a rowdy Prime Minister's Questions session, Mr Miliband branded Mr Cameron a "dodgy Prime Minister surrounded by dodgy donors" who was "up to his neck" in the HSBC tax avoidance scandal.

But Mr Cameron hit back by pointing out a former Labour donor featured on the list of account-holders. The Prime Minister also insisted that "every proper process was followed" when former HSBC chief Lord Green was appointed trade minister at the end of 2010.

The spat came as the bank worker who leaked the account details said he first contacted HMRC in 2008 and authorities have only a "tiny part" of the information available.

Herve Falciani, who initially obtained the list while employed as an IT worker in 2007, told Sky News he had emailed and phoned the tax authorities two years before data was handed over by French authorities in 2010.

"I sent an email, a very naive email, in 2008 ... to England - to the department dedicated to tax evasion - and afterwards I even called them," he said.

"And finally the most efficient move was through the French authorities because when we accepted to work together it was established and agreed that what we were doing should be available to any countries having co-operation treaties signed with France."

In the Commons yesterday, HMRC chief executive Lin Homer insisted that staff had been "diligent" in their approach to the files.

Labour demanded to know why the Government signed up to a tax deal with Switzerland in 2012 which included a commitment that the UK "will not actively seek to acquire customer data stolen from Swiss banks".

Labour's Treasury spokeswoman Shabana Mahmood said: "George Osborne personally trumpeted this Swiss tax deal in his 2012 Autumn Statement. He must now come clean and explain why he signed off an agreement with the Swiss authorities which tied HMRC's hands for the future.

"This deal means that the Government may never again be able to get hold of the sort of information it received in 2010 about tax evasion and which is at the centre of this scandal.

"David Cameron still needs to answer whether he discussed tax evasion at HSBC with Lord Green. And it is time we had a full and frank statement from Lord Green about what he knew about these activities which were happening while he was in charge of HSBC."

Ms Mahmood wrote to Mr Cameron to ask him to make clear when ministers became aware of the allegations against HSBC Suisse.

Earlier this week, Downing Street said: "No Government minister had any knowledge that HSBC may have been involved in wrongdoing in regard to its Swiss banking arm prior to the reports of the last couple of days".

But Ms Homer told a parliamentary committee yesterday that she was "confident" the tax authorities would have informed ministers about the large cache of information handed over by French authorities within a "few months" of its arrival in 2010.

Downing Street sources have stressed that the denial of ministerial knowledge related to allegations of wrongdoing by bank employees, rather than to widely publicised claims that clients at the Geneva branch might have been seeking to avoid tax.

But Ms Mahmood said in her letter to the Prime Minister: "We now know that HMRC informed ministers of this case before prolonged public coverage of its details. It is now inconceivable that no minister was aware of a case of such magnitude, as your office has claimed, and the Government's position invites nothing but ridicule."

London mayor Boris Johnson dismissed Mr Miliband's "dodgy" accusation as "a lot of absolute tripe".

Speaking to LBC radio during a visit to New York, Mr Johnson said: "David Cameron is a man of the absolutely highest ethical standards in every respect.

"I think it is really scraping the bottom of the barrel for Labour to start chucking this sort of thing around."

Responding to Mr Miliband's comments in his speech, Lord Fink said: 'Yesterday I challenged Ed Miliband to repeat the accusations he made in the Commons - that I used an HSBC bank account to avoid tax and that I was a 'dodgy donor'. He did not.

"This is a major climbdown by a man who is willing to smear without getting his facts straight."