ED Miliband has claimed a moral victory on tax dodging after Lord Fink, the former Tory treasurer, drew back from his threat of legal action, admitting he had personally engaged in "vanilla" tax avoidance measures and claimed everyone did at some level.

After the Labour leader accused the peer of "an extraordinary U-turn", an aide was said to have described the confrontation with the Tory Establishment as a "Milly Dowler moment"; a reference to how Mr Miliband had stood up to media magnate Rupert Murdoch during the phone hacking scandal.

Lord Fink's unexpected suggestion that "everyone" engaged in tax avoidance at some level led the Labour leader to insist this was a "defining moment" in David Cameron's leadership and challenged him to say whether or not he agreed with his Conservative colleague.

At an EU summit in Brussels, the Tory leader avoided questions on the subject but later, asked if Lord Fink was right that most people avoided tax, his spokesman replied: "The PM's view is that all taxes that are due must be paid."

Asked if Mr Cameron was disappointed the hedge-fund multi-millionaire had drawn back from his threat to sue Mr Miliband, he added: "That's entirely a matter for Lord Fink."

But the Conservative donor hit back, accusing the Labour leader of performing his own major climbdown after Mr Miliband, who had branded Mr Cameron a "dodgy Prime Minister surrounded by dodgy donors", was asked if he regarded the Tory peer as one of them.

He explained he had used a "general comment", stressing: "I am not saying it about Lord Fink."

The emollient tone was in sharp contrast to 24 hours earlier when the Labour and Tory leaders clashed furiously at Commons question-time.

Mr Miliband's outburst about dodgy donors and Lord Fink's tax avoidance led the peer hastily to warn the Labour leader that if he repeated his "untrue and defamatory" remarks outside the legal protection of the House of Commons, he would sue him for libel.

But as the Labour leader was insisting he stood by his claims, the Tory peer made a surprise admission, saying: "I don't even want to sue Ed Miliband...If he simply uses the words 'Lord Fink did ordinary tax avoidance' then, no, I couldn't sue him. But if he made the statement 'dodgy' about my bank account, that was potentially libellous. That was the issue I took exception to."

But he then admitted taking his own tax avoidance measures "at the vanilla, bland, end of the spectrum", adding: "The expression 'tax avoidance' is so wide that everyone does tax avoidance at some level."

Mr Miliband seized on the remarks, telling pupils at his old school, Haverstock Comprehensive in north London, during a campaign event: "Yesterday, a Conservative donor challenged me to stand by what I said in the House of Commons. I do.

"And believe it or not, now today he confirmed it as well. He has just said, and I quote: 'I didn't object to his use of the word tax avoidance because tax avoidance - everyone does it.'

"David Cameron must explain why he appointed a treasurer of the Conservative Party, who boasts about engaging in tax avoidance and thinks it is something that everyone does."

Responding to the Labour leader's comments, Lord Fink appeared equally bullish, saying: "I challenged Ed Miliband to repeat the accusations he made in the Commons; that I used an HSBC bank account to avoid tax and that I was a 'dodgy donor'. He did not. This is a major climbdown by a man who is willing to smear without getting his facts straight."

Meantime, the high-powered Commons Treasury Committee said it was concerned about the HSBC tax evasion allegations and announced it was launching an inquiry into them. Representatives from the bank and HM Revenue and Customs are to be called to give evidence.