Shadow Chancellor Ed Balls said the public would not understand why such an investigation had not been ordered, especially if future abuses emerge.
Following extraordinarily angry clashes yesterday in the House of Commons, between the Labour frontbencher and Chancellor George Osborne, MPs voted last night to order a probe by MPs instead.
But doubts have emerged over whether that inquiry can report by the January deadline necessary to allow ministers to kickstart a clean-up of the City.
The Coalition Government has asked for recommendations quickly so that necessary changes to the law can be included in next year's bill on banking reform.
But the Attorney General Dominic Grieve appeared to suggest that the probe might face difficulty acting in tandem with ongoing criminal investigations into the rate-rigging scandal.
Last night Mr Balls said that Mr Grieve's "devastating interventions" had "exposed serious questions about the scope of this inquiry".
He added: "The Prime Minister and the Chancellor have made a very grave error of judgment.
"And any time any future scandals emerge, people will ask why are we not having the full independent inquiry this country needs?"
Mr Balls and Mr Osborne traded increasingly acrimonious and personal blows over responsibility for the scandal in front of other MPs.
Mr Balls accused the Chancellor of a smear attack following Mr Osborne's implication his opposite number had questions to answer over rate-rigging allegations.
He called on the Chancellor to produce evidence for the claim and "put up or shut up".
He also accused Mr Osborne of a lack of "integrity" when the Tory minister refused to apologise for his comments.
But the Chancellor hit back citing evidence from former Barclays chief executive Bob Diamond to the Commons Treasury Select Committee earlier this week.
Mr Diamond, who was forced to resign in the wake of the scandal, for which his bank was fined £290 million, told MPs he had felt he had been warned by the Bank of England that "ministers in Whitehall" were concerned about Barclays rates.
Despite agreeing to work with the parliamentary inquiry, Labour also signalled it could object to loyal Osborne backbenchers sitting on the committee that will conduct the inquiry.
Earlier, Andrew Tyrie, the Tory chairman of the committee, warned that he would not head the inquiry if it did not have Labour support.
The Coalition also admitted the inquiry needed a "degree" of cross-party support. Labour insisted that it would continue to press for a full, independent judge-led inquiry into the banking sector, which it said was the only way to prevent future abuses. However, Mr Osborne welcomed the parliamentary inquiry which he said would "get to the bottom of what went wrong in our banking industry and what went wrong with the Libor scandal and make the changes in legislation to make sure it never happens again".
Last night, Shadow Scottish Secretary Margaret Curran said: "Given the scale of outrage in Scotland at the behaviour of some in our banks, it is appalling that the Tory party, in collusion with their Liberal Democrat colleagues, have voted down Ed Miliband's call for a full and proper judge-led inquiry into the banks."
The Commons refused Labour demands for a Leveson Inquiry-style investigation into the culture of the City, rejecting the proposal by 81 votes.
MPs voted by 330 to 226, majority 104, to set up the inquiry, which will be led by Mr Tyrie.