The criticism of the Prime Minister comes as the UK Government tries to calibrate carefully a series of high-profile appearances by the Tory leader in Scotland in the run-up to ballot day.
The comments will be seen as a sign of increased tensions within the anti-independence campaign and in particular between Labour and the Conservatives.
The Labour source insisted the presence of Mr Cameron north of the Border was a big negative to the No campaign, which, he claimed, would win in spite of him.
He added: "Cameron is toxic. Everyone knows that."
Last month Labour sources accused the Defence Secretary Philip Hammond of hurting the campaign by travelling to Scotland to deliver a "negative" attack on the SNP, playing into charges of 'project fear'.
There is also disquiet at recent reports that Mr Cameron is prepared to take part in a television debate with Ukip leader Nigel Farage, despite the Prime Minister's repeated refusal to go head to head with First Minister Alex Salmond on independence.
The Tories reject the charge that they are too negative on independence, although sources say they have accepted the need to be more "overtly" positive.
The "toxic" verdict is a marked change on last year when Alistair Darling, leader of the Better Together campaign, suggested the Tory leader was irrelevant to the independence campaign.
At the time he said: "Cameron, for this argument, is neither here nor there. This is an argument we have to have out in Scotland."
Earlier this year, in response to the suggestion that the Prime Minister as "a Tory toff from the Home Counties", would be the last person the No campaign would want as its representative, Mr Cameron admitted: "I humbly accept, while I'm sure there are many people in Scotland who would like to hear me talk about this issue, my appeal doesn't stretch to every single part."
Downing Street has a carefully prepared timetable from now until the referendum on September 18.
A number of key events are planned to try to gain maximum advantage in the independence, including the publication of the Treasury's number-crunching exercise on the SNP Government's White Paper, due within weeks.
It is expected that Mr Cameron, as well as Labour leader Ed Miliband, will clear the decks to concentrate on the referendum battle after the European elections on May 22.
Plans for the Prime Minister are understood to include an appearance at the Commonwealth Games and at the launch of the Royal Navy's new aircraft carrier built on the Clyde.
A spokesman for Yes Scotland said: 'It's no surprise that Labour is unimpressed with David Cameron's contribution to the anti-independence campaign. This is further evidence of deep division within the No camp.
"After all, the Prime Minister seems happy to debate with Ukip's Nigel Farage, whose party doesn't have a single seat in the House of Commons, yet refuses to debate Alex Salmond on the future of Scotland. It is an astonishing and increasingly untenable position for Mr Cameron to maintain."