Tory MPs last night turned on David Cameron over allegations one of his closest political allies called party activists "mad, swivel-eyed loons".

Lord Feldman, the chairman of the Conservative party and the Prime Minister's tennis partner, who was not named in original reports, has strenuously denied the allegations.

But Conservative MPs expressed anger at their party leader last night, saying it would come as no surprise if people in his inner circle did hold that attitude towards members.

Tory backbencher Brian Binley said Mr Cameron's inner circle had a disdainful view of the party's volunteers.

Mr Binley, who is also a member of the party board chaired by Lord Feldman, said he "certainly will be wanting to know more about [the allegations]" at its next meeting.

The row followed reports that a close friend of Mr Cameron had downplayed his party's problems with its MPs and grassroots over the EU and gay marriage.

He was reported to have said: "It's fine. There's really no problem. The MPs just have to do it because the associations tell them to, and the associations are all mad, swivel-eyed loons."

Lord Feldman, who has been a friend of Mr Cameron's since they went to Oxford together, said it was completely untrue he had made the comments. "I would like to make it quite clear that I did not nor have ever described our associations in this way. Nor do these alleged comments represent my view of our activists," he said.

But party members still used the allegations to make clear their frustration with Mr Cameron.

Bob Woollard, chairman of the Conservative Grassroots group, which organised a protest against gay marriage, said: "Do I look like a mad, swivel-eyed loon?

"I don't know who said that, none of us know who said that, but whoever said that, if it was somebody close to David Cameron, then the Prime Minister needs to disassociate himself from those words, reprimand the person concerned and, if the person isn't prepared to give a fulsome apology, then sack them. There's a disconnection between the grassroots of the Conservative Party in the country as a whole and membership is dropping rapidly because people are switched off.

"Same-sex marriage is really a tipping point, a bellwether issue if you like – people have just said 'I've had enough, I'm off, I will never vote Conservative again'."

But other politicians came to Lord Feldman's defence. Tory vice-chairman Bob Neill stepped up the war of words between Conservative central office and the press, saying: "We have to be very, very wary of this, I think, rather slipshod bit of journalism."

Party treasurer Lord Fink said Lord Feldman had "always spoken about the activists who run our party with enormous respect and admiration".

Backbench MP Steve Barclay said the majority of Conservative grassroots activists did not believe reports that a senior Tory figure called them loons.

"The bulk of the voluntary party know... where the parliamentary party stands," he said, "and they know these allegations, they're not true – Lord Feldman has made clear that they weren't put to him before they were published. He's consulting his lawyers on it."

Kate Devlin