The claim is contained in a new memoir by Mr Brown's former spin doctor Damian McBride.
Mr McBride also sets out how he plotted against former Labour home secretaries John Reid and Charles Clarke.
Last night the Conservatives demanded that Labour leader Ed Miliband, another former key Brown ally, come clean on what he knew of Mr McBride's "dark arts".
The call followed a comment by former Labour minister Dame Tessa Jowell that she was "sure" Mr Miliband was aware of Mr McBride's activities.
The spin doctor was one of Mr Brown's closest advisors in No 10 until he was forced to resign after it emerged he had tried to smear senior Conservatives.
In Power Trip, Mr McBride writes how in 2008 Ms Alexander was embroiled in a scandal about an unlawful campaign donation and had just been censured by a Holyrood committee.
However, she had also issued her famous "bring it on" challenge to Alex Salmond to call an independence referendum, a strategy Mr Brown opposed.
In the book Mr McBride says of the now Shadow Foreign Secretary: "Dispassionately he told the Prime Minister his sister had to quit to avoid further damage. Douglas warned Gordon she'd need to make it clear it was nothing to do with the donation or the referendum. If I was sometimes cold-blooded about how I did my job, I had nothing on Douglas that day."
However, Mr Alexander said: "He was discredited when he left Downing Street and that's really all there is to say. I always supported my sister and I never supported Damian McBride. That might explain why he writes about me in those terms."
Asked if Mr McBride was lying, Mr Alexander said: "I've said what I'm going to say. It's what Damian McBride does."
In his book Mr McBride also details how he collated a "black book" of stories about Lord Reid.
He also boasts how he "orchestrated a briefing war" between Mr Clarke and one of Mr Blair's advisers, which contributed to the politician being sacked. He says he gathered information through a network of Westminster moles.
He writes: "Labour, Conservative or Liberal Democrat; Ministers, MPs or advisers; if they'd ever shared their secrets with colleagues in Westminster, the chances were I ended up being told about them, too. Drug use; spousal abuse; secret alcoholism; extra-marital affairs.
"I estimate I did nothing with 95% of the stories I was told. But some ended up on the front pages of Sunday newspapers."
His motivation was, he claims, to protect Mr Brown. He writes: "I offered him the best press he could hope for, unrivalled intelligence about what was going on in the media and access to parts of the press that no other Labour politician could reach."
Dame Tessa said: "I don't think (the book) is damaging for Ed Miliband. I'm sure he knew that this was going on. He was actually away a lot of the time.
"One of the most important things that Ed Miliband has done is to outlaw this kind of briefing in his shadow cabinet."
But Tory MP Henry Smith said: "Ed Miliband must now tell us what he knew about McBride's smear campaigns."