Savile's family said they made the decision to ensure the "dignity and sanctity" of Woodlands Cemetery in Scarborough.
Scarborough Borough Council said work to dismantle the memorial headstone, which bears Savile's image and lists his accomplishments, was carried out shortly after midnight.
In a statement the council said: "We have been advised by the funeral directors, Joseph A Hey and Son, acting on behalf of Jimmy Savile's family, that the headstone at Woodlands Cemetery has been removed."
Savile's family asked for its removal after it emerged police believe the former presenter could have abused up to 25 victims over 40 years.
A family spokesman said in a statement: "The family members are deeply aware of the impact that the stone remaining there could have on the dignity and sanctity of the cemetery.
"Out of respect to public opinion, to those who are buried there, and to those who tend their graves and visit there, we have decided to remove it."
A number of memorials to Savile have already been removed, including an inscription on the wall at Leeds Civic Hall in recognition of his charity work, and a street sign in Scarborough. A plaque outside his home has been defaced.
Police said yesterday that Savile's abuse might have been on a "national scale".
Scotland Yard has formally recorded eight criminal allegations against the former Top Of The Pops presenter so far in its investigation, named Operation Yewtree.
The alleged abuse involves teenage girls as young as 13 and includes two complaints of rape and six of indecent assault, with officers looking into up to 120 lines of inquiry.
Commander Peter Spindler, head of specialist crime investigations at Scotland Yard, said the allegations span four decades, the earliest of which dates back to 1959.
Mr Spindler said although it is early in the inquiry, the information so far suggests Savile possessed a "predilection for teenage girls".
He said: "The reality is this really has captured the public's mind. We are getting calls from victims, from witnesses and third parties who believe they know something about it.
"We have formally recorded eight criminal allegations against Savile. Two of those are rape, six of indecent assault.
"These are primarily against girls in their mid-teens, so between 13 and 16, and it spans four decades of abuse.
"The pattern of his offending behaviour does appear to be on a national scale."
Mr Spindler said the first allegation dates back to about 1959 but most claims seemed to be from the 1970s and 80s.
Scotland Yard has contacted ITV and the BBC, which in turn are contacting alleged victims to see if they will co-operate, he said.
Mr Spindler added: "We believe there are probably another 20 potential victims there.
"It is too early for us to give you an accurate picture of what 120 lines of inquiry will distil down to but we believe we will come up with between 20 to 25 victims."
A spokesman for Scotland Yard said at least five forces - the Met, Surrey, Sussex, Northamptonshire and Jersey - are investigating allegations.
As well as claims relating to alleged abuse at the BBC, Jersey's Haut de la Garenne children's home, and Duncroft Approved School for Girls near Staines, Surrey, police have contacted Stoke Mandeville Hospital and Leeds Royal Infirmary, where Savile did charity work.
The BBC is not being investigated, Mr Spindler said, but officers are working to identify any individual who could be subject to criminal investigation.
Mr Spindler said police will only launch investigations into living individuals about whom allegations are made if they get some evidence, which is most likely to come from witness statements.
The BBC's internal investigations unit has passed information to Scotland Yard and is fully co-operating with police.
Calling it an "assessment" rather than an investigation, he said Scotland Yard will produce a joint report with the NSPCC to look for lessons that can be learned and conclusions drawn.
It is being carried out by a team of officers from the force's Serious Case Team to help bring it to a "swift conclusion".
Peter Liver, from the NSPCC, said the charity has received 40 calls in the last five days after the claims emerged.
Of these, 24 have been referred to police or other agencies that support victims of abuse, and 17 directly relate to Savile, he said.
There were also 21 unrelated calls to the helpline stemming from publicity over the allegations.
Prime Minister David Cameron addressed the issue of the prospect that Savile could be posthumously stripped of his knighthood.
The move would require a change in the law as technically, when the former DJ died last year, the honour ceased to exist.
A campaign by The Sun newspaper is calling for that change, so that honours can be revoked posthumously by the Forfeiture Committee, responsible for looking at honours and their removal.
Mr Cameron told ITV1's Daybreak yesterday: "These stories are deeply, deeply troubling and I hope that every organisation that has responsibilities will have a proper investigation into what happened, and if these things did happen, and how they were allowed to happen, and then of course everyone has to take their responsibilities."
The Prime Minister's comments came after the chairman of the BBC Trust gave his backing to inquiries by police and the corporation.
Lord Patten said the allegations against Savile could not be excused as behaviour from a time when "attitudes were different".
He told a business dinner in Cardiff on Monday: "It's no excuse to say 'I'm sure the same thing used to happen with pop groups and others at the time'. Those things may be true but they don't provide an excuse."
BBC director-general George Entwistle also apologised to victims of Savile's alleged sex abuse and said the corporation would hold its own inquiry, to follow a police probe.
