Others stayed silent because they were convinced they would not be believed as Savile was such a powerful and influential character.
Some victims were even told they were "lucky someone like Savile had paid them attention", according to the NSPCC report.
A significant number of the men and women interviewed have still not confided in friends and family about the abuse, the children's charity said.
Savile died aged 84 in October 2011 - a year before allegations that he had sexually abused children were broadcast in an ITV documentary.
The revelations prompted hundreds of victims to come forward with claims that they were attacked at BBC premises or in other institutions, including hospitals.
According to the report, Would They Actually Have Believed Me?, some of the victims, who were aged between eight and 26 when Savile assaulted them, told hospital staff, who dismissed their claims.
One of the 26 victims interviewed by NSPCC counsellors went to the police but no action was taken.
The NSPCC said the research, which was commissioned by Her Majesty's Inspectorate of Constabulary, highlighted the "devastating scars" that victims had suffered from the abuse, with some turning to drink and drugs to cope. Others have suffered mental illness, poor relationships or contemplated suicide, it said.
According to the report, victims had "largely positive experiences" when they finally went to the police after Scotland Yard launched Operation Yewtree in the wake of the TV expose about Savile.
But they wanted to see improvements, including new ways to report sexual abuse and additional specialist training for officers receiving and investigating allegations to ensure they fully appreciate the long-term emotional impact of the crime, the NSPCC said.