But Roger Ordish said he did know that Savile had a "predilection for younger females".
In his first interview since the disclosures about Savile's alleged abuse of possibly dozens of underage victims over a period of many years, he described the presenter as a "manipulative" man.
But he told ITV1's This Morning he saw no abuse during the two decades in which he worked with the late TV and radio host.
"I didn't see anything and nothing was reported to me," he said. During their years working together, Mr Ordish said Savile, whom he described as a private man who "compartmentalised" different areas of his life, even stayed at his home.
"He slept in a bedroom next to my 14-year-old daughter and I hope that is some indication that we had no suspicions of anything of this nature at all," he said.
The disclosures about Savile’s private life were made in an ITV documentary a fortnight ago which had dismayed Mr Ordish.
"I was absolutely shocked and shattered and I was surprised that I had not been approached by the programme," he said of the Exposure documentary.
He maintained he knew nothing about Savile’s activities, although he told This Morning hosts Phillip Schofield and Holly Willoughby there had been rumours.
But he went on: "You hear rumours about everybody famous, there must be rumours that go around about you two.
"We were aware that Jim had a predilection for younger females," he added, saying it was obvious from the way he chatted to girls.
Mr Ordish said Savile would compliment them on their beauty, "but nothing obscene".
"These are girls who are young females, they’re not children. They could have been 14 looking like 18, but they were not children in my definition of the word," he said.
David Cameron has been facing Labour demands to set up an independent inquiry into the Savile sex abuse scandal.
Labour leader Ed Miliband insisted that a broad inquiry into Savile’s activities at the BBC, Stoke Mandeville hospital and Broadmoor was essential to "do right by the victims".
But Culture Secretary Maria Miller has dismissed calls for an independent inquiry, saying she was "confident" BBC chiefs were taking the claims "very seriously".
The BBC has announced two inquiries as a result of the furore. One will look into whether there were any failings over the handling of an abandoned Newsnight report about Savile, while the second will examine the "culture and practices of the BBC" during the years the presenter worked there.
Police believe the DJ and television presenter’s alleged catalogue of sex abuse could have spanned six decades and included about 60 victims.
Some have claimed they were abused in Savile’s dressing room, but Mr Ordish said there would have been little opportunity for him to be unaccompanied with young guests during recordings of Jim'll Fix It.
"I don’t think he’d be alone in his dressing room at all, because there was always somebody there. There was wardrobe, there was make-up, the researchers had to take him through the people that were 'fixing'," he said.
If anyone had come to him at the time with allegations, Mr Ordish said he "most certainly" would have taken it higher within the BBC.
Asked if he may have blocked his ears to any claims, he said: "I don’t think so. There wasn’t ever a climate where it was acceptable to molest under-age females.
"The youngsters that were on Jim’ll Fix It always had chaperones, usually in the form of parents, with them."
One guest on Savile's show had claimed to have been abused when he was a nine-year-old in his cub scout uniform. An emotional Mr Ordish said it was "possible" Savile could have molested him in the dressing room following the recording.
"There’s such overwhelming evidence that these things happened, that they can not all have been fabricated. But Jimmy Savile was a very clever man and he seems to have succeeded in hoodwinking a prime minister, the Vatican, the civil police forces, the NHS hospitals up and down the country and some members of the BBC staff, including me. I’d say he was a manipulative man."
He said Jim'll Fix It "is being regarded as guilty by association and I want to assure people who were on it that the joy and the fun they had has not gone and not diminished".
Mr Ordish added: "If he could hoodwink all these powerful people, he could hoodwink me as well.
"And I don’t think I could have gone out on a busy studio day being an investigator to see if I thought he was doing something untoward because I never for a moment imagined that he was."
Responding to Mr Miliband’s call, a Downing Street spokeswoman told a regular media briefing in Westminster: "Nothing has been ruled out, but our view is it would be premature to prejudge the outcome of the two BBC inquiries and the police investigation."
Nick Clegg has not ruled out backing an independent inquiry into the Savile allegations. The Deputy Prime Minister today told MPs he believed it was important police investigations were allowed to finish before any further probe was set up.
But under questioning by Labour deputy leader Harriet Harman, Mr Clegg said there "may be a case for an inquiry".
Ms Harman repeated Labour calls for a broad inquiry into all aspects of the allegations made against Savile in recent days.
Mr Clegg told MPs: "I certainly accept there may be a case for an inquiry and if an inquiry were to be held which is as broad ranging as you suggest it should be, it should be independent to look at the full range of the shocking revelations as they have come to light.
"So we are not ruling it out, I just think the first priority must be to allow the police to conduct its work into what are deeply shocking and troubling revelations and allegations."
Mr Clegg continued: "I constantly ask myself how on earth is this possible on such a scale over such a long period of time in so many different settings.
"It is in many ways the sort of thing, the dark side of the cult of celebrity which maybe intimidated people from speaking our earlier. Now we know these things and they are coming to light we should proceed in a way which is led by what the police find and keep an open mind on the issue of an inquiry."
Ms Harman had asked: "Because it has come to light (Savile) carried out these crimes at the BBC but also at other institutions do you agree we need one inquiry that looks into what happened in each of these institutions to see if there were patterns of systemic failure so we get a coherent picture?
"Do you agree any inquiry must be completely independent, that is the very least the victims of Savile would expect to get to the truth and learn the lessons?
"Will the Government now set up an independent inquiry?"
Replying to Mr Clegg, Ms Harman followed up: "Can I put it to you that because the police are carrying out very important investigations which obviously should not be impeded, it does not mean that an independent inquiry shouldn’t be set up now.
"I would ask you to reflect on that because daily revelations are coming forward and the victims of these abuses need to hear firmly the truth is going to be discovered.
"Can I assure you, we would be ready to discuss terms of reference to make sure we have the full and thorough inquiry which is no less than what the victims deserve."
Mr Clegg said: "You said, reasonably enough, there is no reason you can’t establish an inquiry while the police are doing their work.
"But I think the issue actually is slightly the other way around - what kind of work can the inquiry do while the police are conducting their investigations.
"I don’t think we should imagine that an inquiry which cannot pursue certain avenues of inquiry because the police have their own investigations would necessarily be the best answer for the victims at this time.
"But let us at least agree, firstly we must do everything we can to make sure proper answers are given to the victims and secondly I’m grateful for your signal you are prepared to work on a cross party basis on this as we work together to get to the bottom of what on earth happened."
Speaking in the Commons yesterday, Culture Secretary Maria Miller outlined three separate internal investigations into the allegations surrounding Savile.
She said: "These are undoubtedly very serious matters that have wide-ranging implications for a number of public institutions, not just the BBC.
"It is now crucial we understand what went wrong and how it can be put right."
Mrs Miller said there was no need for a wider inquiry while the police investigation was going on. It was crucial that detectives were allowed to continue their investigation "unfettered" by other inquiries.
Police believe the DJ and television presenter’s alleged catalogue of child sex abuse could have spanned six decades and included around 60 victims.
BBC director general George Entwistle announced last Friday that two inquiries would be launched into the abuse claims.
One will look into whether there were any failings over the handling of the abandoned Newsnight report. A second independent inquiry will look into the "culture and practices of the BBC during the years Jimmy Savile worked here", Mr Entwistle said.
Conservative MP Rob Wilson, who secured yesterday’s urgent question in the Commons, said the BBC had been "slow to react" to the scandal and accused its Trust of acting as a "cheerleader and defender".
"I have a number of major concerns that the inquiries by the BBC will not be sufficiently independent, transparent and robust to give the public confidence," he said.
"Any hint of a cover-up by the BBC of its own role in this dreadful affair will cause huge damage to public and audience trust."
