Bruce Harris, 90, gave evidence at Southwark Crown Court yesterday via videolink from Sydney, Australia, and said he would have shouted at his younger brother if he had seen him putting his hand up the woman's shorts.
Mr Harris told the court he became the artist and musician's manager in 1981.
He said: "He was unhappy with his management team in the UK. He had fallen on relatively hard times and was not happy with the way they were treating him."
The former advertising chief also took control of his brother's finances.
Rolf Harris is accused of 12 counts of indecent assault on four women between 1968 and 1986, all of which he denies.
Prosecutors have called a number of supporting witnesses who are not named in the charges, including the make-up artist who claimed Harris was nicknamed "the octopus".
The freelancer, who was in her 20s at the time, said he repeatedly put his hands inside her baggy shorts as far as her hips while making a programme in Australia in the mid-1980s.
Bruce Harris said: "That's ridiculous. He would never do that and I wouldn't let him do that, and he knew I wouldn't let him do that. That's just not possible.
"I would have shouted at him, I would have demanded that he stopped. But it never happened, it would never have been something that he would have done."
The court then heard from Tina Fletcher-Hill, a senior executive producer for the BBC, who gave evidence via videolink from the US.
She first met Harris in 1999 while working on Animal Hospital, and later collaborated with him on the series Rolf on Art.
Mrs Fletcher-Hill said: "Rolf is a hugger. Rolf is loving, he's kind, he's affectionate. I've never witnessed anybody pulling back or feeling uncomfortable with his tactile nature."
The trial continues.