The source said the talks were preliminary as Israel had not formally found its prison authorities at fault in the death of Ben Zygier, which was made public earlier this week by an Australian television report that described him as a Mossad officer.
A Mossad link has been neither denied nor confirmed by Australia or Israel. The silence has fanned media speculation that Israel believes the 34-year-old from Melbourne had betrayed the intelligence agency.
Israel's Haaretz newspaper said the state agreed to pay "several million shekels" in damages to Zygier's family around six weeks ago, when an internal inquest declared his death a suicide.
The inquest result was disclosed by the Justice Ministry on Wednesday. The statement said a judge had also ordered an "evaluation regarding issues of negligence".
Avigdor Feldman, an Israeli lawyer with whom Zygier briefly consulted while in prison, said he knew of no compensation deal.