The Prime Minister rejected a Labour call to boycott the Commonwealth Heads of Government Meeting (CHOGM), insisting it was right for him to attend so he could have "very tough conversations" with the Sri Lankan government.
Nobel peace laureate Aung San Suu Kyi, visiting Mr Cameron at No 10, said it was right to engage with regimes but opposition voices must be heard.
Shadow foreign secretary Douglas Alexandersaid Labour would support the Prime Minister stayed away.
Canada's prime minister Stephen Harper has said he will not attend the November 15-17 meeting, as Sri Lanka was failing to uphold the "core values" of the 53-nation grouping.
UN High Commissioner for Human Rights, Navi Pillay, warned after a visit in the summer Sri Lanka was "heading in an increasingly authoritarian direction" since the end of its 26-year civil war in 2009, while the Foreign Office's annual human rights report names it as a "country of concern".
A House of Commons Foreign Affairs Committee report, earlier this month, found "scant evidence of progress in political and human rights".
Mr Cameron said he was "not happy" with Sri Lanka's human rights record and pledged to visit the north of the country, where Tamil separatists fought a long and bitter civil war.
He said: "My decision is... The right thing for us to do is to go to the Commonwealth conference, as leading members of the Commonwealth, and have some very tough conversations with the Sri Lankan government.
"I'm not happy with their human rights record, I'm not happy with what they have done following the conflict and we will have some very frank conversations to make those points. Where I think Labour have got it wrong is that if you don't go, you can't make those points.
"No one is going to be listening to the British Foreign Secretary and the British Prime Minister if we're not there."
Burmese opposition leader Ms Suu Kyi said she had often been asked if people should come to Burma during the long years of the military regime.
"I have always said that I believe in engagement, but they should engage with us, the opposition, as well."
She said if Mr Cameron went to Sri Lanka he should "engage with all those who are involved, all other stakeholders and not just the government".
Mr Cameron responded: "Very wise words. I will be going to the north of the country as well and I think what Aung San Suu Kyi has said is absolutely right."