Kelman, who is the only Scot to have won the Booker Prize, with How Late It Was, How Late in 1994, said the thought of the British Council's involvement in the conference made him feel "nauseous".
In an interview in today's Scottish Review of Books to discuss his new novel, Mo Said She Was Quirky, he said: "What I object to is the British Council and its involvement as co-organisers of the Writers' Conference, I can't stomach it."
The Glaswegian writer, who has said he will vote Yes to independence, has previously criticised the British state for harbouring hardline colonialist attitudes to the present day.
Widely regarded as Scotland's greatest living writer, Kelman explained his decision to withdraw, but stressed this was a personal decision and he did not want other writers to feel guilty for taking part.
He said: "The British Council is the British State. I stopped being involved with them years ago. It means I don't get many invitations abroad because they do most of the foreign funding. It turns my stomach to see them listed as co-organisers.
"The British Council is not some autonomous, free-thinking arts body. It is sponsored by and accountable to the Foreign and Commonwealth Office.
"It's the FCO's cultural wing, that's what they are, and fair enough, they're quite clear about their remit in terms of pushing English culture and English language – although of course their charitable status has allowed them to outsource English jobs to India to save money."
Kelman will be in conversation with Liz Lochhead during the Edinburgh International Book Festival. Of the book festival itself, he said: "I don't have a problem there. I don't really see it as a Scottish literary festival, I see it as an international book festival that takes place in North Britain."