The acclaimed painter, illustrator and author of Lanark said he was right to use the word as a former director of the Edinburgh International Film Festival (EIFF) and a playwright weighed into the row over his remarks.
Northern Ireland-born filmmaker Mark Cousins said Gray's intervention had left him feeling unwelcome, and playwright David Greig said he was mistaken.
Gray, who made the comments in an essay in a book called Unstated: Writers on Scottish Independence, said the row had left him mystified.
He added: "I do include in the essay that I thoroughly approve of settlers but I might regret colonists.
"I was using the words with great care, I am a professional writer you see, and I explain in the essay why I used them.
"I think any kind of writer who was afraid of saying something that they believe, because it would cause annoyance, would be very silly."
Mr Gray said his own experiences of the 1990 Glasgow City of Culture celebrations and the running of the former Third Eye Centre in Glasgow were specific examples where English administrators had been employed.
They had, in his view, downplayed or showed a lack of understanding of Scottish culture or talent.
He added: "Of course I am not anti-English. Apart from anything else, some of my best friends are English.
"My mother's parents were English, from Nottingham, and then became settlers here in Scotland."
Mr Cousins said on Twitter: "As a N Irish person who's lived in Scotland for 30 yrs, I've always felt so welcome. Until, that is, Alasdair Gray's recent remarks."
Mr Greig said: "Alasdair Gray's got it so wrong ... what people give to Scottish culture has nothing to do with birthplace or length of residency."
In the essay, Mr Gray, 77, said: "Immigrants into Scotland, as into other lands, are settlers or colonists.
"English settlers are as much a part of Scotland as Asian restaurateurs and shopkeepers, or the Italians who brought us fish and chips. The colonists look forward to a future back in England through promotion or by retirement."
He added that some arts administrators, who were invited to Scotland by Scots, could be classified as colonists because "their work for institutions originally created to encourage art in Scotland actually depressed it".
Gray's comments were followed by the revelation in yesterday's Herald that Vicky Featherstone, the lauded founding artistic director of the National Theatre of Scotland, said she suffered a crisis of confidence after criticism that focused on her Englishness.
The Scottish Government said it "disagreed" with Gray's sentiments. A spokesman added: "Ministers stand for a welcoming and inclusive Scotland.
"In addition, recruitment policies by non-departmental public bodies are a matter for the individual organisations and their boards to decide on, based on the best person for the job, wherever they come from."
Ms Featherstone, who is leaving her post to run the Royal Court Theatre in London, said of the anti-English criticism: "The hardest thing for me was that if people had criticised the programme, I could have defended it, but when people are criticising the programme because I am English, that is indefensible."