Dr Paisley, who founded the Protestant fundamentalist Free Presbyterian denomination in the early 1950s, was synonymous with Martyrs Memorial church in east Belfast for six decades.
But in a new documentary on his life and times, Stormont's former First Minister claims he was forced to retire in early 2012 after receiving a letter the previous autumn from elders in the church requesting he step down.
This came four years after he was pressured to resign as moderator of the denomination, due to anger among some members at his decision to lead the Democratic Unionists into power with Sinn Fein.
The internal strife that generated within Free Presbyterianism was well publicised at the time, but his later decision to step down as preacher at Martyrs Memorial has not been associated with the same degree of controversy.
But Dr Paisley has now indicated his retirement as preacher was forced on him. He said a letter from the Kirk Session of the church, signed by the elders, called for the move.
"It was hurtful that was the way they thought they would treat us," he said.
"They did that and they will have to answer to the people and they will also have to answer to God at the end of the day."
Asked why his family no longer attends the church, he said: "I think they are better not going to worship there because they would not be happy and you don't go to church to sit on nails, you go to church to sit in a place where there is rest and peace."
In the BBC documentary Paisley: Genesis to Revelation, the churchman's wife Baroness Eileen Paisley said the letter from the kirk was "absolutely shattering".
"We just could not believe that Ian after 65 years ministry in the same church, continuous ministry for all those years, in leading the church and building it, that these men take this attitude and all of a sudden boot him out.
"We just could not fathom it, we could not understand why."
Mrs Paisley said the family have not gone back since Dr Paisley delivered his last sermon in January 2012.
"It was almost like a death, it was almost you had that feeling that this person has gone, everything has gone and it will never be the same again - can't ever be the same again," she said.
In the wake of the DUP move to enter power-sharing with Sinn Fein in 2007, an unofficial delegation from the Free Presbyterian Church went to Stormont to demand that Dr Paisley step aside as moderator.
It triggered an internal dispute that eventually saw the Democratic Unionist leader agree to resign from his leadership role in the church he established.
Mr Paisley said his opponents in the church at the time were only interested in serving their "own egos".
His wife said the episode had an impact on the whole Paisley family.
"Our hearts were all broken for Ian," she said.
"The children and myself as well, I felt he had been deeply wounded in the house of his friends and I just felt it was really iniquitous of them and a really dreadful, hurtful, nasty, ungodly, unchristian thing to do."
Of the move to remove her husband as moderator, Baroness Paisley said the family felt the family had not been defeated by enemies, rather betrayed by friends.