Stephanie Noble moved Grey Lady Too into her home in Back, on the Isle of Lewis, in December 2011 after a dispute over its grazing area.
She said there was nowhere else suitable for the pony and was concerned for her safety.
Western Isles Council, Comhairle nan Eilean Siar, issued Ms Noble with an animal care notice last year, saying alterations were required to door widths and roof space in order to keep the pony indoors.
The council said Grey Lady Too was removed from the property today by animal welfare officers.
A spokesman said: "In the interests of animal welfare, Comhairle nan Eilean Siar today took possession of a pony from the Back area of Lewis.
"This action was taken after numerous animal health interventions and following a vet certifying that the pony was likely to suffer if its circumstances did not change."
Speaking to BBC Scotland in 2012, Ms Noble explained why she took the pony indoors: "Sheep and cattle people will not allow ponies on their land, crofters are primarily interest in the sheep industry and common grazings are not fenced.
"I don't sleep well, I haven't eaten an awful lot over all this worry but I don't mind because this to me is more important, the welfare and safety of this pony."
The council said it is seeking a court order "to allow us to make more suitable provisions for the future of the pony".