The house in Bishopbriggs, East Dunbartonshire, is to be torn down after it was built six feet higher, nine feet longer and four feet wider than originally applied for. It was also built without a permit or approved planning permission in the hope of consent being granted retrospectively.
A demolition team arrived shortly before 10am yesterday and put up fences to prevent access to the property.
Neighbours of the three-floor home, which is owned by Andrew Murray, said the scheduled work may take up to three weeks to complete.
Mary Houston, 74, whose bungalow next-door is dwarfed by the condemned building, said it was "too tall and wide" but it was a "shame" for the owner.
She said: "It's too tall and wide according to the council which makes it too big for the area. The council say it's to come down so I suppose the sooner the better.
"It's a real shame that it's got to go as the poor chap put so much money into it.
"I just can't see how the council let it get this far in the first place if there were so many problems with it."
The original plans for the house stated it was to be only two floors high, but an unapproved third floor was later added.
Another neighbour, who did not wish to be named, said she thought the "nightmare" had gone on long enough. She said: "It is sad for the family and I do feel for them, but for those of us who actually live on this street it's been a nightmare.
"No one's lived in it since it was built and it's been broken into and kids can just wander in off the street as there's never been any fencing put up to close it off.
"It's dangerous for young children due to the rubble and glass but also an eyesore since it's just fallen into disrepair from neglect."
The four-bedroom house cost £300,000 to build and £164,000 for the plot and had a £27,000 fitted kitchen. Its owner, car sales company manager Mr Murray, 51, also spent a further £25,000 on four bathrooms.
Mr Murray, who lives with his wife Frances, 53, and their family in Strathaven, South Lanarkshire, were forced to pay £11,500 to have the property razed to the ground.
The family will also have to sell their current £350,000 home to meet their cost of their debts and the demolition.
In a letter to East Dunbartonshire Council, Mr Murray wrote that he had put "trust and faith in the professionalism of our friend who we understood to be a reputable builder who would oversee the project on our behalf.
"However, it is now evident we were too trustworthy and that we did not give the project the attention that we should have."