It follows allegations they were targeted for "Trojan Horse" takeovers by hardline Muslims.
He told MPs that the UK Government will take "decisive action" following yesterday's findings of Ofsted, as well as the Education Funding Agency (EFA).
He warned that all schools could now be subjected to unannounced inspections while schools that have failed will be taken over and put under new leadership.
Ofsted issued a damning verdict on the running of a number of schools in the city.
Inspections conducted following claims of a takeover plot by hardline Muslims found that a "culture of fear and intimidation" has developed in some schools and, in several, governors exerted "inappropriate influence" over how they are being run.
A separate EFA report into Park View Educational Trust (PVET), which runs three of the schools rated inadequate by Ofsted and has been at the heart of the alleged takeover plot, concluded it has "many weaknesses" and restricted its curriculum to a "conservative Islamic perspective".
And EFA's report into Oldknow Academy, another school placed into special measures, found it was "taking on the practices of an Islamic faith school".
Mr Gove warned that in future any school could be subjected to tough, on-the-spot, inspections "with no advance warning and no opportunities to conceal failure".
He acknowledged that there were questions for the Department for Education, Ofsted and Birmingham City Council about whether there were "warning signs" of problems in Birmingham schools that had been missed.
It came as Home Secretary Theresa May denied authorising the release of an explosive letter that led to her adviser being sensationally sacked.
Mrs May said she did give permission for the exchange with Mr Gove, accusing his department of failing to tackle extremism in English schools, to be published.
But questions remained over who did give the go ahead for the release to be placed on the Home Office website last week. She told MPs the inquiry cleared her.