Lord Howell, who caused a furore last year when he suggested fracking could go ahead in the "desolate" North East (of England), said starting shale exploration in southern England, the Home Counties or in rural areas would cause longer delays, more hostility and higher costs.
When ministers said fracking should go ahead across Britain, not just in remote and derelict areas, they lost thousands of Tory votes, the peer, who is Chancellor George Osborne's father-in-law, said.
In an article for the US-based Journal of Energy Security, the former energy secretary also warned that the view on the unconventional fossil fuel source from ministers was much too optimistic.
As part of efforts to get local people to accept the controversial hydraulic fracturing drilling technique, communities are being offered incentives to host shale gas wells.
But drilling in the UK has faced opposition from people concerned about the impact of traffic and development as well as potential water pollution, minor earthquakes and the impact on efforts to cut climate change emissions.