The former Labour minister, who was making his first visit to the area since the flooding began at Christmas, said he remained "very proud" of the Environment Agency (EA) staff.
Residents spoke of their anger at Lord Smith's refusal to apologise for the EA's handling of the crisis and said they were not satisfied with the answers he gave to their questions.
Prime Minister David Cameron later saw first hand the damage caused by the floods and vowed that the Government would do "everything that can be done" to help those affected.
He met farmer Tony Davy at Goodings Farm, which is under water in Fordgate, and visited evacuated residents at a private meeting in nearby Bridgwater.
"It's a Biblical scene," Mr Cameron, who arrived by helicopter, said as he surveyed knee-deep water surrounding the farm. "Clearly people here have faced a very tough time."
As the two politicians visited the scenes of devastation, Royal Marines helped evacuate some 140 properties in the village of Moorland.
The Government has faced criticism of its handling of the situation and Alex Salmond yesterday challenged Mr Cameron on the issue after the Prime Minister called on the people of England, Wales and Northern Ireland to send a message to Scotland to remain in the UK.
Mr Salmond said: "The people of England would rather he turned his attentions to issues like the flooding in Somerset or the policies which see a growing wealth and equality gap between London and the rest of England."
Lord Smith insisted the top priority for authorities was "protecting lives", followed by protecting homes and businesses, and said he had "no intention" of resigning.
But local resident Jim Winkworth, a farmer and pub landlord, said he was "bloody mad" at Lord Smith's refusal to apologise.
"We thought that's the least he could do today and he's not apologising or admitting any liability," Mr Winkworth.
"He hasn't come down here to apologise, which is what he should be here for.
"If you apologise it means you're admitting you got it wrong, I made a mistake, I'm sorry, I messed up but he's not fit to do that."