Muriwai Beach, near Auckland, was closed after the fatal attack, one of only about a dozen in New Zealand in the past 180 years.
Pio Mose, who was fishing at the beach, said he saw the swimmer struggle against the "huge" shark. He told the man to swim to the rocks, but it was too late.
"All of a sudden there was blood everywhere," Mr Mose said. "I was shaking, scared, panicked."
Police Inspector Shawn Rutene said the swimmer, in his 40s, was about 300 yards offshore when the shark attacked. He said police went out in inflatable surf- lifesaving boats and shot at the shark, which they estimated was 12ft to 14ft long.
"It rolled over and disappeared," he said.
About 200 people had been on the beach at the time of yesterday's attack. Police said Muriwai and other beaches nearby have been closed until further notice.
Police did not say what species of shark was involved in the attack. Clinton Duffy, a shark expert with the Department of Conservation, said New Zealand was a hotspot for great white sharks, and other potentially lethal species also inhabit the waters.
Attacks are rare, and Mr Duffy estimated only 12 to 14 people have been killed by sharks in New Zealand since record-keeping began in the 1830s.
"There are much lower levels of shark attacks here than in Australia," he said.
He said that during the Southern Hemisphere summer, sharks often come in closer to shore to feed and to give birth, although that doesn't necessarily equate to a greater risk of attack.
"Ninety-nine percent of the time they ignore people," he said. "Sometimes, people get bitten."