Mark Shand, 62, was taken to hospital on Tuesday night after reportedly slipping while lighting a cigarette and hitting his head on the pavement outside an after-party for a charity event.
Camilla, who is preparing for a high-profile tour of Canada with the Prince of Wales next month, was said to be "utterly devastated" by the news.
A Clarence House spokesman said: "It is with deep sadness that we have to confirm that the Duchess of Cornwall's brother, Mark Shand, has today passed away in New York.
The spokesman said Mr Shand died in hospital as a result of a serious head injury which he sustained during a fall.
"The Duchess, the Prince of Wales and all her family members are utterly devastated by this sudden and tragic loss.
"Mark Shand was a man of extraordinary vitality, a tireless campaigner and conservationist whose incredible work through the Elephant Family and beyond remained his focus right up until his death."
Mr Shand was the chairman of the Elephant Family charity and is known for his work as a conservationist and travel writer.
He was in New York for an auction held at Sotheby's in aid of a charity for underprivileged children and the Elephant Family, which aims to save the endangered Asian elephant from extinction in the wild. It was the finale for the month-long Big Egg Hunt NYC by Faberge, which saw egg sculptures sold to the highest bidder. The charity event raised $1.6 million (£950,000) for charity and was also attended by Princess Eugenie, according to reports.
A spokesman for the Elephant Family said: "Today we have lost the head of our family. Mark Shand was a true force for conservation.
"He was both a legend and inspiration and above all our great friend. We ask at this time that people's thoughts are with his loved ones. We will miss him always."
Mr Shand, Camilla's only brother, wrote numerous books and featured in documentaries, with elephants and India as recurring themes. The adventurer's book Queen Of The Elephants won the 1996 Thomas Cook Travel Book Award and the Prix Litteraire d'Amis, and was made into a BBC documentary.