They are asking the Court of Appeal to rule that a decision to create a marine park around the British-controlled islands in the Indian Ocean banning fishing is unlawful and violates their human rights.
The decision to create the park was upheld by the High Court in 2013, but lawyers for the exiled Chagossians say the judges went wrong in law and misdirected themselves.
The case raises for the first time the question whether a document leaked by WikiLeaks is admissible in evidence in an English court.
The park - referred to as a marine protected area (MPA) - was set up in 2010. Commercial fishing was banned through "no take" rules in areas where the former residents and their descendants would like to earn their living from fishing.
They say the reserve was created for the "improper purpose" of depriving them of a livelihood and discourage their return.
Many Chagossians gathered outside the Royal Courts of Justice in London to protest and support their leader, Louis Bancoult, as he brought the appeal.
Nigel Pleming QC, for Mr Bancoult, told the appeal judges the last islanders were removed in May 1973 for the Americans, who built a US Air Force base on the largest island, Diego Garcia.
Mr Pleming said what had "echoed down history from the 1960s" and in a series of legal battles for the right of return was the "deeply offensive" reference to the islanders as "Man Fridays".