A large sea snail, two kinds of clam and a marine worm were found during Marine Scotland surveys around the Rockall plateau in the north Atlantic.
The creatures had previously avoided detection despite decades of research in the area, located hundreds of miles off the north-west coast.
International experts have confirmed the molluscs were previously unknown to science and the discovery is said to be potentially very significant.
Researchers say the existence of the clams and worm at a single site could indicate the presence of a cold seep, where hydrocarbons are released from the sea bed. If confirmed it would be the first cold seep to be discovered in the area.
Jim Drewery from Marine Scotland Science, who oversaw the research on the deep water invertebrates, said: "The discovery of these new species is absolutely incredible, especially when you consider that the sea snail measures a relatively large 4 inches yet has gone undetected for decades.
"The project we were undertaking was designed to provide advice that would help balance both commercial fishing and conservation interests in the Rockall area.
"The potential cold seep and its dependant community of marine life is a great find as it is just the sort of habitat we were hoping to pick up on these surveys."
He added Marine Scotland Science had been able to narrow down the location of what they believed was the cold seep to a small area west of the Hebrides in the Rockall-Hatton basin.
Mr Drewery said: "Further research is now needed, which would involve going down to take a look at the ocean floor three-quarters of a mile underwater."
He said he was particularly excited by the discovery of the marine worm Antonbrunnia which is the first of its kind to be found in the Atlantic.
It was discovered by international bivalve expert Dr Graham Oliver inside one of the clams he was confirming as a new species at his laboratory at the National Museum of Wales.
Scotland's Environment Secretary Richard Lochhead said: "Scottish waters cover an area around five times bigger than our land mass and are miles deep in places, and these hidden gems offer a fascinating glimpse of the treasures that still await discovery under the waves.
"The area where these species were found is not fished and the confirmation of a cold seep is likely to result in the region being closed to bottom contact fishing."
WWF Scotland director Lang Banks said: "Perhaps only in Scotland would you find people getting excited about something known as a 'cold seep'.
"However, if true, then it is no less important a discovery as the much better known hydrothermal vents, found in other parts of the world. They would give us a unique opportunity to observe some species unlikely to be found anywhere else on the planet.
"These latest discoveries underline the need for a precautionary approach in the management and use of our seas."
The clams and marine worm were discovered around 260 miles west of the Hebrides at a depth of about three quarters of a mile.
The sea snails were found in water around a mile deep over an area approximately 80-260 miles west of the islands.
The sea snail Volutopsius scotiae and clam Thyasira scotiae have been named after research vessel MRV Scotia while the clam Isorropodon mackayi has been named after mollusc expert David W Mackay.