Factors causing the landslips included "absent or ineffective" drainage, said the report from the Rail Accident Investigation Branch (RAIB).
Many Network Rail (NR) earthworks were constructed with steeper slopes - leading to a greater likelihood of landslips - than would be achieved with modern design procedures, the RAIB added.
It looked at six incidents, including three landslips in Scotland. The report said: "NR's ongoing earthwork improvement programme is unlikely to achieve modern criteria in the foreseeable future."
The RAIB, which made five safety recommendations, said: "The landslips were caused by factors including heavy rain, absent or ineffective drainage and activities undertaken, or not undertaken, on neighbouring land.
"In several instances trains were being operated without special precautions when there was a significant risk of encountering a landslip."
In two of the incidents in Scotland, trains were derailed. The report said that in the third case, a passenger train which hit debris near Falls of Cruachan in July 2012, the danger was that "if the train had been derailed by the landslip debris, it is possible that it would have fallen down an adjacent slope".
The report continued: "There is a lack of clarity about who should be carrying out visual checks for risks which can develop on neighbouring land between examinations which take place at intervals of up to ten years. The mandated process for collecting information about neighbouring land is, in parts, difficult to implement and not usually followed."
A Network Rail Scotland spokesman said: "We have already changed our operating procedures in response to these incidents and will carefully consider the RAIB report and take any further action as appropriate."