The crashes happened within hours of each other at the Jim Clark Rally in the Scottish Borders.
Police said the most serious crash took place at about 4pm when a rally car came off the road near Kelso, hitting spectators.
Three were pronounced dead at the scene and a fourth is in a critical condition in Edinburgh Royal Infirmary.
Two hours earlier, a rally car hit five people.
Four were treated at the scene and another is in a serious condition in hospital.
Police Scotland said the the rally was cancelled after the fatal crash.
Official confirmation of the crashes came almost seven hours after the first collision.
Eyewitnesses to the fatal crash described "terrible" scenes.
Spectator Tony Cowan said: ''It was just one car which lost control. It went sideways one way and then to the other side of the road and ploughed into four people.
''It was terrible, absolutely terrible. I ran to help but there was little I could do. The air ambulance arrived after about three quarters of an hour. There were police cars and ambulances. It was chaos, just chaos.''
Another spectator, Tommy Tait, wrote on Facebook: ''We must be like cats have nine lives never have we been so close to getting wiped out by a rally car our thoughts go out to the 4 who got hit.''
The racing competition takes place over three days on closed roads in the Duns and Kelso areas.
The rally is named after Scottish Formula One driver Jim Clark, who grew up in the area. He was killed in a motor racing accident in Hockenheim, Germany, in 1968.
Local MSP John Lamont said: "This is tragic news for the Borders, and my thoughts and prayers are with the victims and their families.
"I know that the rally organisers place considerable emphasis on the safety of the spectators and drivers but with any event of this nature, there will always be an element of risk.
"I live just a few miles away from the scene of the accident and I know that everyone is shocked and saddened by this dreadful news."
First Minister Alex Salmond said: "This is desperately sad and difficult news from the Borders of the tragic circumstances that have occurred at the Jim Clark Rally.
"The police have confirmed three fatalities and my thoughts are with all of those involved and the families and loved ones of those who have lost their lives.
"The government is in contact with Police Scotland and we await the outcome of their inquiries.
"The Jim Clark Rally is a long-standing event of over 40 years. It is much loved in the Borders and by the rally driving community who I know will share in our sadness at what is a black day for the Borders and for Scotland."