A 72nd-minute strike from Ricky Lamie kept the hosts in the competition as David Bingham's Lowland League side threatened to pull off an upset.
Queen's Park led at half-time thanks to Blair Spittal's 32nd-minute goal, but Jamie Brown equalised with a right-foot shot after 58 minutes and Gary McCormack put the East Lothian team ahead before Lamie's intervention.
Whitehill Welfare produced similar heroics at Forthbank, drawing 2-2 with Stirling Albion. The home side's Jordan White had an early penalty and Wayne McIntosh then put Whitehill in front. Kieran McAnespie equalised after 38 minutes and White made it 2-1 early in the second half, but Steve Manson netted to force a replay.
Buckie Thistle held visitors Annan Athletic 0-0. The League Two side had Steven Sloan sent off after 70 minutes.
Doubles from Alan McKenzie and Craig Gunn gave Elgin City a 4-0 win at Keith and Mark McGuigan scored the only goal as Albion Rovers won 1-0 at home to Spartans, who had John Grant dismissed.
Berwick Rangers came from behind to beat 10-man Peterhead 2-1 at Shielfield Park. Rory McAllister had Peterhead ahead at half-time but Darren Lavery equalised after 53 minutes. The visitors had David Cox sent off late on before Kenny O'Brien hit Berwick's winner.
Clyde beat Gala Fairydean Rovers 3-0 with Kevin Watt, Michael Daly and Stuart McColm all on the scoresheet. Colin Wilson scored the only goal as Montrose won 1-0 at Dalbeattie Star.