Rhodes, who scored a hat-trick for Scotland Under-21s on his previous trip to the Stade Josy Barthel, headed home from close range on 10 minutes and then slotted a simple second.
However, Scotland failed to build on their lead and survived some nervous moments after Lars Gerson's well-placed free-kick brought Luxembourg back into the game 90 seconds after the break.
Both Scotland goals were followed by cries of "Are you watching Craig Levein?" from the visiting support, who vastly outnumbered the home fans in the attendance of 2,521.
Their mood was no doubt lifted by caretaker manager Billy Stark's 4-4-2 formation with Kenny Miller partnering Rhodes up front and Steven Naismith and debutant Andrew Shinnie playing in advanced wide positions.
Levein, who paid the price for taking two points from Scotland's first four World Cup qualifiers, had last played two strikers from the start in a 2-1 win against Liechtenstein at Hampden in September 2010.
Inverness midfielder Shinnie was the only debutant in the starting line-up but there were rare starts for Matt Gilks and Grant Hanley, while Charlie Mulgrew lined up in central midfield.
The Celtic player saw a lot of the ball in the opening stages as Scotland dominated possession on a pitch that was already cutting up.
Rhodes had his first chance when Luxembourg defender Guy Blaise missed his kick after Naismith had helped the ball up the left, but the Blackburn striker was well wide as he steered the ball past the onrushing goalkeeper.
However, Rhodes only had two minutes to wait for the opener after more poor defending.
Scotland worked the ball to Paul Dixon on the left wing and he sent over an inviting cross which Ante Bukvic turned against his own post. Rhodes was on hand to head the ball over the line from close range.
Gilks made his first save in the 16th minute after Naismith had lost right-back Tom Schnell at a short free-kick. The defender sent over a deep cross that was met by Mario Mutsch but the Blackpool goalkeeper got down to hold the header.
Mutsch soon fired wide from 22 yards after Scotland again found themselves a man short down their left.
Miller curled over after a Rhodes lay-off before the Blackburn forward hit his second in the 23rd minute. Shinnie collected Whittaker's cross from the right and hit the byline before driving the ball towards goal.
His effort was blocked but fell for kindly for Rhodes who slipped the ball home with his left foot from six yards.
Miller was soon through on goal after a series of short, sharp passes with Naismith but the flag was raised as the striker shot straight at Jonathan Joubert.
Scotland continued to control possession but they were almost caught out before the break when Mutsch broke down the left and sent over a low cross to right-winger Charles Leweck, whose drive was held by Gilks.
Kilmarnock midfielder Liam Kelly replaced Mulgrew at half-time to make his debut but he soon gave away a foul that allowed the hosts back into the game. The execution was superb as Gerson curled his 25-yard free-kick into the top corner to leave Gilks with no chance.
Scotland continued to dominate possession but were making little headway with Naismith growing frustrated at the number of fouls committed by the home defenders.
Leigh Griffiths became the third new cap in the 70th minute when he replaced Shinnie with Miller moving wide right.
Scotland soon had a let-off after a move broke down on the right with Whittaker out of position.
Hanley backed off after David Turpel collected the ball and the Luxembourg forward turned and sent the ball inside him and out to Mutsch.
The home captain's cross found Stefano Bensi in a great position but Gilks did enough to unconvincingly palm the header wide for a corner.
Before the match, Stark had signalled his intention of making six substitutions but the game was in the balance and Scotland's formation, now looking more like a 4-2-4, was giving the hosts space to attack.
Rhodes thought he had his hat-trick as he headed home Kelly's cross but the flag was immediately raised and Bensi tested Gilks again with a 20-yard drive that the goalkeeper got down well to hold.
Gilks then saved Turpel's header before Murray Davidson won his first cap as the midfielder came on for Rhodes to see help see out the match in injury-time.