The Liberal Democrat's appointment signals a tough new phase in the Coalition Government's approach to the battle for Scotland's future.
The campaign was further bolstered last night as Labour's Jim Murphy said he planned to play a bigger role in Better Together after he lost his position as Shadow Defence Secretary.
Sources close to Mr Carmichael said he would "take the fight to the Nationalists".
The new Scottish Secretary himself said: "I'm not going to take any nonsense from anybody, and I mean absolutely anybody. You can draw your own conclusions there."
Scottish Liberal Democrat leader Willie Rennie said Mr Carmichael would bring a "feisty" approach to the debate after his elevation to the Cabinet in Nick Clegg's surprise reshuffle in the run-up to the poll on September 18, 2014. However, the Orkney and Shetland MP, who replaced Michael Moore, denied he would be a "bruiser," adding there were quite enough already in Scottish politics.
The SNP said the move smacked of panic in the No camp.
Mr Carmichael suggested he would challenge the SNP on the central question at the heart of next year's vote, adding: "This referendum is not about nationhood or whatever else the nationalists might tell you.
"It is about consolidating Scotland's place in the UK so that we can continue the economic growth that has now started, so that we can secure the jobs for the people of Scotland."
Within the LibDems, Mr Moore had been widely regarded as a strong performer who had kept the SNP on their toes.
However, some of his colleagues are understood to have expressed concern that Mr Moore was not combative enough on certain issues.
Mr Clegg, in a letter demoting the Berwickshire, Roxburgh and Selkirk MP, said he felt different experience was required in the run-up to next year's vote.
He told Mr Moore on Friday he was due to be sacked.
Mr Moore said he was disappointed not to be continuing in Government, but was "delighted" for Mr Carmichael.
The axed minister said he was proud of his record, which included ensuring that the referendum would ask a straight Yes/No question.
He said: "The assumption for a long time was that it would be for the SNP and the First Minister to decide everything about the referendum and that we would have no role in that. We turned that on its head."
He added he would continue to be involved in the campaign, albeit in a different way.
Mr Carmichael is thought to be closer to Mr Clegg's inner circle than his predecessor and was widely credited with having done a good job as LibDem chief whip since 2010.
Mr Rennie said Mr Moore was "leaving on a high" after three years in which he had "repeatedly outwitted" Alex Salmond.
He added Mr Carmichael's "feisty style combined with his charm, wit and intelligence is just what we need for the last 12 months in our efforts to safeguard our partnership with the rest of the United Kingdom".
Angus Robertson, the SNP's Westminster leader, said the decision to replace Mr Moore was a sign of panic.
He said: "The problem for the No campaign isn't the messenger, it is the unremittingly negative nature of its message, best summed up by people in the No campaign itself as Project Fear."
The switch was carried out on a day of widespread changes across Westminster.
Prime Minister David Cameron unveiled a new set of Tory junior ministers, which resulted in a number of female MPs being promoted.
Meanwhile his deputy Mr Clegg, the LibDem leader, fired one of the party's rising stars, former Home Office minister Jeremy Browne, reportedly for being "too right wing".