Mr Moore is said to be the only Lib Dem Cabinet-level casualty planned, with Deputy Prime Minister Nick Clegg thought to want a more attacking approach in the run-up to the referendum.
The Prime Minister announced the appointment of Mr Carmichael on Twitter, also confirming that he will be making wider changes later. He is expected to promote women and northern MPs in what has been branded a "flat cap" reshuffle.
Nick Clegg said "different experience" was required in the run-up to the independence referendum as he sacked Michael Moore from his post.
The Deputy Prime Minister and Liberal Democrat leader left the door open for a possible return to the front bench for Moore in future, telling him he had "no doubt there will be an opportunity for your talents to be deployed" in government again.
A letter from Clegg revealed that the Lib Dem leader told Moore he was going to be axed on Friday.
Clegg said Moore became Scottish Secretary at a "critical time" in the country's relationship with the rest of the UK and "managed the challenges of the situation with great skill and effectiveness".
But he added: "As we discussed when we spoke on Friday, I believe we now need to draw on different experience in the final year running up to the referendum itself and I am keen that, just as we have benefited from your formidable skills over the past three years, that we take advantage of other experience within our ranks during this period.
"I am immensely grateful for all the work you have done at the Scotland Office and for the very significant contribution you have made to the first coalition government in 70 years. I have no doubt that there will be an opportunity for your talents to be deployed in government in the future."
Moore replied: "This has been, and will continue to be, a hugely important time in Scottish politics and that has made it a challenging and rewarding time to be Secretary of State for Scotland."
He added: "In leaving the Scotland Office I am pleased that Alistair will be succeeding me. As a good friend and long-time colleague, I believe he will do a superb job. I wish him all the best."
Moore said he was disappointed, but that he would continue to play a role in ensuring Scotland remains a part of the United Kingdom.
He said: "I'm disappointed to be leaving office right now. But I'm very pleased at what I have been able to achieve in the last couple of years, particularly in the constitutional debate on the Scotland Act and the Edinburgh Agreement."
The Scotland Act 2012 had the aim of devolving further powers to Scotland, while the Edinburgh Agreement (2012) enshrined in law a vote allowing the Scottish public to say whether or not they wish to remain to leave the UK.
Moore added: "But this big decision that we're taking as a country is bigger than one individual, bigger than one party. I look forward to continuing to play a really big role in the constitutional debate over the course of the next 12 months."
SNP Westminster leader Angus Robertson MP said: "Michael Moore was a courteous opponent, and he can take pride in the constructive role he played in the negotiation of the Edinburgh Agreement, which helped paved the way for next year's independence referendum. I wish him well for the future.
"Nicola Sturgeon bested Mr Moore in a Scottish Television referendum debate earlier this year, and the fact that the case for Yes has now prevailed over No in a series of broadcast debates may have played a role in what can only be viewed as a panic reaction by the UK Government in sacking him.
"Mr Carmichael's new role in the 'indefensible' Scotland Office is presumably to help spread more scare stories about Scotland and independence - something the Tories must have believed Michael Moore wasn't sufficiently adept at or committed to.
"In any event, this panic reaction by Westminster indicates serious problems in the No campaign - which appointing someone to the post who believes the post should be abolished will do nothing to solve."
Carmichael, born in 1965, was raised in the Western Isles and graduated from Aberdeen University with a law degree, later becoming a procurator fiscal in the north-east.
Elected to Westminster in 2001, he replaced Jim Wallace as MP for Orkney and Shetland.
He lives in Orkney with his wife, Kate, and two sons, Sandy and Simon.
Mr Carmichael was appointed deputy chief whip of the party in 2010 and became deputy leader of the Scottish Lib Dems last year.
Widely seen as more combatitive than Mr Moore, his appointment was welcomed within the party as a good move for the campaign to keep Scotland within the UK.
Scottish Lib Dem leader Willie Rennie said: "Mike Moore is leaving on a high after three very good years as Scottish Secretary.
"He persuaded the nationalist Government to approve the Scotland Act they previously condemned as a poison pill. Against the odds he secured an agreement with the nationalists on the conduct of the referendum. Mike Moore repeatedly outwitted Alex Salmond.
"As we move to the next stage of the campaign, I am looking forward to working with Alistair Carmichael. His feisty style, combined with his charm, wit and intelligence, is just what we need for the last 12 months in our efforts to safeguard our partnership with the rest of the United Kingdom."
Tory Esther McVey has been pushed up the ranks at the Department for Work and Pensions to become employment minister, while Greg Hands - a close ally of Chancellor George Osborne - has been made deputy chief whip for the Conservatives.
Greg Clark has been moved from the Treasury to take on responsibilities for cities and constitutional reform at the Cabinet Office, and former Northern Ireland minister Mike Penning becomes a minister at Work and Pensions.
Sajid Javid moves up a rung to replace Greg Clark as Financial Secretary to the Treasury, while Mr Clark moves to the Cabinet Office to become Cities and Constitution Minister.
Mike Penning is leaving the Northern Ireland brief to become a minister at the Department for Work and Pensions.
In one of the biggest shocks so far, Home Office minister Jeremy Browne - on the right of the party and touted by some as a future leader - was also replaced by transport minister Norman Baker.
Exchanging letters with Browne, Clegg said it was "always very difficult to move colleagues out of government".
"But as you know, I have always been keen that we provide the opportunity for as many in our ranks as possible to contribute their skills to ministerial office during this Parliament so that, just as the Government has benefited from your contribution over the past three years, it can also gain from those of other colleagues in the remaining years of this Parliament," he wrote.
In his response, Mr Browne insisted he remained "supportive of the Government".
"I hope the Government will continue to strive to be reforming and innovative and avoid the danger of lapsing into transactional trade-offs and deferred decision making," he added.
Hugh Robertson, who has held the sports brief for the Tories since 2005 and earned praise for his work on the Olympics, is rewarded by being made minister at the Foreign Office.
Camborne MP George Eustice - Mr Cameron's former press secretary and a prominent EU rebel - has been appointed environment minister.
The shake-up is expected to see women and northern MPs promoted to more senior posts as the Prime Minister gears up for the 2015 election. No further changes are due to the Cabinet.
Don Foster, who was a junior minister in the Department for Communities and Local Government, has replaced Carmichael as Lib Dem chief whip.
Richard Benyon, an early supporter of Mr Cameron's Tory leadership bid in 2005, confirmed that he had stepped down as environment minister.
"On back benches! 3 and half really fun years with much achieved. Really appreciate time working with outstanding Ministers and Officials," he wrote on Twitter.
Mark Hoban has left government after being ousted as employment minister, bringing to an end a 10-year spell on the Conservative front bench.
Another Tory, Mark Prisk, posted: "Asked to step aside from Housing for a younger generation. Disappointing but it's been a great eleven years on front bench."
Nicky Morgan has been promoted from assistant whip to Economic Secretary to the Treasury, No 10 confirmed.
Alistair Burt, who leaves his job as Foreign Office minister, posted on Twitter: "Standing down today. The last 3 1/2 years have been a pleasure working on an extraordinary portfolio with talented people. Thank you all!"
David Heath was sacked from his post at the Department for the Environment, Food and Rural Affairs, with Lib Dem MP Dan Rogerson appointed to a junior ministerial job in the department.
Stephen Williams, the Lib Dems' former backbench Treasury spokesman, has been given a post in the Department of Communities and Local Government.
Tory Jane Ellison has been promoted from the backbenches to become a junior minister in the Department of Health, No 10 said.
Tory Anna Soubry has been moved sideways from her junior ministerial role in the Department of Health to the Ministry of Defence.
Earlier this year the former television presenter said she supsected Mr Cameron gave her the job of public health minister because she is a woman and it was seen as a ''soft bloody girly option''.
But she told Total Politics: "'It's bloody well not, it's one of the most important jobs.''
Mr Cameron's parliamentary aide, Sam Gyimah, has been moved to the Whips office along with Karen Bradley.
MPs Amber Rudd, Claire Perry, Gavin Barwell and John Penrose have been appointed as assistant Whips.
Cabinet Office minister Chloe Smith, deputy chief whip John Randall and transport minister Simon Burns stepped down ahead of the shake-up of the Tory team.
Ms Smith, who was only elected to the Commons in 2009 at the age of just 27, said in her resignation letter last night that she wanted the chance to "develop other ways of giving public service, both inside and outside Parliament".
As economic secretary to the Treasury she once suffered a humiliating verbal mauling at the hands of Newsnight presenter Jeremy Paxman over a fuel duty freeze.
The interview led to Chancellor George Osborne being branded an ''arrogant coward'' after sending her into the TV studios to defend his decision to delay the 3p hike in fuel duty.
Mr Cameron applauded her "positive impact in the departments you have served" and said he looked forward to receiving her recommendations on encouraging young people to vote.
"As you know from our previous conversations, this is a topic I take seriously and believe that there is scope for action," he told her.
Mr Randall, 58, has been the Conservative MP for Uxbridge and South Ruislip since a 1997 by-election and told the Prime Minister at the end of last year that he wished to step down.
He said it had been "a great privilege and honour" to serve for 13 years in the Whips' Office in opposition and Government.
The Prime Minister said he "could not have wished for a more loyal, discreet, patient, trustworthy and committed colleague" and that he "had rather hoped this day would never come".
Meanwhile, Labour leader Ed Miliband is also preparing to make changes to his top team.