Mr Carmichael said Whitehall had allowed the Nationalists to "hollow out" the UK Government's presence north of the Border and the lesson of the referendum campaign was that this could never be allowed to happen again.
With less than two months to go and with both camps planning their final onslaughts, Mr Carmichael said he was not suffering from referendum fatigue but enjoying the most important campaign of his political life.
Before the last General Election, the MP for Orkney and Shetland was famously hostile to the office he now holds, saying the argument for maintaining it was "indefensible"; his solution was for the offices covering Scotland, Wales and Northern Ireland to be merged into a Department for the Nations and Regions.
Now it appears the Liberal Democrats are wedded to keeping the historic office. One senior LibDem source close to Nick Clegg made clear the party would abandon its previous plan to scrap the role of Scottish Secretary, saying it had become clear the UK Government's voice in Scotland needed to be "enhanced rather than diminished" in light of the referendum campaign.
Mr Carmichael was more circumspect, saying he would not "second-guess where the party will be as we write the manifesto".
But he stressed that, however constructed, there would, under LibDem plans, be a strong Scottish voice in Whitehall and a strong UK Government presence in Scotland.
He said: "Part of the reason we are where we are today is we have allowed the Nationalists to hollow out the role of the United Kingdom Government in Scotland for the last seven years and in the same way the Olympics were about reminding people they had a British identity, the Scotland Office or the UK Government has to be there, reminding people they have two governments."
Mr Carmichael said it was absolutely imperative that, in the event of a No vote on September 18, politicians at Westminster did not slap their hands and say it was "job done" in Scotland and then ignore the nation thereafter.
"Once the vote is over we have to learn the lessons of how we came to be here," insisted the Secretary of State.
"Part of that process has involved the UK Government not being sufficiently visible in Scotland and we can't allow ourselves to go back to that in the future. Whether you do that through somebody called the Scotland Office or some other department, it is not relevant; it's the outcome that matters, not the process."
Mr Carmichael insisted there had been a firm Coalition commitment to putting a "strong UK Government presence in Scotland in the last four and a half years and I want to maintain that".
He referred to a member of his advisory business board saying recently that it had been heartening to see the Permanent Secretary of the Department for Transport in Scotland. "He was absolutely right; that's what we need to keep seeing," insisted the Scottish Secretary.
"It will be the job of the Scotland Office to be making the case for Scotland with the Department for Transport and all the other UK-wide departments to make sure they are coming to Scotland to hear from the horse's mouth the issues and challenges that businesses, for example, have there."
A spokesman for Yes Scotland said: "Mr Carmichael is actually making a compelling case for a Yes vote. Westminster has had its chance and failed Scotland. Only with independence is Scotland guaranteed the full range of powers we need to ensure our wealth and resources are used to create a fairer country."