Mr Osborne told the UK's Treasury Committee that a break-up of RBS would be a "very, very complex operation".
He said: "There are lots of problems with it. It sounds like a good idea, in the sense of, 'let's just break up the banks'. When it comes to the practicalities of delivering it, I think it makes you think twice."
Mr Osborne added: "If we had thought that breaking up RBS was a practical proposition, we would have done it. We haven't done it."
And he said: "RBS has shrunk dramatically, it has weak assets."
A spokesman for RBS declined to comment last night on Mr Osborne's remarks.