The Chancellor described the move, agreed with China after two days of talks, as "good for UK jobs and investment".
The Prudential Regulation Authority will enter discussions with Chinese banks in London which will allow them to apply to set up wholesale banks in the UK.
The agreement also contains plans to cement London's position as the global hub for trading in the Chinese currency renminbi (RMB) by giving investors the chance to invest RMB directly into China through London under a pilot scheme.
London will become the first place outside China to be granted an RMB Qualified Foreign Institutional Investor (RQFII) quota, which will initially be set at 80 billion RMB (£8.2 billion).
Mr Osborne said: "A great nation like China should have a great global currency.
"Today we agreed the next big step in making London - already the global centre for finance - a major global centre for trading and now investing the Chinese currency too.
"More trade and more investment, means more business and more jobs for Britain."
The agreement is the result of two days of talks between Mr Osborne and his Chinese counterpart, Vice-Premier Ma Kai, as part of the Economic and Financial Dialogue between Britain and China.
In the joint statement following the meetings, Britain and China said: "Both sides welcomed this as an important step that cements London's major role as one of the most important global centres for RMB trading."