Bob Dudley's comments were seized on by opponents of independence, with Chancellor George Osborne saying they represented "a further example of the potential economic damage ... and cost to the people of Scotland" of a Yes vote.
In contrast, First Minister Alex Salmond stressed Mr Dudley was expressing a personal opinion, that there was massive ongoing investment in the North Sea and there were "many chief executives in favour of independence".
The BP chief's intervention, the most significant of any business leader so far and made days after remarks by the Bank of England governor on currency union, suggests the referendum debate is moving into a higher gear.
Mr Dudley said of independence: "It would create extra costs for our business. We have to duplicate the centres and do things. And again the currency question, I don't know the answer to."
BP's future in the North Sea in the event of independence would, he said, "depend on what it really led to".
But the company chief stressed BP would continue to invest in the UK industry; its current plans amount to £10 billion between 2011 and 2016.
He said: "These investments are big, they are under way, we want to see them developed. It depends on what tax regimes are there and on currency.
"We have a lot of people in Scotland, we have a lot of investments in Scotland. These are quite big uncertainties for us and we are continuing to invest at a pace because these projects are under way. But it's a question mark.
"All businesses have a concern. My personal view is Great Britain is great and it ought to stay together.
"It does not seem the right thing to me for the country to drift off. That's not a company view, that's from me."
The BP boss's remarks, ahead of David Cameron's keynote speech on Scotland on Friday, were seized upon by Downing
Street and Better Together. The PM's spokesman said: "It is important when one of the UK's leading businesses and investors and an important employer raises this issue."
Alistair Darling, the former chancellor who heads the No campaign, said: "Bob Dudley is quite right to express concern about the issue of currency. It is far from certain what currency we would use if we leave the UK."
A spokesman for the Scottish Government said it welcomed all contributions and would be happy to meet Mr Dudley to discuss the future of the industry in an independent Scotland.
He added: "BP already operates in more than 80 independent countries around the globe and an independent Scotland with full control of its economy and huge resources will offer an attractive and stable environment for businesses in the offshore and other sectors.
"And an independent Scotland will keep the pound as part of a sterling area, which is in the overwhelming economic interests of the rest of the UK."
Meanwhile, answering questions from the House of Lords Economic Committee, Chancellor George Osborne claimed Bank of England Governor Mark Carney's speech last week had "effectively demolished" the First Minister's assertions on a currency union.