Chief executive Ralph Roberts said the company took the decision to withdraw its application to operate the Mediterranean island's main transport system because of lack of openness by the Maltese government.
He also said McGill's had a "gut feeling" that excessive state interference would prevent it running a profitable service.
The U-turn, which comes a month after the Greenock-based company announced its bid for the contract, sends the £35 million-turnover firm back to the drawing board in its efforts to find expansion opportunities in highly competitive domestic and international markets.
McGill's said that the attraction of Malta was the absence of the giant, well-networked rivals which have carved out profitable urban routes throughout UK and Europe.
The Maltese government is looking for a new operator for Malta and neighbouring Gozo after the Sunderland-based multinational Arriva withdrew from the densely populated islands after sustaining losses of €70 million (£57m) in under three years.
The Maltese bus service, previously a patchwork of micro operators, was reorganised as a single contract in 2011, with Arriva being the first operator. On January 2 the centre-left Maltese government bought the shares of Arriva Malta, and its agency Transport Malta is now running it as a nationalised service.
Roberts, who was previously Arriva's west of Scotland director, said: "We decided last week that we're not going to pursue [the bid] any further. There were more questions than answers. I submitted about 30 questions where I needed further information which they have not provided.
"Their first option was to have someone take over an effectively nationalised service and assume the assets and liabilities.
"While they made that clear up front, they didn't say what this amounted to, so we had no idea what the liabilities were, what the assets were, what the revenue was, or the patronage numbers.
"I have been waiting three weeks for that [information] and we got to the point where if we are going to put a robust submission together we have gone past the point of no return. There were too many holes, too may gaps in the information."
Roberts, who has made several trips to the Maltese capital, Valletta, has held discussions with potential local partners and had lined up high-profile people involved in public life as potential non-executive directors.
"We're businessmen so we're analytical, but we operate on gut and it just didn't feel right. I've been involved in negotiations all over Europe, and the Maltese government's way is not the way you do negotiations."
"[Governments should] get all the information that's needed out as soon as possible, to allow companies that are bidding to get as much time as possible to get something together before the March 24 deadline. This is not the way they are working; they have a very different culture and it wasn't sitting well with us.
"We heard stories that it was difficult to pin down the Maltese government to fulfil their promises. On the other hand they were very eager to get the franchise-holders to deliver their contractual promises. It was a one-way street, in other words."
Roberts said that the company had been attracted to Malta as it offered a relatively small player a "level playing field" which would have enabled McGill's to double in size.
However, he noted that Arriva had made predictions about passenger growth that never materialised, partly because of government control of routes.
McGill's, which has roots going back to the 1930s, doubled in size between April 2010 and March 2012, and doubled again after acquiring rival Arriva Scotland West in 2012.
It is owned by the Easdale brothers, who were recently reported to be looking for buyers for the company and increasing their investment in Rangers International Football Club.
The company operates about 350 buses across more than 50 routes, mostly in Inverclyde, Renfrewshire and East Renfrewshire.
Transport Malta, an agency of the Maltese government, did not respond to the Sunday Herald's requests for comment.