Epoch-defining trade deals are as much about international powerplay as they are about selling more stuff to each other in the form of goods and services. Even so, the debate around the EU-US TTIP (usually pronounced T-tip) is taking political noise to the extremes. The furore is threatening to overwhelm what started out as an attempt to shape the future of global commerce, but has ended up seeing Brussels and Washington pulling in different directions.

The Transatlantic Trade and Investment Partnership, the 10th negotiating round of which ("services and regulatory coherence") has just concluded in Brussels, is touted as one of the blockbuster negotiations of the post-crash 21st Century, potentially setting the "gold standard" for all trade deals to come.

By improving market access, addressing regulatory mismatches and non-tariff barriers, and by sorting out rules on intellectual property, customs and trade, it was meant to supply the missing trade link between the friendly giants of global trade, while a counterpart TPP (Trans Pacific Partnership), laid out equivalent standards between the US and various South American and Asian countries, before a resurgent China got to impose the rules (see panel).

In Brussels and Washington DC, the Sunday Herald has been given unique access into the workings of the TTIP-making machine, attending events such as the European Commission's "Stakeholder presentation events" in which a vast array of industry groups, quangos, public services, unions and NGOs set out their hopes and fears for TTIP in open forum, to the attentive ears of the two negotiating teams.

We talked to the lead negotiators, the US State Department and European Commission officials, the lobbyists, the trade unions, the business organisations, the trade technocrats, the parliamentarians, the consumer interests, the think tanks and the business organisations. Perceptions of the desirability of the prospective deal vary greatly, in line with debates about free trade and globalisation. But most agree that what was conceived as a shot in the arm for leading economies - Europe's especially - and a last-minute prop to the Obama administration's legacy, is not working out as planned.

In a hot conference room in the labyrinthine corridors of the US State Department, Under Secretary for State for economic growth, energy and the environment Cathy Novelli talks, perhaps wishfully, about a "new age of momentum" for the TTIP deal following important votes of confidence in the Congress and the European Parliament.

Like many other supporters of TTIP's aims on both sides of the Atlantic she betrays frustration with the "sniping", particularly in the European discourse over TTIP, which Washington sees as residual suspicion of US business, and a tendency to focus on snags rather than opportunities. They bristle at perceptions that America has generally lower regulatory standards, than the EU, a fact that their EU counterparts also tend to support. They're not lower, they're just different, in some cases stricter.

"Why are we doing the TTIP?" Novelli asks rhetorically "If you think about the US-EU relationship, we have one third of the world's goods and services production, we have over $4 trillion in investment between the two of us, half of all US foreign direct investment is in Europe, we are the largest source of FDI in Europe and two-way trade in 2014 was $700bn and 13m jobs are supported from this trade.

"The whole idea of TTIP was to do what we can to strengthen and deepen, and make it easier for all that economic activity to flow in a way that still continues to protect public health and safety and our differences, which we celebrate, while putting some special attention on SMEs which drive the US economy as well as the European economy. It is sometimes hard for SMEs to have the wherewithal to deal with barriers, navigating regulatory approvals, we want to do what we can to strengthen SMEs to engage in transatlantic trade and more globally as well."

Nobody foresaw so much of a problem achieving a landmark deal that streamlined regulation, standards, and investment protections and which allowed SMEs from both sides of the Atlantic to penetrate each other's markets in ways that only multinationals currently can. After all it was building on Europe and America's existing shared values, low tariff regimes and general goodwill.

All of which should be good news for Scotland, an SME-heavy economy with especially strong trading and investment relations with the USA. When Alex Salmond and then EU trade commissioner Karel de Gucht attended discussions of TTIP and its potential benefits for US-facing Scottish exporters at Gleneagles in September in 2013, it all looked easy enough, with initial projections for a deal by the end of 2014, that could grow Scotland's sales to its largest (by far) overseas market from the current £3.9bn or 14% of total exports. And for consumers generally there would be wider choice and lower prices on both sides of the Atlantic.

Since then progress on TTIP has slowed to walking pace, as US and EU teams try to grind out agreement against a background of sometimes shrill social media-borne public protest and complaint. Experienced Brussels operators are downplaying expectations of an early deal. Meanwhile politicians from the SNP to UKIP to the unions to some parts of Scottish Labour, who are more fixated on self-proclaimed democratic priorities than on trade and business, are joining the "Stop TTIP" team, along with anti-Brussels Tories like Dan Hannan MEP, who has distinguished between a "pro-business" deal like TTIP and a genuine pro-free trade agreement. All of this commentary has been fanned by social media and Wikileaks of "secret" negotiating documents, which is clearly setting up opposition to this "Trojan horse treaty" as a progressive badge of honour.

Opponents of TTIP accuse the negotiators of acting "behind closed doors", although Cecilia Malmstroem the European trade Commissioner co-sponsoring with the US Trade Representative Michael Froman, argues that TTIP is the most transparent trade process ever, and that total openness about each side's position is not possible if anything is to be achieved at all. Showing "red lines", as both sides have had to do in response to domestic pressure, tends to be a compromise killer that could result in a damp squib of an outcome.

Meanwhile the onset of the US Presidential election means that TTIP, despite the delicacy of the diplomacy surrounding it, is being placed squarely in the political football pitch. Anything could happen, as Republicans and Democrat nominees sniff the wind for potential advantage. Already Hillary Clinton, has watered down her support for TTIP and TPP, aligning herself with the American left's traditional hostility to free trade (although the American trade union leaders the AFL-CIO is "absolutely not opposed to TTIP" as a means of importing European good practice in employment and public services.)

What promoters describe as "sniping and irresponsible fearmongering " against TTIP, mostly in Europe is directed at what one US administration source describes as "straw men", and the confusion of the normal discretion necessary in a delicate negotiation with undercurrents of secrecy and corporate conspiracy against the public.

An active trade union-led campaign against the deal in the UK has raised fears of a takeover of the NHS by American companies, a lowering of food and safety standards, and the notorious "Investor State Dispute Settlement" (ISDS) seen by its opponents as a means of giving US companies the right to sue European governments for loss of profits when legislating in the public interest.

The level of public noise surrounding these and other issues is hampering what is already a tricky negotiation. But why is it so difficult? We are supposed to be friends after all.

In fact, counter-intuitively, a deal between two equal, similar partners is far harder than one between two wildly diverse systems - even old enemies like the US and Vietnam. Neither Washington nor Brussels has ever dealt with an equivalent big beast before, and neither is inclined to give up their accustomed negotiating "template" to accept the other party's.

Also, as with all negotiations, the devil is in the detail, far more than was originally assumed. "I guess we didn't know each other as well as we thought we did" one official told the Sunday Herald ruefully.

"It's less than 50-50 that TTIP will get to the floor of Congress by the end of the Obama administration, when it would get caught up in the meat-grinder we call the Presidential election process" says Dr Stephen Billet, head of the graduate school of political management at Washington's George Washington University. Shaun Donnelly of the US Council for International Business estimates that "we're only 30-35 per cent of the way through" the negotiation.

The likelihood of TTIP taking years rather than months is reinforced by the brave face being adopted by officials, who say it would "not be a disaster" if TTIP was postponed well into the next administration. They point out that trade agreements (see also Uruguay Round, NAFTA et al) have tended to span administrations, while their risks and benefits were hyped up beyond reality. "It's optimism that keeps us going" one senior official says wistfully.

So is TTIP all about "big bad American companies coming over to Europe and doing bad things" as Marjorie Chorlins of the US Chamber of Commerce scathingly puts it with withering irony? If so, then both the EU and the US will have been guilty of one of history's most barefaced bluffs. The truth is that it is impossible to judge if TTIP is treasure trove or trap until the actual treaty is before us. Until then, the opponents might want to dial down the heat they are generating on hypothetical ideas about the deal, unless of course it is politically expedient for them to do just that.

TTIP: how did we get here?

Traditionally, the progressive US tradition of labour unions and environmentalists that nurtured Barack Obama, has not been friendly to free trade, so where did these matching pair of deals TPP and TTIP come from? According to the economist Fred Bergston, Director Emeritus of the Washington think tank the Peterson Center for International Economics, Obama initially wanted to "do nothing on trade" but the impetus came from Lee Kuan Yew, the late Singaporean leader who convinced Obama in 2009 that he must forge closer links across the Pacific. "He said 'Mr President, unless you take the initiative in Asia, you are going to give Asia to China.' It was only then that Obama started to make speeches about how China will write the rules unless the US acts."

"Once he had launched that, then they had to agree with the Europeans who said, having done this towards the Pacific Rim, you can't do anything less for us."

Although Obama's famous "pivot" towards Asia, and the fact that TPP preceded TTIP means that the former takes priority, Bergston points out that circumstances have increased Europe (and NATO's) importance to the US, with the rise of ISIL, nuclear tensions over Iran and Putin's activities in the Ukraine keeping US committed to European diplomacy, despite its exasperation with failure to fix the Eurozone.