The Department for Work and Pensions' role in influencing the Smith Commission’s "disappointing" package of new welfare and employment powers for Holyrood will be kept under wraps after the release of key documents was blocked.

The UK Information Commissioner's Office (ICO) has ruled that correspondence between Iain Duncan Smith's department and the cross-party group, which decided on the new package of powers for the Scottish Parliament in the wake of the independence referendum, could interfere with the Scotland Bill if made public.

The Herald asked to see the documents through Freedom of Information laws after groups across Civic Scotland said the welfare settlement did not go far enough and the DWP faced allegations that it had played a key role in watering down initial, more radical proposals.

However, the request and subsequent appeal has been turned down with the ICO, in a provisional decision, backing the UK Government. It said that while the information was "of importance to the public" potential prejudice to the "formulation and development of government policy" as the Scotland Bill makes its way through Westminster meant "the exemption currently outweighs the arguments for transparency."

While the Smith Commission recommended the devolution of some welfare powers, such as handing Holyrood the ability to top up benefits, vary aspects of universal credit and decide on disability payments, large sections of the welfare state are to remain reserved with just 14 per cent of spending to be under direct control of MSPs.

Among the benefits to remain in Westminster control include child benefit. Universal Credit, which is being rolled out UK-wide and includes tax credits and jobseeker's allowance, will remain administered and delivered by the DWP.

Sources close to the talks said that reaching a welfare agreement proved by far the most challenging area for the Commission while there were a series of "frantic" exchanges with the DWP in the days before the deadline.

Patrick Harvie, the Green MSP who took part in the talks, expressed disappointment that the release of the documents had been blocked.

He said: "The Smith Commission never lived up to the promises of an open and participative process, and it seems now that the UK Government wants to keep its contribution under wraps nearly a year after the Commission reported. I’d urge full openness about the DWP’s submissions, and I hope that other former members of the Commission will say the same."

The SNP had lobbied for all welfare policy and administration to be devolved, although unionist parties argued that elements of welfare formed an important part of pooling and sharing resources within the UK, a principle a majority of Scots voted to maintain.

Paul Cairney, Professor of Politics and Public Policy at The University of Stirling and a member of the Centre on Constitutional Change, said the outcome was "disappointing but not surprising".

He added: "One part of the explanation for such secrecy is that the division of reserved and devolved powers is driven by politics and negotiation, not a relatively sensible overall plan to ensure an effective multi-level policy making system, which can be defended in public more easily."

The ICO hinted strongly that it would expect a similar request after the Scotland Bill becomes law to be successful. The DWP declined to comment.