The former LibDem leader said judgments under such a system would be open to judicial review, as Lord Justice Leveson is due to present recommendations from his phone-hacking inquiry on Thursday.
Since such cases would be unlikely to attract legal aid, only the well-off would be able to afford to take a case, he added.
Sir Menzies also said he had a "genuinely open mind" about regulation and was awaiting publication of the Leveson report with interest.
He told The Herald: "The thing that seems to me to be fundamental is that any system of regulation must command public confidence.
"I am a firm believer in freedom of speech but with freedom comes responsibility. And unless the press can satisfy everyone that they are capable of self-regulation then inevitably there will be enormous pressure for statutory regulation.
"But if I could sound a note of caution: if you have a statutory regulatory system, the decisions of any council or tribunal could be subject to judicial review, which could mean issues that ought to be quickly resolved would be dragged out and involve considerable legal expense."
He warned it was unlikely such cases would attract legal aid. If they did not, "considerable resources" would be needed to take appeals, he said, adding that press organisations would likely have greater means to do so than many individuals.
Lord Justice Leveson is due to present the findings of the highly anticipated report on Thursday.
His inquiry was set up in the wake of the phone-hacking scandal, which saw the News of the World accused of hacking into the mobile phone of missing schoolgirl Milly Dowler.
Campaign group Hacked Off has called on all party leaders to implement the proposals as long as they are fair and proportionate.
Liberal Democrat leader Nick Clegg has said his party would never back "illiberal statutory regulation" but has drawn a distinction between that and an independent regulator with statutory underpinning.
The World Press Freedom Committee yesterday warned that any British media controls would send an "appealing message" to the world's oppressive regimes. Richard Koven, the committee's European representative, said "a chill will go through the world's media" if there was statutory regulation.
Chris Jefferies, the retired teacher wrongly accused of killing his tenant and neighbour Joanna Yates in Bristol at Christmas 2010, backed tougher rules. He said failure to accept the proposals would put Prime Minister David Cameron in "an almost impossible situation, politically and in the country at large".
Graham Foulkes, whose 22-year-old son David died in the 2005 London bombings, called state involvement in newspapers a "step back to Stalinism". Mr Foulkes was told last year that his personal details were held by an investigator working for The News of the World. However, he believes the Leveson Inquiry was "railroaded by the celebrity circus".
There was speculation at the weekend that Mr Cameron would fall short of backing such measures. A newspaper reported that the Prime Minister would back a new, tougher model of self-regulation to replace the Press Complaints Commission, but with the threat that a statutory system could be brought in later if matters do not improve.
Number 10 played down any suggestion that the Prime Minister had already made up his mind."The Prime Minister is open-minded about Lord Justice Leveson's report and will read it in full before he makes any decision about what to do," a spokesman said.