Mr Milne wants the Chancellor to prioritise spending on housing and infrastructure in the Autumn Statement next month to provide the boost to the economy he believes is essential to get the housing market out of the doldrums.
Complaining there is no sign of any improvement in market conditions four years after the market turned down, Mr Milne put the onus on the UK Government to kickstart activity.
"What there has been a major focus on to date is all the austerity measures and taking costs out of the whole system, but what there hasn't been alongside that is any clear strategy of how we're going to grow the economy, and that's what everyone is desperate to see," he told The Herald.
Mr Milne added: "We would certainly like to think that infrastructure and housing are two of the major areas that could potentially start to drive the economy forward and address some of the unemployment issues at the same time."
While Mr Milne's eponymous building group announced a 21% increase in core profits for the latest year, the Aberdeen businessman said the company is still battling very difficult conditions in the housebuilding market.
Eighteen months after complaining that one in five potential buyers had been withdrawing from potential deals because they could not get a mortgage, Mr Milne said conditions have not improved. Around 20% of the deals the group agrees in principle founder because people who have passed its checks are being denied funding.
Mr Milne welcomed moves by governments on both sides of the Border to help people get 95% mortages but said they are having little effect.
While putting responsibility on the UK Government to get the market moving, Mr Milne accused some lenders of making matters worse by applying inconsistent criteria, without naming any.
Asked whether the group should reduce its asking prices to help potential buyers, managing director Glenn Allison said: "It's not a pricing issue.
"If there was a lower price it would still not mean that the person who wants to buy could get a mortgage."
Mr Milne did not express an opinion when asked if he had any concerns about the possibility of Scotland voting for independence in 2014, saying he wanted to focus on Stewart Milne Group matters.
He said the group has continued to make progress amid challenging conditions.
Directors expect the group will increase turnover and profits in the current year following the decision to focus on areas like housebuilding and making timber frame kits for homes.
The group sold its construction arm to Kier Construction after directors concluded it was not big enough or sufficiently specialised to grow in what has become a difficult market.
It recorded a £6.9 million charge related to exiting the construction business during the year to June, when pre-tax profits increased to £0.5m from £0.4m in the preceding year. However, Mr Milne highlighted that the remaining operations, including housebuilding and timber frame, grew operating profits by 21% annually, to £19m, from £15.7m the preceding year.
Stewart Milne sold 684 homes compared with 509 the year before, helped by the introduction of an entry level mews property. The group has taken advantage of the downturn in the market to strengthen its land bank. It has 3500 plots consented plus 9000 under conditional contract, enough to meet its needs for the next five years.
Around 55% of the plots are in Aberdeen and the north-east, with 35% in central Scotland and 15% in north-west England.
Mr Milne received total emoluments of £865,000 in the year to June, compared with £1.3m in the preceding year.
Turnover increased to £268m in the year to June, from £229m in the preceding year.