The former Conservative prime minister urged David Cameron to open talks with other European countries.
But Downing Street rejected that suggestion, saying it was concentrating on analysis of the UK's current relationship with the EU.
Mr Cameron pledged an in-out referendum last month if the Tories win the next General Election.
But he said that the vote would be after the UK negotiated a new relationship with the EU.
Sir John did back Mr Cameron's referendum pledge, although he described it as a gamble.