GPs worry they are being used to help "ration" food bank services by assessing their patients' eligibility for free groceries.
Dr John Ip, a GP in Paisley and secretary of the British Medical Association (BMA) Glasgow Local Medical Committee, said doctors were concerned they were being put in an "impossible situation".
The issue is expected to be raised at the BMA's annual Scottish Local Medical Committee conference in Glasgow this month.
Ip said: "GPs have been expressing concerns that patients who are wanting to access food bank services have been told they are full up and to access services they will need a referral from their GP.
"Food banks and charities like that have historically operated as a charitable service providing support for the poor and vulnerable people. It is a sign of the times that a lot more people are accessing them, they are under pressure and they are needing to try and ration their services.
"But using general practice as a means to ration I don't think is appropriate."
Earlier this month a snapshot poll carried out by GP magazine Pulse found around one in six GPs across the UK had been asked to refer patients to a food bank in the past year.
Ip said: "The information I am getting is that some food banks are telling patients that they need to see the GP to get a referral.
"Frankly, food isn't something that we are directly involved with, we are involved with all other aspects of people's health and mental well-being.
"It is an impossible situation - why would we turn people down?"
He acknowledged doctors had a role in referring patients to other services if appropriate, such as hospitals or mental health services.
But he said: "Food bank services, in my view and certainly the view of the vast majority of my colleagues, should be self-presenting. If people have problems with their finances and they are having issues in managing their budgets, then they are the people who know best about their situation.
"I think putting GPs in that loop adds to our workload and frankly we would refer everyone, as everyone should have access to food."
Dr Miles Mack, chairman-elect of the Royal College of General Practitioners Scotland, said the increasing demand for GPs to refer patients to food bank services "impacts on what little time they have with those who are most in need".
He added: "We are deeply concerned about the rising number of people who are dependent on the services of food banks.
"Clearly such social problems have a huge and detrimental effect on the health and wellbeing of patients."
Last week, the Sunday Herald revealed how independent food bank Glasgow City Mission, which normally helps more than 100 people a week, had to temporarily close its doors after running low on supplies following high demand.
Fundraising manager Graham Steven said there had been a "remarkable" response to the crisis, with donations of food and monetary support.
He said the food bank operated a policy of helping anyone who turned up at the door for the first time, who would subsequently be assessed and advised on how frequently they should return, depending on their situation.
"They don't need to be referred by anyone," he said. "We know of a lot of people that just fall through the gaps, so that is where we took our cue from."
Ewan Gurr, Scotland development officer for The Trussell Trust, said the organisation did not have a policy of asking people trying to access the services to provide a referral from their doctor.
He said: "We would never turn anyone away from our food banks. If someone turned up and did not have a referral, we would speak to them and find out what they needed."