Scotland Office minister David Mundell acknowledged to Holyrood's welfare reform committee that food back use had risen since his Government's welfare reforms began.
But he insisted better reporting and the Government's decision to advertise food banks in job centres had contributed to the rise.
Only one-third of people visiting Trussell Trust food banks used them more than once, he said.
Less than one-tenth of benefit recipients have been paid outside the UK Government's preferred timescale, he added.
Mr Mundell, the only Conservative MP in Scotland, said the Tory-led Government had responded positively to Holyrood's recommendations, withholding direct payments from drug and alcohol abusers, allowing people without broadband to submit claims by hand and helping people in large rural areas stay in their homes.
But Labour and SNP MSPs pointed out the "horrendous" benefits cuts facing some people.
Mr Mundell said: "Some of the increase may be down to more reporting of that use.
"Some is obviously down to the greater availability and visibility of food banks."
Mr Mundell said he was not getting involved in "political point scoring" after SNP MSP Annabelle Ewing asked him about figures showing that 22,000 children had relied on food banks over the past year.