With one in eight people in the world already going hungry, the pressure of rising temperatures and extreme weather events on production means more people will not be able to afford enough to eat, Oxfam warned.
The number of people at risk of hunger could rise by 10%-20% by 2050 compared to a world where the climate was not changing, undoing efforts to tackle the problem, the aid agency said.
Oxfam has previously estimated that the average price of staple foods is likely to more than double in the next 20 years, compared to 2010 prices, with up to half the increase down to climate change.
In a report ahead of a major global assessment of climate change science that is due out on Friday, Oxfam said yields were already being hit by global warming.
The aid agency pointed to research which found that global yields of maize over the past three decades were 3.8% below what they would have been without climate change and wheat was 5.5% lower than it would otherwise be.
The report said food production would be negatively affected in most parts of the world by rising temperatures and shifting weather patterns, while extremes such as droughts and floods would damage crops, kill livestock and affect distribution of food.
Agricultural disasters were likely to become more frequent, it said.
In the UK, 2012 was the second wettest year on record, and the wettest ever in England, causing UK wheat yields to fall to their lowest levels in 20 years, and the country had to import 2.5 million tonnes of wheat, the report said.
Tim Gore, head of policy for Oxfam's Grow campaign, said: "Just as the evidence of man-made climate change is becoming stronger, so too is our understanding of how it hits people, especially around hunger."