The finding may help explain links between deprivation and poor mental ability and school performance.
US scientists conducted magnetic resonance imaging (MRI) brain scans on 145 children aged six to 12.
They found poverty was associated with smaller hippocampus and amygdala brain regions. Both have roles in processing memory, and the amygdala is also linked to emotional reactions.
Amounts of grey and white matter were also lower in poor children's brains.
Grey matter contains the cell bodies of neurons while white matter consists of the axons, or wiring, connecting parts of the brain. The effect was more pronounced in children with stressful home environments whose parents lacked nurturing skills.
The research is published in the latest issue of journal JAMA Pediatrics.