More surprisingly, sex in the summer months has a similar effect, scientists found. Conversely, abstaining from sexual activity for more than six days improved the "morphology" of sperm.
There was also reassuring news for prospective fathers who might be considering drastic lifestyle changes.
Common lifestyle factors including smoking cigarettes and drinking alcohol had little effect on sperm quality.
When less than 4% of a man's sperm has a normal size and shape, statistics show he will find it harder to father a child and may have to attend an IVF clinic.
Lead scientist Dr Allan Pacey, senior lecturer in andrology at the University of Sheffield, said: "The take-home message is, if you're a habitual cannabis user, stop, and you need to stop for at least three months.
"I can't tell you definitively that your sperm will improve, but that's a reasonable assumption.
"The other side of the story is if you're trying to have a baby, pay attention to the risks, but don't become a monk and make yourself miserable. Recognising that will contribute to happy lives and relationships."
The study, the world's largest investigating the effects of lifestyle on sperm morphology, recruited 2,249 men from 14 fertility clinics around the UK.
Participants were asked to fill out detailed questionnaires about their medical history and personal habits.
Scientists compared information from 318 men with less than 4% normal sperm and 1,652 whose sperm was higher quality.
Men in the first group were nearly twice as likely to have used cannabis in the three months before giving a sample, if they were aged under 30.
They were also nearly twice as likely to have produced a sample in the summer months from June to August.
The scientists believe younger men were most affected by cannabis simply because they were more likely to take higher doses of the drug.
Chemicals in the drug itself, not the tobacco used in "joints", were probably responsible since cigarette smoking had little effect on sperm morphology.
Dr Pacey said the summer influence was unexplained but did not appear to be linked to heat.
"We didn't expect this," he said. "It's complete conjecture on my part, but it could be a sunlight effect - day length affects our hormones and vitamin D in all sorts of ways."
Previous research has shown that only sperm with the right size and shape can easily get past a woman's cervix, which is surrounded by a "quality control" barrier of thick mucus.
Sperm with a poor morphology also swim less well because of their abnormal shape.
"You can compare a sperm cell with an aircraft," said Dr Pacey. "If you've got a sleek aerodynamically shaped aircraft it's going to fly better than one that looks like an old biplane with bits hanging off."
The research is published in the latest edition of the medical journal Human Reproduction.
Co-author Dr Andrew Povey, from the University of Manchester's Institute of Population Health, said: "This research builds on our study of two years ago which looked at the risk factors associated with the number of swimming sperm (motile concentration) in men's ejaculates.
"This previous study also found that there were relatively few risk factors that men could change in order to improve their fertility. We therefore have to conclude again that there is little evidence that delaying fertility treatment to make adjustments to a man's lifestyle will improve their chances of a conception."
It was still possible that unmeasured factors, such as the quality of sperm DNA, might have influenced the results, said the researchers.
Exposure to paint strippers and lead has also been linked to poor sperm morphology.