Doctors should now look at whether the over-the-counter drug should be the first choice treatment for people with the hugely debilitating condition, said scientists behind the research.
The study of more than 1,500 patients, published in The Lancet, found patients given the common analgesic recovered no more quickly than those given a useless substitute.
Dr Christopher Williams, of the George Institute for Global Health at Sydney University, Australia, who led the research, said: "Simple analgesics such as paracetamol might not be of primary importance in the management of acute lower back pain.
"The results suggest we need to reconsider the universal recommendation to provide paracetamol as a first-line treatment for low-back pain, although understanding why paracetamol works for other pain states but not low-back pain would help direct future treatments."
The study involved 1,652 people with acute low-back pain at 235 care centres in Sydney.
They randomly received one of three treatments, up to four weeks of paracetamol in regular doses, paracetamol as they needed it or a placebo.
They all also received "advice and reassurance" and follow-up assistance for three months.
All three groups returned similar figures for recovery time, the study found, with the placebo group having a median recovery time of 16 days, a day faster than the other two groups.
Dr Williams also suggested that given his team's results, it was worth looking at whether advice and reassurance for sufferers was a more effective form of treatment than drugs.
The research, funded by the National Health and Medical Research Council of Australia and GlaxoSmithKline, was welcomed by other scientists who called for more research to be carried out.
Writing in The Lancet, Bart Koes and Wendy Enthoven, from the Erasmus University Medical Centre in Rotterdam, said: "Although the findings from this high-quality trial are clear, the content of guidelines should not be changed on the basis of a single trial; more robust and consistent evidence, including verification of the results in other populations, is needed.
"Furthermore, efforts to establish if prescription of other simple analgesics has additional benefit to advice and reassurance of the favourable prognosis for acute low-back pain are very welcome."
Professor Sallie Lamb, director of the Oxford Clinical Trials Unit, said: "This study may cause us to reconsider the way we treat patients here with acute low back pain."