Research among almost 1000 employees and 800 managers found that a lack of support from employers was to blame for limiting time off among men after the birth of their child.
The Institute of Leadership & Management (ILM) said its survey also found that fewer than one in 10 new fathers take more than two weeks of paternity leave, down to 2% for managers.
The ILM said plans for shared parental leave from April 2015 will have little impact if "ingrained" attitudes were not tackled.
The report also warned that low levels of paternity pay affected take up of leave, with just 9% of fathers receiving anything longer than two weeks at full pay.
Chief executive Charles Elvin said: "The introduction of shared parental leave is a crucial step towards enabling more women to progress into senior roles, yet our research revealed cultural barriers are impeding the uptake of both two weeks statutory paternity leave and additional paternity leave.
TUC general secretary Frances O'Grady said: "Britain is the worst country in Europe for decently-paid parental leave. Many dads simply can't afford to take time off, particularly as employers rarely top up their statutory pay."
A spokesman from the Department for Business, Innovation and Skills said: "We want the introduction of shared parental leave to drive a real cultural shift and help working dads play a greater role in their child's early months."