Harriet Harman said that all-male discussions undermined "the quality of decision making" within organisations.
Her comments come as 50 UK firms are expected to be named and shamed tomorrow for having no female board member.
Lord Davies, the former Labour trade minister, is expected to set out the list as he launches his annual review into the issue with the Business Secretary Vince Cable and the Culture Secretary Maria Miller.
Ms Harman said meetings where there were no women were too common.
Speaking at a fringe event at the Scottish Labour conference in Perth at the weekend, Ms Harman said: "We need to have a culture where if men are in a 'men only' meeting they feel uncomfortable about it.
"And they say 'OK guys there is obviously something wrong here' … and actually feel that it is not right."
She added: "There are too many meetings still where there is just one woman there.
"But then there are obviously lots of other meetings where there is no one female there at all.".
She continued: "The problem is it undermines the quality of decision making if you are narrow and exclusive.
"And obviously, it is not based on merit - otherwise there would be women there, having a voice."
The issue was one that affected many different types of organisations including company boards, political parties and non-governmental bodies, she said.
Speaking at the same event Margaret Curran, Labour's shadow Scottish Secretary, said that she had been shocked to discover the extent of high-profile organisations that still had men-only boards.
She called for all-male boardrooms to be rejected as an "affront".
Lord Davies, the former Labour trade minister, has been tasked with increasing the representation of women on the boards of British companies.
At the weekend he said that firms had no excuse for failing to recruit women.
"If you are a chief executive officer and you don't have gender diversity or diversity in general as a top issue, then you've been asleep at the wheel for the last few years," he warned.
The UK Government has a target to ensure that 25% of non-executive directors at UK public companies are female by 2015.
As he unveils his report, Lord Davies is expected to predict that that figure will be met.
However, he will also warn that there are still too few female executive directors, saying that that is now the "bigger issue" within industry.
Women now fill around one in five non-executive positions in boardrooms, up from 12.5pc in February 2011.
Vince Cable has suggested that he could support all women shortlists for board positions.
The Liberal Democrat politician said that he had "no problem" with the idea, althougvh experts have suggested that such a move could be illegal under EU law.
Mr Cable has also called on companies to tackle their "deeper cultural elements" which he said were holding back the advancement of women in the workplace.