The attack came as David Cameron announced all-women shortlists for the next General election.
Brown faced tough questioning from Labour backbencher Parmjit Dhanda, who pointed out that before Mr Brown became PM two years ago there were two ethnic minority members of the Cabinet.
“There are none now, yet there are four white, Scottish men. Do you think this an acceptable state of affairs?” Mr Dhanda asked.
A seemingly discomfited Mr Brown replied: “I think you are looking at people ... I would point you to people who sit at the Cabinet table. There are people who sit at the Cabinet table who do not necessarily hold a full departmental responsibility but are equally important to the running of the government.
“I would say that there are seven women sitting round the Cabinet table, you have the first Asian cabinet minister sitting at the Cabinet table, representing issues of transport.
“At the same time you have the Attorney General who is the first black Attorney General and a very successful one at that.
“So I would ask you to reflect on the wider picture.”
Mr Brown was previously accused of sexism by former housing minister Caroline Flint, who quit the government in June, saying he was running a “two-tier” Cabinet where women were treated as “window dressing”.
Although Ms Flint attended Cabinet, she was never a full member.
Tory leader David Cameron indicated that he was ready to draw up all-women shortlists of candidates for the general election.
He said there were “many very, very good women” on the Tories’ priority list of candidates who had not yet been selected and he wanted to give them a chance at the next election.
Appearing after Mr Brown in front of the Speaker’s Conference, Mr Cameron said the under-representation of women and black and ethnic minorities was a particular problem for the Conservative Party.
“It’s a real problem for Parliament and it’s been an even greater problem for my party, and one that I desperately want to address and have tried to address,” he said.
Mr Cameron told the committee it was “bad for politics” that certain groups were under-represented in the Commons, adding that he would “do frankly whatever is necessary” to tackle the problem.
“We need to make sure that the conversation we have within the Conservative Party and the conversation we have within Parliament is like the conversation that’s going on in the rest of the country,” he said.
The Tory leader said the party’s selection procedure had recently been altered so that between now and Christmas new shortlists of candidates would be drawn up between Conservative Central Office (CCO) and the relevant local association.
From January, the party’s “by-election procedure” would come into effect, with CCO providing the shortlist.
“It’s my intention, if we continue as we are, that some of those shortlists will be all-women shortlists to help us boost the number of Conservative women MPs,” he said.
He said 29% of Tory candidates were women, but added: “There are many very, very good women on our priority list of candidates who haven’t yet been selected and I want to give them the chance to serve in Parliament.”
In the event of the Tories winning a one-seat majority at the next election, he said, there would be “nearly 60” women Conservative MPs. There are currently 19.
Mr Brown said there was a danger that root-and-branch reform in the wake of the MPs’ expenses row could prevent people on lower incomes entering politics.
“If the final conclusion of all the various reviews, including the Legg and then the Kelly review, were to give the impression that the only people who could afford to become Members of Parliament were people who had very substantial incomes... in the first place, before they are considered as representatives, then that would be a very big blow to the opportunities and possibilities of those from poorer backgrounds, low-income backgrounds and backgrounds where there are huge barriers to overcome to get into Parliament,” he said.
“So we must make sure in everything that we do that we do not create new barrier to representation.”
Mr Brown signalled the extension of all-women shortlists to 2030 and that there would be a majority of black and Asian candidates in certain constituencies.
He also pledged more support for disabled people and members of the lesbian, gay, bisexual and transgender communities in getting into Parliament.
Mr Brown said he was proud of the progress that had been made since he entered Parliament in 1983, when the Commons was all white and there were only 23 women MPs.
But he went on: “We have not done enough yet to address under-representation in our society. Seen from the outside, Parliament is not yet fit for the 21st century.”
Mr Brown said he expected the number of Labour women MPs to rise to between 120 and 140 at the next General Election.
The Equalities Bill would extend all-women shortlists to 2030 and allow relevant constituencies to have a majority of black and Asian candidates.
The Prime Minister also urged the Speaker, John Bercow, to allow civil partnerships to take place in Parliament for the first time.
“I’m committed to diversity in Parliament, not just because it’s at the heart of our Labour Party values but because it is in the interests of the whole country that we keep the promise of democracy, not for some but for all the people of Britain,” he said.
“I don’t believe sexism, racism and disability discrimination are indelibly woven into the fabric of our society.”
Mr Brown added: “We’ve got to recognise that MPs - and this is one of the things that comes out of Legg and also comes out of what will be in the Kelly review - you have got to realise that people are living in two places at once, they have got family responsibilities, and there has got to be some place for showing that the financial arrangements for MPs take account of that.”