Last night a former defence minister hit out at the idea, saying Westminster was not a "creche".
East Dunbartonshire MP, Mr Swinson, who gave birth to her first child, Andrew, on December 22, had criticised what she described as a bizarre ban.
"I hardly think it would be too much of a disruption," she said.
"You can take a sword through there but you can't a baby".
Ms Swinson, who is married to Nick Clegg aide and fellow LibDem MP Duncan Hames, added: "I think when you are perfectly capable of walking through the lobby holding a small baby, I think there would be a better way of just allowing that.
"But parliament moves but slowly."
But Tory MP Sir Gerald Howarth said it would be "wholly inappropriate" for MPs to bring babies into the voting lobbies.
That part of the House of Commons is "not a creche", he said.
"We are engaged in a serious business of running the country," he added.
"Of course mothers will want to look after their young children but I don't see the child's welfare is served by taking them through the lobbies."
Babies are allowed into MPs' offices but not into the Commons chamber or the voting lobbies.
Licia Ronzulli, the Italian MEP, is often photographed voting in the European Parliament with her toddler daughter sitting on her knee.