Mrs Justice Parker decided that the baby should be taken into care after being told how his father had assaulted one social worker and threatened to kill another.
The judge said the father appeared to have become frustrated by what he saw as an "invasive" approach by social services staff - and she said she thought that he could be "dangerous".
She also had concerns about the vulnerability of the baby's mother, who had been diagnosed with a learning difficulty.
The judge accepted that the couple loved the baby and described the case as "terribly sad", but she concluded that the combination of the father's hostility and the mother's vulnerability created a high risk of the boy suffering significant emotional harm, and a possibility of him being caught up in violence.
Mrs Justice Parker outlined her reasoning in a public ruling following a private hearing at a family court in Watford.
The father says he and his partner aim to appeal.
Mrs Justice Parker said no one in the baby's family could be identified, but the local authority with responsibility for the baby - Hertfordshire County Council - could be.
She made no criticism of the council and described a social worker involved in the case as "excellent".
Mrs Justice Parker said the couple's two-year-old son was taken into care last year after another judge raised similar fears and described the father's behaviour towards social services staff as "dangerous".
She said the baby had been taken into interim care days after his birth, pending a final court ruling on his future.
The judge said the mother also had an older third child - by another man - who had gone into foster carer as a toddler and lived with a relative.
Mrs Justice Parker described a history of problems between the father and social services staff in her ruling.
The judge said the father did not think his partner needed local authority support when looking after children, and believed he could provide all the support she needed.
He appeared to have become "increasingly frustrated and intolerant" about what he thought was an "invasive approach" by social services staff.
She said he had been hostile and refused to co-operate.
He had punched a male social worker in the face a number of times at a court hearing last year and had been convicted of threatening to kill a female social worker, she said.
Mrs Justice Parker also said she had been concerned by his demeanour when he was giving evidence at the latest hearing.
"I think I'm a fairly hardy plant," she said. "But I have to say I found his simmering anger quite difficult to cope with."
She added: "I think he can be dangerous."
Mrs Justice Parker said she thought there was a significant risk that the father's behaviour would cause emotional harm to the baby.
She said the baby could also be at risk of physical harm if the father became involved in further confrontations with social services staff when he was nearby.
Mrs Justice Parker said the baby's mother made excuses for her partner. The judge said the mother had not faced up to the father's behaviour and she doubted whether the couple could live apart.
"I am in no doubt there is a high risk of significant harm to baby," said Mrs Justice Parker. "Due to a combination of the vulnerability of the mother and the father's attitudes and behaviour."
Mrs Justice Parker said there was no evidence that the father or mother had deliberately physically harmed either of their sons.
"This is a terribly sad case because father and mother, each of them, have many excellent qualities," said the judge. "It is absolutely plain to me that both of them love each of their sons, their boys, from the bottom of their hearts."
But she said she was "quite clear" that a placement for adoption was the only answer.
She said no other family member was available to care for the baby.