The decision pins some of the blame for Europe's worst massacre since the Second World War on the Dutch state.
A district court in The Hague said Dutch peacekeepers in Srebrenica, a Bosnian Muslim enclave in Bosnian Serb-held territory, could have known the 300 men who had sought refuge in their base in the village of Potocari would be murdered if deported from the Dutch compound.
The court said the Netherlands was not liable for the deaths of those who had fled into the forests surrounding Srebrenica, where many of the men and boys were later buried in mass graves.
The ruling could set a precedent with implications for future peacekeeping deployments by the Netherlands or other countries.
During the Bosnian war, the Dutch battalion Dutchbat had been deployed to protect Srebrenica, which had been designated a safe haven by the UN but surrendered to the much larger Bosnian Serb army commanded by Ratko Mladic who is on trial for war crimes at an international court in The Hague.
The case was brought by the Mothers of Srebrenica, a group representing surviving relatives of the victims. They had failed in their bid to have a court find the UN responsible for the massacre.
Judge Peter Blok said: "At the moment that the men were sent away, Dutchbat knew or should have known the genocide was taking place and therefore there was a serious risk those men would be killed."
In an emotionally charged hearing at a civil court in The Hague, Presiding Judge Larissa Alwin added: "By co-operating in the deportation of these men, Dutchbat acted unlawfully."
Relatives of the dead welcomed the limited finding of liability but lamented it did not go further.
Munira Subasic, president of the Mothers of Srebrenica group that filed the case, said: "Obviously the court has no sense of justice. How is it possible to divide victims and tell one mother that the Dutch state is responsible for the death of her son on one side of the wire and not for the son on the other side?"
Ms Subasic said her organisation would "keep fighting for truth and justice. And in the end we will win".
The court did not say how much compensation the families should receive.
Earlier in the long-running case, judges said relatives of the victims could not sue the UN in Dutch courts because its immunity from prosecution is a cornerstone of peacekeeping operations around the world.
A lawyer for the relatives, Marco Gerritsen, said he would carefully study the 89-page ruling before deciding whether to appeal.
The failure of Dutch soldiers to protect the Muslim men and boys of Srebrenica has left a deep scar in Dutch politics, contributing to the resignation of the Dutch government in 2002. Bosnian Serb forces sorted the Muslims by gender, then trucked the males away and began executing them. Their bodies were piled into hastily-made mass graves in what international courts have ruled was genocide.
The three-year Bosnian war, in which at least 100,000 people were killed, was the bloodiest of a series of conflicts that accompanied the break-up of Yugoslavia in the 1990s.