Investigators homed in on the four after a man told police he had seen the actor at the New York apartment building where they were arrested on Tuesday.
He said he believed that is where Hoffman got the heroin.
In searches of two flats in the building, police found hundreds of packets of heroin in one of them, according to a criminal complaint.
But prosecutors declined to pursue charges against one of the four, saying there was no evidence he had control of the drugs or the flat in which they were found.
And two of the others were charged only with possessing cocaine, not heroin. Only one, jazz musician Robert Vineberg, was facing a charge of heroin possession with intent to sell.
Lawyers for those charged denied their clients had any role in the death of Hoffman.