Ruaridh Jackson narrowly missed the chance to win the game when his late penalty bounced back off the posts and leave the Warriors winless from four games in Pool Four.
That record fails to tell the complete tale of their campaign so far, however, following tight matches already against Northampton and now Castres both home and away.
The Warriors had led 8-3 at half-time today but Townsend was left lamenting his side's inability to hold on to the advantages they have gained in Europe this term.
"We have been very competitive in every game and were 15-0 up at Northampton and we were in the game against Ulster," he said.
"We have had two close games against Castres and we need to put those nearly wins into victories.
"We expected a tough match. Castres are fourth in the French league and have an excellent home record and I think that is seven wins in a row they have put together. We knew how hard it would be but we believed in the team we had and we put them under pressure."
Warriors rang the changes for the visit to the south of France and Townsend believes there were plenty of positives to take away.
"For large parts of the game we took the game to Castres," he said.
"In the second half we struggled to get the same amount of possession from line outs. I was really proud of the players for their effort and ambition and we moved the ball really well. We looked dangerous."
Warriors paid the price for ill-discipline as Tom Ryder, who this week signed a new contract to keep him at the club until 2015, and Byron McGuigan were both sent to the sin bin before Yannick Caballero scored the match-winning try.
Townsend, who enjoyed a spell at Castres between 2000 and 2002, added: "Our scrum came under pressure, especially when we went down to 14 men. At the end of the day what killed us was two yellow cards.
"Even when we went down to 13 men I thought we put a huge effort in to protect the try line and to get that chance to win the game at the end was great. It hit the post and went out and that was the difference between winning and losing."
Townsend added: "It is up to individual responsibility and discipline. I thought on a couple of occasions in the first half we had the choice to get back and defend or go for the ball that was verging on offside.
"I believed we would improve in the second half but when you are playing away in France and the crowd really get behind their team then you have to have emotional control and we didn't in the second half."
With three big matches against Edinburgh - home and away - and Treviso to come in the next two weeks Townsend's men will look for a quick recovery as they seek a play-off spot in the RaboDirect Pro12.