Wilfried Bony scored a late winner to earn the 2-1 victory, inflicting United's fifth home loss of the season in all competitions.
"We are a bit unlucky to lose but we probably didn't do enough to win it," said Moyes, whose team also sit seventh in the Barclays Premier League. "We made some opportunities, not enough, and when you're down to 10 men you want to see it out and take a replay but we didn't quite get that far."
The Manchester United manager had seen midfielder Fabio dismissed for a studs-up challenge on Jose Canas, which came with the score at 1-1 after Wayne Routledge and Javier Hernandez had traded goals.
"I have no issues [with the card]. The referee called it right," Moyes added. "We played quite well at times, we got near the edge of the box quite regularly but didn't make the opportunity to score a goal, or when we did get an opportunity, take it. That was the disappointing thing, as we went behind and came back. We had periods of good control but we didn't get a goal or create enough chances."
Former United manager Sir Alex Ferguson was in the stands for the match, which ended with some booing from the home fans. Moyes' assessment was backed by midfielder Fletcher, who played for the duration of the match, and said the team had let down Moyes and the club's supporters. "It's a massive blow, let's not kid ourselves," said the Scotland internationalist, having returned to the Manchester United first team on Boxing Day, his first start in the senior side since their 4-3 win at Reading on December 1, 2012. "This is a really bad result and there are a lot of angry, hurt players. We feel we have let the manager and the fans and everyone down."
Swansea's opening goal was a well-taken effort, with Routledge lifting the ball over Anders Lindegaard before it dropped into the net with 12 minutes played. However, the home side levelled three minutes later as Hernandez finished at close range from Alexander Buttner's cross.
With neither side then doing much to threaten, Buttner appeared to carry most of the home side's threat - dropping one far-post cross just over Antonio Valencia and curling a low free-kick towards goal that was gathered by Gerhard Tremmel - but the stalemate continued long into the second half.
The match was finally galvanised by Fabio's dismissal, setting the stage for Bony's powerful near-post header, with the Ivorian now having scored 13 goals in his first season with the Welsh side. "It's good, we won - we came here not to lose and it was perfect for us. That was our wish and thank God we did it. I am enjoying life. It's now important for us to take it step by step and think about the next game."
Moyes, meanwhile, is currently short of attacking options, and was unable to call upon forwards Robin van Persie or Wayne Rooney. "It'll be great when we get them back, the sooner we get them back the better but they're not ready yet," the Scot said. "If you lose games at any club, it's difficult. At Manchester United it's even bigger, but that's the job and we get on with it."