Sunday's edition of the ITV1 programme had an average audience of just 7.9 million – including the viewers who followed the show on the +1 channel.
In previous years the programme has tended to build its audience from the launch, but this time the series has been shedding viewers, with the Sunday show 800,000 viewers down on the first edition.
The last time the programme pulled in fewer than eight million viewers is thought to have been December 2007. Sunday's show had been up against the closing ceremony of the Paralympics.
The X Factor is struggling to match even last year's figures, with Saturday's audition show down almost three million viewers on the figure for this stage of the competition in 2011.
The programme pulled in 8.2 million viewers , which was 2.9 million down on the previous year.
There have been criticisms about the minimal amount of time in each programme that is actually devoted to music.
In a further blow for X Factor supremo Simon Cowell, there have been reports that his programme Red Or Black? may be axed by ITV.
However, an ITV spokeswoman said: "No decision has been taken on a further series. We are concentrating on the current run."