Music mogul Simon hinted that the long-running ITV talent show needs to pull in huge ratings, otherwise it could be in jeopardy.
He told the Daily Star on Sunday: "It does feel like make or break this year but I feel like every year is make or break, to be honest.
"There were certain years when certain people won and it was like 'Oh Christ'."
But Simon said last year's show - which was won by Ben Haenow - made him realise that there's still hope for the talent show yet.
"Last year's show made me realise there's still a lot of talent to be discovered," he added. "You never forget those days when you feel a star is being born."
Last week, news about X Factor's major overhaul was confirmed when it was announced that Nick Grimshaw and Rita Ora would be joining the judging panel.
Replacing Louis Walsh and Mel B, Radio 1 DJ Nick and former Voice UK coach Rita will sit alongside Simon and Cheryl Fernandez-Versini.
That's not the only change, as following Dermot O'Leary's exit as host after eight years, former Xtra Factor duo Olly and Caroline have been brought in to replace him.
X Factor's most shocking moments: