Despite reports of viewers' discontent with the current series and ongoing competition for ratings against Strictly Come Dancing, the music mogul has insisted the show is doing well and could be around for years more.
In an interview with The Sun, he said: "I wouldn't worry about the future of X Factor. We're going to be fine. If we're sitting here in ten years' time we'll have a very large celebration. Ten years on, to be getting ten million viewers in this day and age – it's a miracle."
Simon admitted he'd been slightly knocked back after the first week of the show, which co-stars Cheryl Fernandez-Versini, Rita Ora and Nick Grimshaw, saying he had wondered whether it was going to work… but over the weeks, the ratings began to rise and he felt more comfortable.
He added: "We had to hold our nerve and then over the weeks it has started to build and you see he catch-up numbers and they are real… Some people out there have almost an obsession with it failing. But it is only a TV talent show. It is only there to entertain people and it is a fun show to watch and it creates stars."
In another chat with Daily Star, he said: "Ratings are meaningless, completely. People are definitely changing their viewing habits. You can't discount the fact that three million more people are watching the show after it has aired and that adds up to 10 million. These are really good numbers. In America, we would be heroes with these figures."
Simon shook things up for the latest series of X Factor, ditching Louis Walsh and Mel B from the judging table and bringing in Rita and BBC Radio 1 Breakfast Show host Nick.
Dermot O'Leary left as presenter and was replaced by Caroline Flack and Olly Murs.
X Factor's most shocking moments: