Fleur, who was runner-up on the ITV talent show last year, responded to the constant discussion about X Factor's ratings and its TV battle with BBC One's Strictly Come Dancing.
It was even previously reported that the show might take a hiatus, but Fleur said Saturday night's without The X Factor just wouldn't be the same.
Fleur told the Daily Star: "They can't rest it. I can't imagine a year without The X Factor. What would fill that gap? The same people who were craving changes last year are probably the same people complaining now."
In September, X Factor's executive producer Richard Holloway admitted the show could go off air for a while to be brought back stronger.
According to Billboard, he said: "[X Factor] will have a finite life span. But I think you then rest it, then you bring it back. As long as we produce the show to a standard, which the public really, really wants to see... then there is no reason why we can't produce it for years and years and years.
"I think it could run for many, many years. It's our most profitable show."
Even Little Mix, who won X Factor in 2011, seemed to agree. Jade Thirlwall told our colleagues over at Digital Spy: "I think The X Factor is a brilliant show, but it has [been on] every year and maybe people will appreciate it more."
Meanwhile, Jesy Nelson added: "It's getting a bit predictable. You kind of know what's going to happen."
Try Reveal's digital edition on iPhone & iPad. Subscribe now & never miss an issue!
X Factor's most shocking moments: