Music mogul Simon, who is a judge on the ITV talent show, dismissed the BBC One baking show as a competitor, saying he finds it a tad boring.
Simon was reportedly made his comments to Piers Morgan at the Mipcom TV programming conference in Cannes, France, on Monday (13 October).
The Guardian quotes Simon saying: "My belief is, it's more exciting to see someone like Leona Lewis win and sing a fantastic song, than look at a chocolate eclair at the end of the show."
Simon's comments came as voting for the National Television Awards opened up today, revealing that he will go head-to-head with Bake Off's Mary Berry and Paul Hollywood for the country's top TV judge gong.
The new award category will see the TV personalities pitted against each other. Other names in the running include Simon's fellow X Factor judge Cheryl Fernandez-Versini, Strictly Come Dancing's Bruno Tonioli and The Apprentice's Lord Alan Sugar.
Since the new series came back on our screens, X Factor's main competition has been arch-rival Strictly Come Dancing.
The two talent shows came head-to-head again last weekend with Strictly ratings recorded at 8.65 million viewers, compared to X Factor's 7.66 million.
Meanwhile, last week's Great British Bake Off final saw a staggering 12 million viewers tune in.
Reveal is now on Google Plus! Join us here for the latest pictures, fashion and celebrity news!