The music mogul will join Cheryl Fernandez-Versini, Nick Grimshaw and Rita Ora on the judging panel when the ITV show returns this autumn.
In a statement today, he joked it had "taken a while" but he was happy to confirm Rita and Nick were the new judges, adding: "The combination of Nick, Cheryl and Rita is going to be a lot of fun!
"They all bring a lot of enthusiasm and passion for music to the show. I have a feeling I am in for a bit of a rough time but sometimes you have to embrace the craziness and now, with luck, we hope to find a star."
Simon will be hoping for another win after leading van driver Ben Haenow to success on the 11th series of X Factor last year.
X Factor's most shocking moments:
For exclusive competitions and amazing prizes visit Reveal's Facebook page NOW!