Despite going on to become the world's biggest boyband, music mogul Simon still wishes the 1D boys had taken the X Factor title back in 2010.
"I still can't watch the moment when they came third," he revealed.
Speaking ahead of the X Factor final this weekend, Simon said: "I was so arrogant about the fact that [One Direction] were going to win. I was waiting for someone else's name to be called out.
"Even though they've done well. I still would have liked their history to have said they won The X Factor."
But, even though they missed out on the top spot in the competition, the five-piece have still gone on to bag themselves three number one singles, four chart-topping albums and a whole host of awards to fill their shelves.
This weekend the boys will head back to where it all began as One Direction will be performing on The X Factor final, along with Sam Smith, Demi Lovato, Ed Sheeran and Take That.
And, it sounds like 1D are going all out for their return to the show.
"I was talking to the One Direction boys in rehearsals yesterday, and they are planning something special," Simon told Reveal.
He added: "It's crazy, they've just come off this stadium tour this year, but their focus is absolutely on what they're going to do on Saturday night because they still feel like they are in a competition!"
But, while viewers are treated to a star-studded lineup, all eyes will be on this year's three finalists Fleur East, Ben Haenow and Andrea Faustini.
All this week, the trio have been drumming up votes ahead of their big night with Ben returning to Croydon, Fleur heading back to Walthamstow and Andrea taking to the stage at the Royal Albert Hall.
While Fleur said she wants to take the title for the ladies, Ben and Andrea claimed they still couldn't believe they had made it to the final three.
"In my mind I'm still in the queue in London at 5am, freezing," said Andrea. "The possibility of being here, the chance of winning, doing what I want to do in my life after - it's crazy."
Ben added: "I didn't believe I was going to get this far. The closer you get to the final, the more you definitely want [to win]."
The X Factor final begins this Saturday 13 December at 8.30pm on ITV.
Reveal is now on Google Plus! Join us here for the latest pictures, fashion and celebrity news!