The music mogul and first time father-to-be is already speculated to be returning to the UK show himself in 2014, since Sharon Osbourne, Louis Walsh and Gary Barlow have all confirmed that they will be leaving.
And now rumours have emerged that Simon is eyeing up former judge Cheryl and chart-topping 22-year-old Rita as potential co-stars.
A source told The Sun: "Simon is planning a huge shake-up next year, as it's highly likely he'll be back. Rita and Cheryl would be his dream team."
Does that mean current judge Nicole Scherzinger will be given the boot?
Earlier this year Louis declared that he would be quitting his longstanding position on the panel, claiming that this would be his final year on The X Factor.
However, he has since hinted that he'd be tempted to stay on if Simon Cowell returned. Hmm.
Meanwhile, Cheryl has been a favourite to come back to the ITV singing competition for some time.
Simon has apparently been trying to woo the Geordie brunette back on to our screens ever since she was sacked from the US X Factor, but she's been reluctant to take up his offer.
Could she be persuaded next year? We have to admit, we'd love to see Chezza on the panel again!
Although Simon, Rita and Cheryl have not addressed these recent reports, the straight-talking X Factor boss did appear to be mulling over the subject of judging panels on Twitter this week.
Last night (19 November), Simon asked his 8 million Twitter followers: "I am interested across Idol, Britains Got Talent, X factor UK, X factor USA and AGT, what have been you favourite panels over the years?"
And low and behold, Cheryl's name popped up a fair amount of times!
Who would be on your dream X Factor judging panel? Let us know below, as the best comment across the website each week wins £50!