Simon described Cheryl as a "really, really good judge" during a radio interview last night (23 December) and said that he "hoped" they would work together again because they had "fun".
Speaking to London radio station LBC 97.3FM, Simon said: "She was a really, really good judge. I mean, she took what she did seriously.
"She used to phone me almost every night - one, two, three o'clock in the morning, talking about songs and what she should be doing.
"That's the kind of judge you want to be working with on these shows, because they really care. I hope we're going to get the chance to work together. We have to find a way because we had a lot of fun."
Cheryl made her debut as a judge on The X Factor in 2008 when she was brought in to replace Sharon Osbourne on the ITV1 show.
The Girls Aloud star was an instant hit with viewers and she went on to secure victory when her act, Alexandra Burke, won the series.
Again, Cheryl won the 2009 series with Joe McElderry but lost out to fellow judge Dannii Minogue in 2010 when the Aussie's act, Matt Cardle, was crowned the winner.
In May 2011, it was announced that Cheryl would be joining Simon in America on The X Factor USA, however they famously fell out when he fired her just weeks after filming began.
Cheryl took legal action against The X Factor USA and, last month, the case was settled out of court for an undisclosed sum.
Simon and Cheryl are thought to be back on good terms now, but only time will tell as to whether she returns to our version of the show.
Do you want Cheryl back on The X Factor? Tell us below, as the best comment from across the website each week wins £50!