Not ruling out whether Cheryl will come back for another series, Simon said he will be waiting until the new year before approaching people about judging on the show.
Both Simon and Cheryl made their return to The X Factor in August following a three-year hiatus but apparently the job is so "exhausting" that the panel can never think beyond the current series.
Simon told The Sun: "Whenever you ask anyone at this stage in the competition whether they want to come back or not, you're knackered - and the idea of doing it again is like, 'My God. I couldn't go through this again'."
He added: "I like to let people finish the show, relax over the new year and then see how they feel. So I wouldn't ask Cheryl the question this year."
In July, Cheryl married French businessman Jean-Bernard Fernandez-Versini following a whirlwind - and very quite - romance.
The Geordie star, who found fame with Girls Aloud, has never hidden the fact that she would like children and she was recently quoted in The Sun saying it would be difficult to juggle The X Factor with a baby.
"You couldn't do them both. Next year is going to have to be the time where I reflect on what I've done and think about what I want to do. I'm not ruling it out," she said.
However Simon, who welcomed his first baby - a son called Eric - with partner Lauren Silverman in February, has jokingly pointed out that he has managed to do The X Factor in the same year.
According to the newspaper, he apparently 'sniffed': "Well, I had one."
Cheryl's X Factor return hasn't been as triumphant as perhaps she would have liked, as she has just one act remaining in the competition - Lauren Platt. Simon, meanwhile, has three, while Louis Walsh has two groups.
Mel B is in the same boat though, as she has one act remaining with Andrea Faustini.
This weekend, the contestants will be taking on either a Whitney Houston or an Elton John track.
We can't wait to see which songs they pick!