The BBC Radio 1 Breakfast Show host, 31, has left the ITV talent show after just one year on the judging panel, suggesting he'd only ever intended to stay for one series.
A rep for the star said: "After careful consideration, Nick's decided to rule himself out as a judge on this year's X Factor.
"He loved working on last year's series alongside Rita, Cheryl and Simon but it was very much a once-in-a-lifetime experience. He remains a massive fan of the show and is looking forward to watching it this year."
Nick had joined the show last year alongside close friend, singer Rita Ora, who made the switch from coaching on The Voice UK to judging on X Factor.
Nick was a surprise choice for many when he was announced as a judge last year, but show boss Simon defended his appointment, saying: "Nick is really funny and that's why I wanted to hire him for the show. Him being on X Factor will show a different side of his personality and that'll help his radio show and Radio 1."
Simon also said: "He's like a really friendly dog, really friendly, he always wants to be stroked and be happy."
Of course, Nick's departure means the speculation can start once again about who may be joining the panel this year in his place. Louis Walsh, anyone?!
A spokesperson for the show said: "Nick was a great mentor on last year's series. He did a fantastic job with the boys' category and we'd like to thank him for all his hard work. We are sad to see him go but wish him all the best.
"Details for the new series of The X Factor are still being discussed and will be announced in due course."
The most recent series of X Factor was won by Louisa Johnson.