The BBC Radio 1 Breakfast Show host announced on Friday (19 February) that he'd quit the show after just one year on the judging panel, explaining he'd always seen it as a once-in-a-lifetime type thing.
Nick, 31, elaborated on his decision on today's radio programme, saying: "It was one of those things. When I finished at Christmas and had time off and had a think about it and sat down, I just thought, it's one of those things I loved – it isn't a diss to the show or me storming out of X Factor like 'Screw you Cheryl!' It's a great show and I loved it but it's a once-in-a-lifetime style thing.
"And you can't have a curry when you're on it on a Saturday night because you're there. So what's really good about X Factor is watching it at your house with friends having curry. When you're there, there's no curry. And Simon promised me curry!"
Nick added: "I loved Simon and Cheryl and Rita and everyone who worked on it, and all the boys, the mentoring stuff, I loved. I genuinely loved it. There is no bad blood. I just want to watch it on the telly."
A rep for X Factor has said: "Nick was a great mentor on last year's series. He did a fantastic job with the boys' category and we'd like to thank him for all his hard work. We are sad to see him go but wish him all the best."
There is no word on who may replace him but names thrown in the ring include former judge Louis Walsh and One Direction's Louis Tomlinson, who served as guest judge during Simon Cowell's judges' houses last year.
Meanwhile, there are also unconfirmed rumours that Caroline Flack could be bowing out as co-host. She and Olly Murs became the show's first-ever presenting duo last year when they replaced long-time host Dermot O'Leary, but speculation is mounting that show boss Simon didn't think the two-person team worked.
A rep has not confirmed this, instead saying no decisions have been made.
For exclusive competitions and amazing prizes visit Reveal's Facebook page NOW!