The presenter stepped down from fronting the main show in May, after eight years in the job.
But speaking to ITV's Good Morning Britain, he said he's got nothing but great memories from his time on the talent show.
"I did eight years and it was a wonderful and fantastic eight years and I won't have a bad word said about it but it felt like the right time this year."
Dermot hasn't revealed what he will be doing next but diddWhich is a shame. Who else thinks he'd be amazing on the dance floor?!
His exit from X Factor means we'll see our first ever presenting duo on the main show this year: Caroline Flack and Olly Murs.
Olly recently joked he'd be taking over hugging duties from Dermot, who was always so good at chatting with the contestants.
"When I was on that 'X' my dream was to be a star and a singer and for me to shatter someone's dream by saying they're not good enough, I couldn't do it. So for me it's better to be next to Caroline and backstage giving them the confidence," he told This Morning.
"I'm a hugger and a kisser and to give them that encouragement to hopefully go out and smash it. I'm just going to be there for the contestants and battle with the judges as much as I can because if I believe that this contestant did a great performance on Saturday night, I'm going to tell the judges if they are wrong!"
X Factor's most shocking moments: