And in a new interview with Reveal magazine, in shops now, Olly, 32, admitted that he didn't miss hosting the talent show but would like to return to TV.
He also joked that perhaps next time he wouldn't follow in the footsteps of Dermot O'Leary, who hosted The X Factor before Olly and Caroline and then returned once they'd left!
"I loved it for that year and I really enjoy presenting, but I'm a singer, and that's the most important thing to me. I'd like to get back on TV at some point, but probably not on a show that's been hosted by someone as good as Dermot!"
Olly told us he's been so busy he hasn't had the chance to watch The X Factor's new series in full, but he's seen a few bits... including Honey G!
"It sounds wrong when I say I haven't [been watching the show]. It's not because I don't want to - it's because I've actually got my life back! I get a chance to live on the weekends now. I've seen bits and bobs, though, like Honey G."
Ah, Honey G! The most talked about contestant this series!
So, what does Olly make of the 35-year-old rapper from North Wheezy who wants to be globally famous on a par with Jay Z, Missy Elliott or Pharrell?
"I don't know what to make of her," he admitted. "It's funny, but at the same time it's like, 'Is this real? Is she a serious artist?' I don't know."
Olly's new album, 24 HRS, is out Nov 11
Pick up Reveal magazine now for more from Olly including why he hasn't seen Caroline Flack too much lately, and whether he's found anyone special now he's dating again.