The former contestant was rumoured to be in the running to take over as series host, following the departure of Dermot O'Leary last month.
Now, Olly has confirmed talks have happened – but the end result isn't clear.
He told Irish radio station Spin 103.8: "I haven't git it yet. We've spoken out it in the papers but nothing's been finalised or nothing's been signed. We've had a few discussions, that's about it really."
Olly co-hosted the Xtra Factor spinoff show alongside Caroline Flack for two years and says he loved doing that. He also added he and Caroline had never wanted or actively gone after Dermot's job as host of the main show.
"I mean me and Caroline we were never interested in taking Dermot's job, it was only, you know, Simon's idea of potentially having me and Caroline do it," he said.
And a very good idea we think it is too!
Olly previously spoke about how great Dermot was at his job, saying: "I have known him since 2009 when I did the show as a contestant. It is obviously a massive shame, he has done an incredible eight years but I think it is now time for a new chapter."
X Factor returns to our screens later this year for its 12th series, with open auditions already underway across the country.
As always, the judging lineup hasn't been confirmed and probably won't be until nearer the time, meaning we have a few more months of 'will this person get fired/replaced/quit?' rumours ahead of us.
Most recently it was claimed that long-time judge Louis Walsh wasn't safe, while Cheryl Fernandez-Versini would be welcomed back with open arms.
X Factor's most shocking moments: