The 'Troublemaker' singer has since been rumoured to be taking over Dermot's position, with reports from the weekend saying Olly will present the show with former Xtra Factor host Caroline Flack.
Speaking to reporters at the launch of his waxwork at Madame Tussauds Blackpool, Olly said: "It's rumours. Everyone is speculating about it. I have presented the Xtra Factor before with Caroline and I loved it."
Last week, Dermot shocked fans when he announced his decision to quit as presenter after eight years.
The 41-year-old said it's "time to move on" while thanking Simon Cowell and the team during his time on the show.
Olly said he will miss Dermot on the show. He added last night: "I have known him since 2009 when I did the show as a contestant. It is obviously a massive shame, he has done an incredible eight years but I think it is now time for a new chapter."
Dermot did give his replacement advice on taking on the role, adding: "You're about to be the conductor on the most brilliant, runaway train in showbiz. Good luck you'll love it."
For exclusive competitions and amazing prizes visit Reveal's Facebook page NOW!
X Factor's most shocking moments: