The Essex singer has been linked to a role on the ITV show's judging panel after current judges Gary Barlow, Sharon Osbourne and Louis Walsh all indicated they'd be leaving at the end of the tenth series.
Olly would certainly be qualified for the role having shot to fame as a contestant on X Factor before going on to find international success as a recording artist. He co-hosted spinoff show Xtra Factor until last year, when he left to focus on his music.
Speaking on ITV's This Morning today, he said: "I love TV and miss that as well but I want music to be my focus this year and it has been. Everyone keeps talking about me being a judge on the X Factor and I would love to do it but at the same time, music is where I want to be focused on and there's so much more to achieve.
"I want to win Brits, I want to have a massive next album, and have more number ones, more tours..."
He added, "But I would love to do it. We all know Simon [Cowell] and he's a great guy, and if he put an offer on the table I'd definitely look at it but I think it's just music really - I love TV, it's amazing - but to try and juggle the two... I did it for two years ago when I first did Xtra Factor and it was so tough."
Olly performed at the weekend's Jingle Bell Ball in London, recently released new song 'Hand On Heart' and is currently working on his fourth studio album, due for release next year.
He can be seen performing alongside Robbie Williams when The Royal Variety Performance 2013 airs on ITV this evening.
Watch Olly's 'Hand On Heart' video below: