After co-hosting just one series of the show alongside best pal Caroline Flack, Olly has revealed he has decided to leave The X Factor to focus on his music.
Speaking to The Sun, the 'Kiss Me' singer admitted he had made his mind up when it came to quitting last month.
Olly revealed: "This was an incredibly hard decision to make and one I didn't take lightly as I've really enjoyed co-hosting The X Factor.
"It's been one big adventure and I loved every minute of it. But my heart belongs to music and I want to concentrate 100 per cent on making my next album and getting back to performing."
He added: "Last November I was doing the show, promoting a new single and I felt increasingly like the music side of things had to be shoehorned into the TV schedule."
News of Olly's exit comes after reports recently claimed he 'should walk before he is pushed' from the ITV talent competition.
A source had told The Sunday Mirror: "Simon Cowell likes Olly and doesn't want him to go under a cloud, so he's being given the chance to walk before he's pushed."
They added that Olly would never consider presenting the show without Caroline by him side.
It's been rumoured she won't be appearing on The X Factor this year either.
Last week, sources claimed Simon Cowell wants a dramatic shakeup of this year's show and that could mean axing Caroline.
It's claimed Caroline could see herself out of the presenting lineup because of "negative feedback" for her efforts with Olly Murs last series. The duo replaced former host Dermot O'Leary, but some viewers preferred the one-person hosting set up rather than having two.
Changes will also be seen to the judging panel as Nick Grimshaw recently announced he is leaving The X Factor too.
Nick confirmed his exit last week. A rep for the Radio 1 DJ said at the time: "Nick was a great mentor on last year's series. He did a fantastic job with the boys' category and we'd like to thank him for all his hard work. We are sad to see him go but wish him all the best."
Nick has since addressed his exit during his breakfast radio show. He told listeners: "I loved Simon and Cheryl and Rita and everyone who worked on it, and all the boys, the mentoring stuff, I loved. I genuinely loved it. There is no bad blood. I just want to watch it on the telly."
Speculation has already begun when it comes to his replacement, with both Louis Tomlinson and Louis Walsh being hotly tipped to take Nick's place.
For exclusive competitions and amazing prizes visit Reveal's Facebook page NOW!