The cheeky Essex singer appeared on ITV's Daybreak this morning (6 September) and revealed that he is going to find it too hard to see his former co-star on The Xtra Factor without him.
He said: "I told Caroline I'm refusing to watch the show, I'm going to be too jealous to see a new guy stood next to her."
Olly presented The Xtra Factor alongside Caroline for two years and the pair hit it off immediately, becoming popular with both the producers and the viewers at home.
However, Olly announced that he was leaving the show this year to focus on his music and was replaced by comedian Matt Richardson. Although, it sounds like Olly isn't too keen on someone taking his spot!
While on Daybreak, the pop star also discussed why he prefers to keep his love life out of the public eye and claimed that celebrity relationships never seem to last long.
"I think it's important if you're going to have a relationship for it to sustain… especially being a celebrity or a popstar, relationship sell by dates are a year," he said.
The 29-year-old has been keeping very busy with his music career in recent months and recently he even landed his dream gig supporting Robbie Williams on tour.
Speaking about going on the road with Robbie, Olly said the opportunity was "a dream come true".
"What you see is what you get with him, he's a gent," he gushed. "He's a top guy, he's funny, he's brilliant to be around... Robbie would be a great person to write with. We could do a 'Let Me Entertain You 2'!"
What do you think of Matt replacing Olly on The Xtra Factor? Are you missing Olly? Let us know below, as the best comment each week wins £50!
Watch Olly's video for 'Right Place Right Time' below: