The TV presenter co-hosted the last series with Olly Murs, but she has now announced that she will leave her role and will not be returning this year.
Caroline said: "I have had a brilliant time working on The X Factor over the last few years, and hosting the main show was just fantastic, I made some amazing friends. I'm looking forward to carrying on working with the team over at ITV."
Caroline's decision to quit comes after Olly also confirmed he would be leaving the show to focus on his music this year.
Peter Davey, commissioning editor of ITV Entertainment, said: "We thank Caroline and Olly for all their hard work on The X Factor and the contribution that they've made over the past few years. We look forward to working with Caroline again soon and wish Olly the very best as he returns to concentrate on his music career."
Richard Holloway, managing director Thames TV and Interim CEO FremantleMedia UK added: "Caroline and Olly were a brilliant double act on The X Factor, from their early days on Xtra Factor to last year at the helm of the main show. I'd like to thank Caroline and Olly for their contribution to the show and wish them all the best."
It looks like there will be change all round on The X Factor, as judge Nick Grimshaw has already confirmed that he will not be coming back this year.
We've been told that details for the new series of The X Factor are still being discussed and will be announced in due course.
Try Reveal's digital edition on iPhone & iPad. Subscribe now & never miss an issue!