The presenter played coy during an appearance on Alan Carr's Chatty Man, to air tonight, when asked about the judging lineup.
"I know who one is and I've got an idea who the other fourth one is but I keep hearing different names. I'm so excited to find out who it is," she said.
Oh, Caroline! You tease!
When asked outright if BBC Radio 1 Breakfast Show host Nick Grimshaw had landed the gig, she would only reply: "I'm not going to say if he is, Grimmy, but I think he's make a very good judge. I think he would be very very good at that, if he was."
Caroline is returning to the show this year for the first time since bowing out of Xtra Factor. She presented the talent competition's ITV2 spinoff show for three years before leaving.
Now, she's returning as host of the main show alongside old Xtra Factor pal Olly Murs.
Caroline recalled her exit from Xtra Factor: "I thought I was going to go back for a fourth [year] but I got wind that they might have been screening other people for my job. So I thought it was best to step down and let them do the changes that they want to do because they do change it a lot.
"I said my goodbyes to the show. I really loved that show, so I really had to mentally say goodbye and leave it and walk away. Then Strictly came up at exactly the same time and I decided to do Strictly because it was something I always wanted to do.
"That went incredibly well and I loved it so much. Simon kept texting me towards the end saying, "You're doing really well. We should meet up for a coffee". So I was like, "yeah we should meet up.'"
Caroline said she went to have coffee with Simon and just chatted about things but funnily enough, nothing to do with X Factor.
So when she was offered the presenting job alongside Olly, it was a bolt out of the blue!
"We just talked about what we had been up to and different things and he was like "oh its nice to see you" and I went home. The next thing I knew Dermot had stepped down and almost on the same day I got a phone call saying would you like to present the show, but with Olly."
Caroline said she phoned Dermot O'Leary to check things through before accepting.
"Before I made any decisions I thought I've got to phone Dermot and make sure I know everything that is going on and he was just lovely. You know what Dermot is like? He really is the loveliest guy you will ever meet."
See the full interview tonight at 10pm on Channel 4
X Factor's most shocking moments: