The duo have taken over from long-time host Dermot, who chose to step down from the ITV show earlier this year after 11 series on air.
Now, Caroline and Olly have said they won't be trying to emulate what Dermot did on the show as nobody can replace him and they want to bring their own unique style to proceedings.
Caroline said: "It won't be the same because there's two of us. We're not Dermot, no one will ever be Dermot. You can't try and better it and you can't try and make it different deliberately, you've just got to be different."
Olly agreed, adding: "Dermot had his own vibe. He was brilliant, but we are trying to create our own thing and have some fun. We are just trying to be ourselves. I would say we are a lot sillier than Dermot. But I think that's what makes us who we are, we're having a good time. We're with the viewers at home, we're fans of the show, when we're watching from the side of the stage it's as if we're watching it at home."
And one thing they've done that Dermot definitely didn't do: cry during auditions!
Olly joked that Cazza's even bursting into tears before contestant's families do: "Caroline has cried so much! It's so funny. I'm getting the tissues and I'm like, 'Here you are Caroline,' before I have even given one to the mum. I think we watch the show like everyone else, we're big fans of the show and it's been hard for us to not be biased. We're trying to keep a fine balance."
We can't to see Caroline and Olly together again after their magical pairing on Xtra Factor.
"It doesn't feel that much different being on the main show because I am still with Olly and we're still having fun," said Caroline. "It's almost the same team, and the same job - we just have to be a little bit more serious.It just feels the same, but we have grown up a bit."
Olly added: "A touch more serious in terms of the contestants but Simon employed us because of what we did on The Xtra Factor, we made our stamp on that show."
X Factor starts this weekend on Saturday and Sunday at 8pm on ITV.