The 'Beautiful' singer will join his old Xtra Factor co-host Caroline Flack in presenting the main X Factor show when it returns to ITV later this year.
And speaking on ITV's This Morning, he said he couldn't be more thrilled to be a host and not a judge!
"The idea of being a judge wasn't really something that I wanted to do. I don't think I could sit there and be horrible to the contestants or say nasty things to them because as a judge you've got to have that opinion and be very opinionated," he said.
"When I was on that 'X' my dream was to be a star and a singer and for me to shatter someone's dream by saying they're not good enough, I couldn't do it. So for me it's better to be next to Caroline and backstage giving them the confidence.
"I'm a hugger and a kisser and to give them that encouragement to hopefully go out and smash it. I'm just going to be there for the contestants and battle with the judges as much as I can because if I believe that this contestant did a great performance on Saturday night, I'm going to tell the judges if they are wrong!"
Olly found fame as a contestant back on the sixth series of the show, finishing runner up to Joe McElderry. He went on to present Xtra Factor and launch a successful solo career.
"Me and Caroline did it for two years of course on The Xtra Factor and my role always with Caroline was that 'Caroline you're the presenter, and I'm next to you having a laugh, having a giggle and just having a good time,'" he said.
"Over the years I've just got more experience and [am] enjoying it. And then when the call came to do The X Factor, it was a bit like 'woah', this is a big one! But I love it. I needed something to kind of excite me, and this was a really exciting thing to do."
X Factor's most shocking moments: