The BBC Radio 1 Breakfast Show host is a new addition to the judging panel this year, alongside fellow newbie Rita Ora and the returning Simon Cowell and Cheryl Fernandez-Versini.
"My friends are mixed about it," he said. "Half of them are like, 'You've ruined our Saturday nights because we can't come to your house and watch it,' and then half of them are super excited because they can come to the show and watch it.
"They've already been at auditions and they want to come again. As a friendship group, we all love it, so overall everyone's excited."
Nick said X Factor is the only show he's purposefully stayed in at the weekends to watch, and it became tradition for people to gather at his house to view it.
"Me and my friends are all at my house, we cook and we watch it every Saturday and Sunday together. It's good, it's like rehab because we don't go out from August til Christmas and then you feel lost for the rest of the year."
Nick said his love of X Factor meant Simon didn't have to convince him to do the show - and that Simon didn't even wine or dine him like he did with new hosts Olly Murs and Caroline Flack!
"He didn't wine or dine me, no wining and dining from Simon whatsoever. I heard that with Olly and Caroline he took them out for dinner, he put on a full curry for Olly, every dish available – but none of that for me, he didn't even treat me to a lunch or anything! I've had nothing, I don't even know if I'm getting paid!"
X Factor starts this weekend on Saturday and Sunday at 8pm on ITV.