The Take That singer, who has returned to the ITV talent contest along with Louis Walsh, Nicole Scherzinger and Sharon Osbourne, said that old pals Sharon and Louis have already teamed up together.
In the first group interview conducted by X Factor host Dermot O'Leary, Gary joked: "Already there's a division here. I think we should swap Louis, right now because you two are pairing up and I want to sit with Mrs O!"
We wonder if Gary will be making a permanent seating move when it comes to the live shows, as the new official shots of the four judges, which were unveiled earlier today show Gary in between Sharon and Nicole! Last year, the 42-year-old sat on the end of the panel.
The judges joined together yesterday for the first time to kick start auditions in Glasgow, Scotland, and when asked if there will be conflict, Gary said yes, but "in a good way".
Sharon, who made a grand entrance to yesterday's proceedings with two semi-naked men on either arm, also admitted that she never expected to return to the show after her exit in 2007.
The four judges are now on a quest for Britain's next star and shared their thoughts on what talents they are looking for this year.
Louis said: "We're all looking for different things, it's not all about the voice." While Sharon explained: "You can have the best voice in the world, and the oldest saying is 'they have the perfect face for radio'. You've got to have the personality."
"We're not looking for people who want to be artists, we're looking for people who are artists already," Nicole added.
Xtra Factor's Caroline Flack and Matt Richardson kick off auditions
Demi Lovato: "Me and Simon Cowell are very close, he's really cool"
Watch the judges talk below: