The Take That star also declared that a novelty act like last year's finalist Rylan Clark would never have the chance of winning the competition...
He said: "I don't believe a novelty act will ever win this competition because when it comes down to it, I do actually believe in the people who are voting to choose the right person.
"I was quite shocked that James Arthur won last time, not because he wasn't the best, but because he is the best and was the best."
James, who will release his debut album later this year, went on to have the biggest and fastest selling X Factor single ever with his rendition of 'Impossible', which sold over 1.2million copies.
Gary added: "I assumed a TV voting audience wouldn't vote for someone like that and they did, it really shows me that the right person will win this competition each time."
Gary, who celebrates the 10th anniversary of the ITV show this year alongside fellow judges Sharon Osbourne, Nicole Scherzinger and Louis Walsh, is yet to win the show with one of his acts.
The pressure is on this year," he said. "I have had a couple of finalists but never had the winner."
Gary, who revealed that the groups and the girls were the strongest categories this year, also shared his highlights from the audition process so far.
"For me, the young girl who sang the Whitney Houston song was amazing, she fluffed every line and usually that would be a car crash but it was an amazing car crash, a gorgeous one. I thought the prison officer was amazing too," he added.
The X Factor will be back with a bang as it returns to our screens with double weekend episodes! Dermot O'Leary will host the main show, while Caroline Flack and newcomer Matt Richardson will co-host the ITV2 spin-off series, The Xtra Factor.
The X Factor returns on ITV1 on Saturday 31 August at 8pm. The next episode will be broadcast on Sunday 1 September, also at 8pm.
Watch Gary Barlow answer some more viewer questions in the video below: