On Saturday night's disco-themed show (2 November), Sam performed 'Relight My Fire', which Take That famously covered, but he received criticism from Gary, who said he "hated" his performance.
Gary, who had a noticeably shocked facial expression when the 19-year-old began performing, said: "You are keen to learn but I hated that performance tonight.
"I thought the vocals were really exposed. The thing is, for me, you're not the best singer in this competition. The problem is everyone else is so much better than you."
Nicole Scherzinger also gave her opinion on Sam's performance. "I don't know if everyone else is better, but the vocals are stronger," she said. "You've got so much charisma, so much likability, but that song exposed your weaknesses."
But it was Gary's comments that riled Sam's dad, who defended his son backstage. He said: "I wonder whether Gary's actually judging the right show, if for him it's all about the vocals, why isn't he judging The Voice?"
"This is The X Factor and it's about the charisma, the stage presence and the performance and that's what Sam has."
Sunday night's results show (3 November) saw Kinglsand Road leave the competition after landing in the bottom two with Tamera Foster.
Sam later admitted to Caroline Flack on The Xtra Factor, that he was "shocked" to be saved.
"To be honest, at first I had to do a double take with the fact that Dermot said 'Sam Callahan' as I didn't expect it to come out of his mouth first," he said. "I was just shocked. It was a crazy bottom two but I think to be honest every week it's going to be a shock now."
Caroline then asked Sam if he was starting to feel a bit like last year's contestant Rylan Clark, who often received criticism from Gary each week.
Sam replied: "I do actually, I thought about that before. I do feel a bit like that, I'm starting to feel a bit like Gary's punchbag at the minute."
Watch Sam's performance of 'Relight My Fire' below: