On Saturday night's live show (2 November), Sam performed Take That remake 'Relight My Fire' for disco week, but had a tough round of comments after Gary was left unimpressed with his performance.
The Take That frontman gave feedback and said Sam's vocals "were really exposed." Gary added: "The problem is everyone else is so much better than you."
Sam, who was saved in the public vote, later explained on The Xtra Factor that he felt he like he was Gary's "punchbag" after receiving criticism from the judge week after week.
However, Shane, who was invited as a guest by Louis Walsh for the judges' houses stage of the competition, told the official X Factor website that Sam's vocals are "strong enough to be on the show."
Shane, who has just released his debut solo album You and Me, has backed 19-year-old Sam, explaining that he has "a huge personality" and "his confidence is brilliant."
"His vocals are strong enough to be on the show or he wouldn't be there," Shane continued. "He may not have the strength in his voice like Nicholas does, but he has the personality and that's part of being on The X Factor."
Shane insists that while Gary gives valuable feedback to all the contestants, he may have got it wrong about Sam.
"You couldn't have anyone better giving advice than Gary Barlow, but it isn't always about the best vocals, it's about putting on a performance too," he added.
Shane isn't the only celebrity supporting Sam. Nathan Sykes from The Wanted also told The X Factor: "Sam and I went to school together. We're so proud of him – he's worked his ass off to get this opportunity and I'm so glad the public have taken to him."
Sam will return to the X Factor stage this weekend in a Big Band themed show.
The X Factor airs on ITV on Saturday and Sunday from 8pm.
Watch Sam's performance of 'Relight My Fire' below: