Singer Gary, 42, was reported to have told bosses of the popular ITV talent show that he will be too busy to do the series again next year as he will be working on a new Take That album and tour.
However, a spokesperson for Gary says that he is yet to make any plans for next year's show, saying: "Gary has always managed his role on The X Factor with his music in previous years. He hasn't made any decisions on next year."
But a source told The Sun (July 13): "Gary's a decent bloke and figured it was best to lay his cards on the table as early as possible. He wants to get back to what he does best, recording a new Take That album and touring with his bandmates.
"It looks like there's no room for X Factor so this is likely to be his last year.
"The producers always feared his day job would get in the way, and they're delighted they've had him for three years."
Gary's reported departure could have seen Simon Cowell return to the show, according to the newspaper, if American Idol fails to win more viewers in the US.
Gary is currently filming this year's X Factor with fellow judges Sharon Osbourne, Louis Walsh and Nicole Scherzinger.
The show returns to ITV in September.
Would you be sad to see Gary Barlow leave The X Factor? Tell us what you think in the comment box below!