The X Factor host apparently made the revelation while on the red carpet last night (20 August) at One Direction's film premiere for This Is Us.
Speaking to the Press Association, Dermot said: "If Simon comes back, the dynamic would be very different. Gary wouldn't be there - he'll be leaving at the end of this year."
Dermot continued to say that Simon would be welcomed back but added that Gary has been a "great head judge" since joining the show in 2011.
Gary is currently on the new series of The X Factor along with Sharon Osbourne, Louis Walsh and Nicole Scherzinger, and will be seen in action when the show returns to ITV on 31 August.
Last month, Take That singer Gary denied reports that he was to quit the ITV talent show, after it was rumoured that he would be too busy working on a new Take That album and tour.
Fellow judge Louis Walsh also announced earlier this month that he would be leaving the show at the end of this series.
Reveal contacted X Factor and they provided us with a statement issued by Gary's rep earlier this month.
It says: "Gary has always managed his role on The X Factor with his music in previous years. He hasn't made any decisions on next year."
Will you be sad to see Gary go if the rumour is true? Tell us below, as the best comment from across the website each week wins £50!