Speaking on Alan Carr: Chatty Man, which will air tonight (31 October), Dermot said the Irish music manager - who has been judging on the ITV show since its inception 10 years ago - always comes out with "something mental".
When asked by Alan Carr if the judges are constantly at loggerheads, Dermot said Louis, Mel B, Cheryl Fernandez-Versini and Simon Cowell "get on really well".
"I think we get to the live studios, they're all fighting for their acts and they all want them to do well," explained Dermot. "They all want to be there or thereabouts in the final – apart from Louis, who doesn't really care... he does care, he does care!
"I honestly couldn't make the show without Louis. I'll be walking on stage and no one will have said anything, and just at the last minute I know Louis is going to come out with something mental and I'll just be like, 'Thank you!'. I don't know where I'd be without him, he's brilliant."
Dermot also called Louis the "dirtiest judge" because "he's the judge that nicks songs" from the other judges. Oops, naughty Louis!
Referring to an incident with Louis last year, Dermot said: "I remember last year with Sam Callahan, and Louis just went up to him on the Friday and went, 'You're going home this weekend' and walked off. I remember Sam just being like crestfallen, like 'but you're my judge'.
"He said it to Sam to motivate him and he got through the next few weeks, but he does say things every now and again, and bless him, he wears his heart on his sleeve."
Elsewhere, today X Factor boss Simon announced a shock twist with a surprise double elimination this weekend.
On Saturday, the first act with the least number of public votes will be instantly sent home from the competition. On Sunday, another act will leave after a sing-off.
Alan Carr: Chatty Man airs tonight (Friday 31 October) at 10pm on Channel 4 with guests Dermot O'Leary, Naomi Campbell, Elijah Wood with music by Paloma Faith.
Watch a clip of Dermot speaking about last week's sing-off with Jake Quickenden and Only The Young below: