Dermot, 41, has been at the helm of the ITV talent show for the past eight years, having replaced host Kate Thornton in 2007.
Speaking about music mogul and show boss Simon, Dermot told Nick Grimshaw on BBC Radio 1 Breakfast Show: "The thing about Simon is, he's always been a good boss to me. He always lets me get on with my job, which is the biggest compliment he can give me.
"This idea that he sits across everything and says, 'I want this to happen, and this to happen' is nonsense."
Dermot added: "He trusts people. He's a good boss, he's a fair boss."
Last week, Dermot cast doubt over his future on The X Factor, saying: "I haven't made up my mind yet."
But Dermot has commented on the interview, saying he would still love to present The X Factor when it returns in October.
With talks being held about the show next month, Demot told Nick: "Of course I'd love to do it! We normally sit down in February and start talking about it.
"If they turn around to me next month and say, 'We want a fresh face', then I'd say that I've had eight great years and I'll go and do something else."
Dermot will present tonight's National Television Awards, which will be broadcast on ITV, live from London's O2 Arena.
Reveal is now on Google Plus! Join us here for the latest pictures, fashion and celebrity news!