Dermot chatted about the coveted role during an interview on This Morning today, but insisted that he doesn't know who will be joining Simon Cowell and Cheryl Cole on the panel.
The X Factor host even said that Louis Walsh's judging role has not yet been confirmed - although he wants him to return!
Dermot said: "I can't do The X Factor without Louis, as he's the only person I can beat up on each week!"
He added: "Only Simon and Cheryl have been confirmed. It's going to be a really tough role. No one knows [who it is] yet.
"Whoever it is, whether they are replacing Nicole [Scherzinger] or Sharon [Osbourne], it is big shoes to fill.
"They've got to be able to stand up to Simon. You can't be a lapdog in that role. You've got to be your own person."
Last month, Cheryl confirmed that she was coming back to The X Factor after three years away from the show.
Rumoured names for the fourth judge have so far been Rita Ora, Fergie and Kesha.
Who would you like to see as an X Factor judge? Tell us below, as the best comment from across the website each week wins £50!