The Irish judge, who has been a member of the panel since the first series, told Radio Times he'd consider retaining his judges' seat if Simon announced he'd be returning to the ITV talent show.
"Now if Cowell came back, I might come back," he said. "The two of us, we would not stop laughing. If Simon called me up and said… but, oh well. Let's see what happens."
Music mogul and XF creator Simon left X Factor UK at the end of the seventh series in order to bring the show to the US, where he currently judges alongside Demi Lovato, Kelly Rowland and Latin singer Paulina Rubio.
There have been several rumours that Simon has been asked to return to the UK version for the eleventh series, although neither he nor XF producers have commented.
Louis, meanwhile, announced earlier this summer he'd be leaving X Factor to focus on other work commitments, including managing former Westlife singer Shane Filan's solo career.
Discussing his decision to leave, he said: "I've been doing it ten years. It's an awful long time to be on the show. I don't want to be pushed. I'd prefer to leave than be pushed. To leave on my terms."
But the silver lining is he's not leaving until the end of the tenth series, which starts on ITV later this summer.
He'll join judges Gary Barlow, Nicole Scherzinger and the returning Sharon Osbourne to search one more time for the UK's next singing sensation.
The boot camp stage of the show wrapped filming this summer, so now it's on to judges' houses - and the start of speculation about which category the judges will have and where they'll be filming!
According to reports, Nicole will be jetting to Caribbean island Antigua, Gary is heading to New York, Sharon is going to Los Angeles while Louis will be flying to the south of France.
Nicole is rumoured to have invited Mary J Blige as her guest judge, while Gary will apparently team up with Xtra Factor host Olly Murs, Louis will be joined Westlife's Shane and Sharon's daughter Kelly Osbourne, and Sharon has yet to decide on her special guest.
Watch the first X Factor 2013 trailer below:
X Factor's winners through the years - pictures:
Copyright: WENN Harsha Gopal