Before X Factor's twelfth series graced our screens, there were constant rumours surrounding Louis and whether or not he'll be able to keep his chair. The Irish music mogul had hinted his time was up on the show and as the new judging lineup was confirmed, he was replaced by Radio 1 DJ Nick Grimshaw.
But despite Louis being absent this time round, Simon has revealed he still hasn't managed to completely shrug off his good pal.
Speaking to Good Morning Britain earlier today, Simon joked: "I've got the ghost of Louis everywhere at the moment."
After being a judge on the talent competition for eleven years straight, many people quip that Louis had become part of The X Factor furniture. While some fans were pleased to hear he had finally gone, others weren't so happy to hear that Louis had been replaced. So much so, that Simon admits he just can't keep up when it comes Louis.
"What's so weird is that when he was on the show, I'd always have people complaining about him," Simon said. "Now he's gone, it's like, 'why isn't he there?'"
Recent rumours have suggested that Louis could be in for a potential X Factor return when it comes to the live shows.
It's said he could be brought back as a 'fifth judge' and Louis hasn't exactly denied such claims, hinting: "There is a possiblity."
When questioned on whether there is any truth in Louis' return, Simon revealed he did like the idea of having a "guest judge".
"I don't know what's going to happen," he replied. "I couldn't just bring [Louis] back... I do like the idea of maybe having a guest judge, something like that, that's a possibility."
In the meantime, Simon has defended his choice to bring in Nick and Rita Ora as judges this year.
On confirming the new panel, Nick was hit by a lot of criticism, but Simon has said he stands by his decision to employ Grimmy on countless occasions.
Reiterating that he fully supports Nick being on the panel, Simon added: "The truth is, when you replace someone like Louis it takes a while for people to get used to the person they've really, really liked... so regardless of who was going to come in and take his role, there was always, 'well you're not Louis', you've got to give people a little bit of time.
"Now I've got to know [Nick[ as a friend. He is passionate, he's fun, loves the job, very different from Louis, but I think he's doing a good job. And Rita I think is fantastic. I like them. They settled in really quickly."
Simon, Nick, Rita and Cheryl Fernandez-Versini are now at the Judges' Houses stage of the competition and will soon have to pick their final three acts to take through to live shows.
While Simon joined his Overs and 1D's Louis Tomlinson in the South of France, Cheryl headed to Rome with the Groups and Jess Glynne, Rita and Meghan Trainor saw what the Girls had to offer in LA, and Nick and Mark Ronson hosted the Boys in the Cotswolds.
Each act has now performed for their mentor, but won't find out if they have made it into the final 12 until a live show that is broadcast from Wembley Arena.
Eek. We can't wait to find out who's made the cut!
For exclusive competitions and amazing prizes visit Reveal's Facebook page NOW!