Former judge Louis joked about his old boss, who's admitted to loving the anti-ageing injections, saying: "Simon had tweaked The X Factor too much this year and he knows that. There are too many new faces.
"It's like his face – he's tweaked that too much too! Simon's done far too much to his face, he needs to leave himself alone. He needs to not have it as tight. He looks younger now than when he did Pop Idol!"
Louis, 63, had been a huge part of The X Factor since its launch 11 years ago, but was dropped in favour of Radio 1's Nick Grimshaw for this series.
The ratings have since plummeted and Louis, 63, thinks there's something missing from the show, besides him.
"People are bored of all the singing! They really are," he says. "The best singer has to win in the end, but it's a family show on a Saturday night and you need fun. Where are the Jedwards and the Wagners? They're the acts people remember!"
Louis says Simon had wanted to bring him back as a fifth judge alongside himself, Nick, Rita Ora and Cheryl Fernandez-Versini – but ITV wouldn't allow it.
He explains: "Simon rings me constantly. We text until 3am some nights. He wanted to bring me back as the fifth judge. I was slightly up for it then someone at ITV said no.
"I wouldn't do it if they called me back to be a judge on the final. No way – f*ck you! It's too late now. The show has dropped, I'll wait until the ratings are up!"
Outspoken Irishman Louis certainly isn't afraid to speak his mind – and he's told Simon exactly what he thinks of this year's X Factor.
He says: "When the first show went out I rang Simon and said, 'You've really f*cked it up – too many changes'. I did a whole rant. He didn't like me saying that, but then he totally agreed with me the following night. He said, 'You're right and I'm glad you told me'."
So which golden TV moment would Louis scroll back to? "I'd watch Olly Murs' X Factor gaffe in slow motion – when he called out the name before he should have.
"And then watch him apologising - not to the act - but to Simon. 'I'm sorry Simon, don't fire me! Don't call Dermot'. I'd watch that again and again."
Sony BRAVIA now has built-in YouView allowing consumers more access to on-demand content than ever before.
Read more from Louis in the new issue of Reveal magazine - OUT NOW!
X Factor's most shocking moments: