Appearing on ITV's This Morning, the Irish music manager spoke in depth about this year's X Factor panel and how he managed to get his job back after being rested as a judge the previous year.
Louis was in a jovial mood as he openly told Holly Willoughby and Phillip Schofield he'd been "sacked, dropped" during the last series, but still put his two cents in with Simon Cowell when it came to criticisng the changes that had been made.
"I would call [Simon] at two o'clock in the morning and say, 'What have you done with the show? You really messed with the show too much.' I cursed a lot. And he'd say, 'What do you mean?' But he knew exactly what I meant."
Phillip then asked the question we all had: why two in the morning?!
"He would have been at the studio [earlier]," explained Louis. "I know he would be at home watching the show back at about two o'clock. I know Simon's routine because I've worked with him 20 years. I knew he'd be sitting at home thinking, 'What have I done wrong?'"
So, did Simon agree he'd made mistakes? "He said he knew he'd made too many changes," said Louis. "He didn't say mistake."
When Holly asked if Louis felt bad saying Simon made mistakes when last year's panel – Cheryl, Rita Ora and Nick Grimshaw - could be watching, Louis replied no, he didn't.
"I don't care," said Louis. "I actually don't care. I prefer to tell it like it is and the panel this year absolutely works. We get on really well, we love each other. It's genuinely good friends and you have to have people with chemistry on the panel.
"The show is back. We know it's good because we've done so much for the show and it just feels right and the talent is really good this year."
Louis went on to say his main priority was getting his own job back this year then getting Sharon Osbourne and Nicole Scherzinger to join him.
"I wanted Sharon. I wanted to get my own job first then I thought, 'Who would I like to sit beside?' Sharon. Simon gave in on that. He had to be persuaded a little – don't get me sacked again! Then Sharon and I said we would love Nicole because we'd worked with her before when Gary [Barlow] was on the show and we know how hard she worked. And then Dermot [O'Leary] back. He's the icing on the cake, he's brilliant and the public love him."
Louis added that he thought Cheryl did a good job on the judging panel but he personally prefers former Pussycat Doll Nicole.
"[Cheryl] was fine, she was really good. I just prefer Nicole, that's all. Her work ethic. She will want three people in the final this year. She works, she gets it, she sings, she dances, she really mentors over and above."
Finally, Louis said he'd been instrumental in securing Rylan Clark-Neal's role as co-presenter on Xtra Factor alongside Matt Edmondson.
"I would say, 'Simon, give Rylan the gig, he will be brilliant on the show!' But Simon didn't know Rylan as well as we know him because he was away that year and Gary was doing it. He didn't know how amazing and talented Rylan is, or how funny and good he is. He'll just be amazing on the live shows."