Popstar and actress Rita, 25, sat on the judging panel for the 2015 series alongside show boss Simon Cowell, Cheryl Fernandez-Versini and Nick Grimshaw, but will not be returning this year.
Could that be because of Louis and Sharon, who are both back on the panel this year?
Speaking to our friends at Best magazine, Louis said: "Sharon and I are old, but that means we're far more experienced, unlike the young whippersnappers on last year's panel who didn't know a thing about talent.
"Sharon and I pushed for Nicole [Scherzinger], we didn't want Rita Ora. It turned into a fashion show. The panel didn't have the right credentials. They were lightweight."
Rita quit X Factor after just one year on the panel, as did BBC Radio 1 Breakfast Show host Nick. Cheryl also stepped down after the 2015 series, leaving Simon searching for three new judges.
In the end, he went for three familiar faces: Louis, Sharon and Nicole Scherzinger, all of whom have served as judges in the past.
Original host Dermot O'Leary is also back, replacing Caroline Flack and Olly Murs who co-presented for one year. Over on Xtra Factor, Rochelle Humes and Melvin Odoom have been replaced by former X Factor contestant Rylan Clark-Neal.
Follow us on Twitter @RevealMag for all your celeb, fashion and beauty news!