In an interview with The Mirror's TV Mag, the music mogul said Louis Tomlinson has expressed an interest in joining the panel in the future.
Simon, 55, said: "Going forward, one of these guys will do one of these shows with us. Louis in particular wants to and he'd be very good, but in terms of girls, it was really Rita that we wanted – and thank God we got her."
While there is no room left on this year's X Factor for Louis to join, there is still judges' houses, when each judge traditionally invites a famous friend to join them in deciding which acts to send through to the live shows and which to send home.
Could we see Louis at Simon's judges' houses this year?!
Louis, 23, would certainly know what he was doing, as it was on X Factor five years ago that he was put into a band with the other 1D boys; a band that became a worldwide phenomenon.
Meanwhile, this year's X Factor judges' houses will have a big twist: it's live! Yes, LIVE! Hosts Caroline Flack and Olly Murs will host from the studio in London, while viewers get to watch all the action live from the judges' houses around the world.
That means act will perform live at judges' houses and we'll find out live who goes through to the main shows and who is going home.
Simon said: "This is either the best idea we've ever had or the most stupid thing we've ever done. Olly Murs is our safety measure. I'm sure he's got his live set ready to go and we will just cut back to Olly's concert. I am not kidding. It's going to be chaos, but god chaos."
Reveal is now on Google Plus! Join us here for the latest pictures, fashion and celebrity news!