According to The Sun, the Irish music manager and former judge has been approached to make a sensational return to the live shows, where he will join Cheryl Fernandez-Versini, Rita Ora, Nick Grimshaw and head judge Simon Cowell.
The decision is thought to be a move to increase ratings and secure a lead against Strictly Come Dancing.
A show insider told the newspaper: "This is a complete bolt from the blue that no one expected. Louis is an X Factor legend and Simon wants him back for the live shows. They're great friends.
"He will be a neutral fifth judge who will sit in a special place away from the others to provide unbiased comments on the contestants."
Louis parted ways with X Factor after 10 years on the show and was replaced by Radio 1 DJ Nick. Louis was the longest-standing judge on the panel, but he himself admitted that he wanted to focus on managing acts, in particular new boy band Hometown.
Another source said: "Louis could annoy a lot of people, but he was brilliant at creating the drama required to make a show like X Factor worked."
Just last week, former X Factor host Dermot O'Leary said Louis was "loving" the show's reported ratings crisis.
Dermot told The Sun: "I haven't seen Louis for ages but I got a text from him last week saying, 'They are missing us, Dermot - have you seen the ratings?' He was loving that."
According to newspaper, Louis is also set to sign a deal with Simon to help launch Ireland's Got Talent as a judge.
Try Reveal's digital edition on iPhone & iPad. Subscribe now & never miss an issue!
X Factor's most shocking moments: