The Irish judge, who has managed Boyzone and Westlife, is said to have been let go after 11 years on the hugely successful talent show.
Louis, 62, seemed to confirm the reports today when he tweeted: "All good things must come to an end."
According to The Sun On Sunday, Simon told Louis personally that he was no longer required on the show.
A source tells the newspaper: "They are close friends so Simon wanted to do this the right way. So Simon spoke to Louis and told him that it's 95 per cent certain that he'll be 'stood down' this year."
However, Louis has apparently been earmarked by Simon for Ireland's Got Talent, and may be tempted back to The X Factor as a music expert and behind-the-scenes aide.
During an interview in April on Irish chat show The Seven O'Clock Show, Louis appeared to confirm that he was leaving The X Factor.
He said: "I'm done with X Factor, honestly. I've done 11 seasons and that's a long time in television - I don't think I'll be going back."
He added: "I think they are going to change the show. I think they are going to get new faces."
After Louis' tweet on Sunday (10 May), one fan responded with the message: "Ah it won't be the same."
Another said: "So they're replacing a fitted fixture with something modern. It's never going to the same without @walshlife on the scene."
A convert admitted: "I've grown to love to hate you over the years! Sad you're going - won't be the same without you! X."
Louis Walsh, you'll be missed!
The X Factor's most shocking moments:
For exclusive competitions and amazing prizes visit Reveal's Facebook page NOW!