The music mogul told ITV's Good Morning Britain that people failed to realise the decision to hire or fire does not rest solely with him.
Rumours about Louis' position on the show have been running rampant for weeks and it seems Simon has had enough!
He said: "I'm not sitting there in the dark going, 'You know what? Today I'll fire Louis.' It is what it is, you know? But it does irritate me sometimes thinking, 'I'm not the only person making this decision!"
So far, bosses have not officially confirmed Louis' status on the show. Louis himself said he has neither been fired or quit and nothing has been set in stone.
If he does return, however, it will be to a rather frosty welcome from fellow judge Cheryl-Fernandez Versini following their war of words this week.
Louis kicked it off by branding Cheryl lazy and saying she was irrelevant, so she hit back, suggesting Louis is "obsessed" with her as he always namechecks her. She also advised him to use his mouth for positivity.
Addressing the feud today, Simon said: "They've never liked each other those two… [When Cheryl was in Girls Aloud] he was their manager. Now Louis managing a girl band, disaster, he just doesn't do it well."
X Factor's most shocking moments: