The X Factor boss was critiquing his act Fleur East when he made a comment about hopefuls who leave the show and then "do anything afterwards" to achieve fame.
After being eliminated from this year's X Factor in week two, Jake has gone on to appear in ITV's I'm A Celebrity...Get Me Out Of Here!.
Speaking to Fleur during Saturday night's live show, SImon said: "There are people who come on this show who just want fame.
"They'll just do anything afterwards because that's why they've entered The X Factor." He then told Fleur he could see she was more interested in singing than being famous.
Meanwhile, last night Fleur sailed through to the semi-finals of The X Factor following the public vote. She was joined by Ben Haenow and Andrea Faustini.
Cheryl Fernandez-Versini's act Lauren Platt found herself in the sing-off with Stereo Kicks and, as the judges were unable to reach a unanimous decision, the vote went to 'deadlock'.
Stereo Kicks were then told by Dermot O'Leary that they had received the fewest public votes, meaning they were eliminated from the show.
It has since been confirmed, however, they will return for an appearance during The X Factor final in two weeks time.
Were Stereo Kicks the right act to go? Tell us below, as the best comment from across the website each week wins £50.