Simon furiously spoke out after Lily slammed the ITV1 reality show for allowing contestant Fleur East to perform 'Uptown Funk' in Saturday's semi-final.
The song, by Mark Ronson and Bruno Mars, was yet to be released by Simon's record label Sony and Lily told fans on Twitter she "smelt something fishy".
Lily has since deleted the posts, but Simon has been quoted in The Mirror saying: "Lily should stop bleating on about all this.
"The irony is I spoke to Lily about being a judge. And from what I took from the meeting, she was seriously considering it.
"Thank God I said no. She'd be a barrel of laughs."
Following her performance, Fleur found herself shooting to No 1 of the iTunes singles' chart - a feat that has never been achieved by an X Factor contestant (other than the winner) before.
Simon also suggested that Lily congratulated the hopeful, rather than think of a "stupid conspiracy theory".
Meanwhile, this weekend Fleur will be going head-to-head with fellow finalists Ben Haenow and Andrea Faustini. The singers have spent the week travelling around the UK on their very own 'Battle Buses' in an attempt to drum up support.
Fleur and Ben sailed through to the finale shows, while Andrea was given his place by judges Simon, Mel B and Louis Walsh following a sing-off with Lauren Platt.
Yesterday, we revealed new pictures of the finalists - and Fleur had got naked for hers!