The music mogul declined to name names but did confess on BBC Radio 1 this morning that not every act who makes it through to live shows is gracious and polite.
"Some contestants you get are hateful," said Simon.
"Literally, to everyone around them - the production team, the runners... Then they go on stage and say they're doing this for their mum or the royal family or whatever. Then they'll go off stage and be hateful to everyone else.
"And I sit there thinking, 'I so want people to know what you're really like...'"
Simon also said if he wanted to know what a contestant was like behind the scenes he turned to the make-up artists, hairdressers or drivers who spent the most time with them.
"The people who are around them all the time will tell me the horrendous ones," he said. "And we've had a few..."
Oh, Simon, come on, names!
X Factor continues Saturday and Sunday night on ITV.
Reveal is now on Google Plus! Join us here for the latest pictures, fashion and celebrity news!