Over the weekend, Cheryl wore a green Lanvin dress as she prepared to judge the contestants for the second week of the live shows, but Simon couldn't resist teasing the former Girls Aloud star about her look.
During a heated debate after Stephanie Nala's version of Blondie's 'Call Me', Simon quipped that Cheryl's gown made her look like the famous Muppets character.
Speaking to Reveal this morning, X Factor's latest casualty Chloe, says she was unconvinced of the comparison.
"That was a Lanvin gown," she tells us. "It was exquisite and sumptuous!"
Chloe, who was mentored by Cheryl on the show, continues: "Kermit is a formidable public figure and very well respected and comical. It's a nice comparison but it's not realistic.
"I mean, I don't see it myself. It's such a ridiculous remark, although, it was trending worldwide on Twitter."
Simon's remark did cause quite a stir on social media, with many users tweeting tongue-in-cheek pictures.
Cheryl appeared to take it all in her stride though, and even joked about Simon's teasing with fans on Twitter and Instagram.
Reveal is now on Google Plus! Join us here for the latest pictures, fashion and celebrity news!