They've also fallen for the charms of his adorable son Eric, two!
Sam Lavery, Gifty Louise and Emily Middlemas will perform in tonight's first live show, which has a theme of Express Yourself.
When we caught up with them yesterday, the girls were full of praise for Simon, who is mentoring the girls' category this year, saying he's devoted loads of time to helping them get ready for the show.
Sam told us: "You see him on screen and everyone has this image of him being really scary and not approachable but he's so reassuring to us and he's always there for us. Anything, you can go to him like if you're a bit unsure about something."
"It just makes us more confident," she added. "He's helping us grow and he knows where he wants us to go and has that vision which makes us feel really good. He believes in us and gives us strength.
"I just think the way he is with us just gives us 100 per cent more faith in ourselves than we had before. He's always reassuring us saying, 'I picked you guys, I believe in you.' That's what we want to hear: that our mentor believes in us."
The girls have all met Simon's cute son Eric, two, who is a familiar face behind the scenes of X Factor and Britain's Got Talent. And, like everyone else who meets Eric, they've fallen for his charms!
"He's so cute!" they all gushed when we quizzed them.
"He gave me a high five yesterday," added Emily, while Gifty said: "He's literally a miniature Simon Cowell."
Gifty added that watching Simon with Eric just shows that he's not the Mr Nasty he's been portrayed as in the past. "When we were in rehearsal yesterday they walked in together and Eric was sitting on the chair next to him and Simon was patting him, like, 'Are you okay?'"
Sam laughed: "When we all sang he was looking at Eric like, 'Do you like them?' We have to impress Simon and now Eric as well!"
"Do you know what?" added Gifty. "I don't think the whole Mr Nasty thing was actually him. I think it's just a perosna. Simon Cowell genuinely is a nice person. He's such a teddy bear! You sit down and have a conversation with him and think, 'I thought you were a nasty person when I used to watch you on TV!' but he makes you feel so comfortable. He's so down to earth. Yes, he's rich, but he doesn't flaunt it. He's just so humble."
The X Factor airs Saturday and Sunday, 8pm on ITV.