The music mogul has teased we'll be seeing an audition in an upcoming episode in which new co-host Olly comes face to face with his doppelganger.
Simon said: "He's more than a lookalike and soundalike, he's a complete stalker! I asked him if he has ever watched a movie called Single White Female, I asked Olly if he had seen the movie too because this guy has literally channelled Olly. He doesn't just look and sound like him, he actually wants to be him.
"I had this vision in my head of Olly getting locked up in a room with tape around him one week while this guy takes his job. I think it might happen. If Olly goes missing, then turn up at this guy's house and look in the garage or the garden!"
Simon has nothing but praise for Olly, who is hosting this year's main shows with old Xtra Factor colleague Caroline Flack, following the departure of Dermot O'Leary.
He said: "Olly is just a dream to be honest with you. Like [new judge] Nick [Grimshaw], he is one of the happiest people I have ever met in my life. He's really enjoying it and I think he's showing that side of him which people genuinely like. He's sold a lot of records but when he's standing on the side talking to the contestants, there's no ego. Him and Caroline have a great chemistry.
"Look, it's diﬀerent. As I have said before, you have to make changes sometimes and see what happens. It brings a diﬀerent energy to the show."
X Factor starts this weekend on Saturday and Sunday at 8pm on ITV.