Simon didn't specify which singer he was referring to, however he said the star originally had dark hair when he auditioned for the show.
The X Factor mogul signed the band in the 90s and was reminiscing about it during an appearance on The Tonight Show With Jay Leno.
Simon, who is set to become a father for the first time next year, began by telling host Jay how he regretted not working with the Spice Girls.
He said: "I knew they were going to be big, but I was too late on the deal. They didn't want to sign for me, so that kind of hurt at the time."
Simon the continued: "Another group auditioned for me and they were called Westlife.
"When they first came along, they were the ugliest boy band in the world. Seriously, like you couldn't get uglier.
"I said to the manager at the time, 'Forget it!' And then he re-auditioned and one of the ugly ones dyed his hair blonde for the second time I saw them and he got back into the group.
"I didn't recognise him but it all worked out fine and they sold about 50 million albums." Ouch!
Despite his harsh words, Simon has remained on good terms with Westlife.
Shane Filan appeared on The X Factor this year at Louis Walsh's Judges' Houses and Simon has now congratulated Kian Egan on his I'm A Celebrity...Get Me Out Of Here! win.
Following Sunday's final, he tweeted: "Well done kian. Very happy for you."
Take a trip down memory lane with Westlife's first music video: