An unfortunate picture of Simon did the rounds online over the weekend after viewers noticed the rather timely position of the music mogul's foot… making it seem as if another part of his anatomy had popped out.
Phoning into Xtra Factor on Sunday night, Simon told Rylan and Matt: "There are one or two things I want to address based on headlines this morning… Toe Gate.
"Just to be clear, it was my toe. It was quite embarrassing when I saw this online. I wasn't wearing any shoes. It was definitely my toe. And I have a very large toe, I'll admit!"
Simon held his Judges' Houses in Malibu this year, inviting Spice Girls Mel B and Emma Bunton to help him choose which of the seven girls would make it to live shows.
"What I thought would happen in my mind didn't happen," admitted Simon. "You always think they'll be as good as you think they'll be, and people maybe not quite so good won't change, but it really did change on the day and half way through I said to the two girls, 'I genuinely don't know what I am going to do here.'"
Simon said Mel B and Emma actually made things even harder for him, as they had strong, differing views on the girls they thought should make live shows.
"We all had different opinions and when we finished I said, 'Let's make this easy, Mel, tell me your three, Emma, tell me your three, and I'll show you my three.' We all had different opinions and strong thoughts. Any one of those seven could have made it through to the finals.
"But you have to make a decision based on who you want to work with, who you can work with, who is going to improve, and I think we did choose the best three."
Simon chose Emily Middlemas, Samantha Lavery and Gifty Louise.
X Factor Live Shows start Saturday at 8pm on ITV.