The music mogul told The Sun it was "humiliating" to have made an error and named Gifty as the act he wanted to eliminate rather than girlband Four of Diamonds.
"It was embarrassing and humiliating! That's the nature of live TV but it was horrible," he said.
"People should have voted for her that's the way it goes."
Host Dermot O'Leary, meanwhile, added: "My heart was in my mouth, I just had to ask him again. I think he jumped the gun a little."
Simon's act Gifty landed in the bottom two alongside Louis Walsh's group Four of Diamonds. After they both sang for survival, the judges had to vote who to send home .
By accident, Simon named Gifty, before quickly correcting himself and saying Gifty was, obviously, the act he wanted to save. Unfortunately for him and Gifty, she went home after the other judges – Louis, Nicole Scherzinger and Sharon Osbourne – all saved the band.
Gifty later joked to Hello magazine about Simon's gaffe: "That was a kick in the stomach, 'Oh great even my mentor wants me out and thinks I'm rubbish.' But I did think it was a mistake and I laughed it off."
Meanwhile, mentor Simon has apologised to Gifty, saying he was to blame for her exit as he'd chosen the wrong song for her to sing this week. He picked Fifth Harmony's relatively unknown 'In Love With A Monster' and admitted on Xtra Factor it was too gimmicky.
Gifty said she personally felt it was "predictable" to do a second Fifth Harmony song in just four live shows and she worried the audience would be bored, but she also understood Simon had her best interests at heart.
"Simon has been absolutely fantastic, he's always had my best interests at heart and has always done what made me feel comfortable. With him saying he picked the wrong song, I don't blame him, I feel he honestly felt it was the right song for me and I would smash it," she said.
"I tried my absolute best. I knew it was a risk doing Fifth Harmony song, like third time. I personally feel people are going to think she's too predictable, but you know what, I did it and I have complete faith in Simon; he knows what he's doing."
Meanwhile, appearing on Good Morning Britain on Monday, Gifty said she held no hard feelings towards Honey G, who has made it safely through to the fifth live show despite being branded a "novelty act" by some viewers.
"Personally, I feel like she's an entertainer, she entertains people and people have fun when they watch her, so her making it through, I wasn't surprised. She has a huge fan base, so good on her. She gives a great performance.
"I don't hold it against her because she isn't the one doing the voting, people do the voting, and obviously she was fan favourite."
X Factor continues Saturday at 8pm on ITV.