The Blue singer has been impressing the judging panel with his fancy footwork lately, but ahead of the Strictly semi-final, Simon is fearful he won't have much support behind him.
"I'm a bit worried everyone's going to hate me," he said. "A lot of people thought Pixie was better."
During Sunday night's results show Pixie was sent home from Strictly - despite being dubbed as the favourite to win the show.
Finding herself in the bottom two alongside Simon, Pixie narrowly missed out on a place in this weekend's semi-final, after head judge Len Goodman opted to save her co-star with his deciding vote.
And now, after seeing unhappy fans of Pixie's on social media, Simon is worried the 'Mama Do' singer's departure will have an affect on him as he was saved instead.
Speaking to The Sun, he said: "When I got through I was ecstatic, but reading social media, people thought Pixie was a better dancer and didn't deserve to go.
"Len chose me... perhaps, he saw something the public didn't? I just want to work hard and try not to let Len down."
Proving she has no hard feelings towards Simon, Pixie took to Twitter to wish him and the rest of the remaining celebrities well, following her exit.
She tweeted: "To all left in the show, I wish you all the best and know you will do us all proud xxx."
I love the class of 2014 To all left in the show, I wish you all the best and know you will do us all proud xxx— Pixie Lott (@PixieLott) December 7, 2014
But, while Pixie may have hung up her dancing shoes, the rest of our Strictly stars have been back in the dance studio all week, perfecting their semi-final routines.
Feeling the pressure, Mark Wright told his fans he "didn't want to fall at the last hurdle" after spending an entire day in rehearsals.
He tweeted: "Just arrived home after dancing all day since 9am. Sooooo tired. I just don't want to fall at the last hurdle after coming so far, so making sure I put every possible hour into the semi finals.
"To make the final would be such a shock but also the best feeling in the world."
And, hoping to make the final too, EastEnders actor Jake Wood told his Twitter followers he will be making "every day count" until he gets there.
Even if training can get a little too much at times...
Earlier this week, Jake tweeted: "Have just learnt two dances in two days. Think my brain has fried @JManrara."
Keep going guys!
Joining Simon, Mark and Jake in the semi-final are Caroline Flack and The Saturdays' Frankie Bridge.
Strictly Come Dancing continues this Saturday 13 December at 6.50pm on BBC One.
Reveal is now on Google Plus! Join us here for the latest pictures, fashion and celebrity news!
Watch our Strictly stars in rehearsals for the semi-finals here: