X Factor boss Simon said he fully supports James' career since he won the show in 2012 but following recent backlash, has advised the rising star to "keep his mouth shut".
"He has my full support as an artist but he's got to keep his mouth shut and he's got to stay off Twitter," says Simon, who spoke to Reveal at Britain's Got Talent London auditions.
"He's got to stop feeling sorry for himself because winning a show like that is a really cool thing if you handle it well. Selling a million records is a cool thing."
James returned to Twitter early last month after ditching the social networking site in 2013 following a number of high-profile spats. Last year, he was embroiled in disputes with One Direction, Matt Cardle and Frankie Boyle, and faced public backlash after making a homophobic slur in a rap song.
'Recovery' singer James did apologise for his actions during a return to The X Factor in December, but last week, the 25-year-old was embroiled in another Twitter spat with a model.
"The only thing he has to complain about is people complaining against him because of what he's said," continues Simon. "If I responded to every negative thing said to me, I'd be on Twitter all day long. He's just got to calm down, maybe he should take yoga classes."
"You can't get into public wars and you can't say things he's said and get away with it. Yes, he's apologised, but he has to stay off Twitter, please!"
After winning X Factor two years ago, James went on to score the biggest selling X Factor single of all time with 'Impossible'. He kicked off his first UK and European headline tour in January and will wrap up on 4 March in Warsaw, Poland.
Got an iPad or iPhone? Reveal Mag has been especially designed to fit!
Watch James' music video for 'Recovery' below: