After just one year on the judging panel, the 'RIP' singer has followed in the footsteps of Cheryl Fernandez-Versini and Nick Grimshaw in deciding to quit the ITV talent contest.
Rita confirmed her exit via Twitter, saying: "I had a ball on The X Factor last year and will miss the team."
She also sent a message to show boss Simon Cowell saying: "Can't wait to work with you again. I'll be round for dinner soon. Thank you for the experience & love."
An X Factor rep has now added: "Rita brought a great energy to the show last year and did a brilliant job mentoring the girls' category, leading Louisa to victory. We wish her all the best with her music and film plans for the rest of the year."
According to The Sun newspaper, Rita had been "very keen" to return to the show but work commitments made the scheduling impossible. "At the end of the day, it didn't work out with her other commitments around the world in music and acting."
Rita's currently filming the second Fifty Shades of Grey movie in Vancouver, reprising her role as Christian Grey's sister Mia.
The 25-year-old joined X Factor last year after making the switch from BBC One rival show The Voice UK, where she'd been a coach. She'd previously served as a guest judge on X Factor in 2012. Rita won last year's series with her finalist Louisa Johnson.
Her exit means the only judge from last year still remaining is Simon himself.
No new judges have been confirmed yet but there has been mounting speculation that former judge Louis Walsh is the frontrunner to take one of the panel spots, while ex-judges Nicole Scherzinger, Sharon Osbourne and Mel B have also been rumoured.
As well as changes to the judging panel, there will be a new host, as Dermot O'Leary returns this year to replace Caroline Flack and Olly Murs, who both left after just one year presenting the main show.