Rita had been a guest judge on The X Factor back in 2012, the year James had won on the show, and James has now revealed that romance did blossom between himself and the 'Never Let You Down' singer.
The 28-year-old has said he and Rita had "seen each other" for a few months.
Speaking to the Evening Standard, James claimed: "Me and Rita were seeing each other on-and-off for a few months. All of a sudden we just stopped talking."
"Maybe it was bad for her image to be seen with me," he added. "I was a little bitter about that because she'd said she had a lot of feelings for me."
Rita and James were first linked back in 2013, after appearing to get close at an X Factor Live party, and later being spotted at a strip club together. At the time, Rita had insisted she wasn't dating anyone and was "too busy" for a boyfriend.
James, meanwhile, had alluded to their romance, after things between them had fizzled out. A rap he had debuted on Radio 1Xtra appeared to hit out at Rita and included the lyrics: "Some people's auras [Ora's] and demeanours can deceive us, what they really want is a piece of your penis."
James has now, however, said that the rap was just "wordplay."
News of James' history with Rita comes after her recently made his musical comeback. James released his new single 'Say You Won't Let Go' and headed straight to the top of the charts, re-signing with Simon Cowell's record label, Syco.
He appeared on The X Factor on Sunday (9 October) to perform his new track and was praised by Simon and his former mentor Nicole Scherzinger.
Nicole had said following his performance: "I wanted to say I'm so proud that we get to be back here together, I've always told you, no matter what, the cream rises to the top and you're back on top. To me, I love you, you're the best X Factor winner this show has ever had."
James, meanwhile, took to Twitter and confessed to slipping up while on stage, thanks to his nerves. He tweeted: "That was the most nerve wracking thing I've ever done so apologies for the pitchyness and incorrect lyrics but thank you so much everyone (sic)."
Follow us on Twitter @RevealMag for all your celeb, fashion and beauty news!
Fancy winning some GREAT prizes? Check out Reveal Competitions here!