The choreographer told his I'm A Celebrity campmate how he became a judge on the ITV talent show – only to lose his job a short time later.
He explained: "I had a meeting with Simon Cowell and he brought me over to the UK as judge of X Factor. It was very brief. They decided to bring Louis Walsh back. They had let him go and brought me in."
Brian added: "I don't know what it was… I think it was always planned, that's how I feel. That he was let go for publicity and I was brought in." (X Factor has denied this in the past.)
Lady C – who has made up with Brian after their rows this series – said: "You have a great personality, even when I wanted to slit your throat I said that. You're bright. I am surprised."
Brian was hired as a judge for the fourth series, to sit on the panel alongside Simon Cowell, Sharon Osbourne and Danni Minogue.
But less than a month after Brian joined the panel, he stepped down and was replaced by returning judge Louis Walsh. Brian remained on the show as creative director and choreographer.
Brian was eliminated from I'm A Celebrity on Sunday night, becoming the second star to leave the jungle. Susannah Constantine was voted out on Friday evening's show.
For exclusive competitions and amazing prizes visit Reveal's Facebook page NOW!