The X Factor choreographer became the second star to leave the jungle on Sunday night, after Ant & Dec said it was between him and Yvette.
Speaking on spinoff show Get Me Out Of Here NOW, he said: "The only person I really felt bad for that I wish I could have swapped places with is Yvette. She is miserable in there and crying all day and I would love to see her go home because she needs it."
Joe Swash later said: "I do think it's a shame [to see Brian go] especially when there are people like Yvette who doesn't like it but wants to go home."
When it was asked why Yvette didn't just leave then, he said: "She can't! She won't get paid! She has to stay in there!"
Most Haunted star Yvette said last week that she wanted to go home, after that massive row between Lady C and Tony, sparked by Tony and Jorgie's refusal to act as servants for Lady C, Kieron and Chris in a trial.
She had said: "I woke up this morning, and I sleep with the canopy over my head, and I was awake for a long time, and for the first time I just thought, I don't want to come out of my little canopy. I don't want to be here anymore."
For exclusive competitions and amazing prizes visit Reveal's Facebook page NOW!