The former X Factor choreographer clashed with posh Lady C on numerous occasions in the jungle before his elimination on Sunday night.
He's since told Daily Star: "When I was a kid in school I got bullied a lot – I haven't really had that happen in my adult life. When Lady C was calling me [names] that immediately came back to me because I felt helpless."
Despite their rows in the jungle, Brian and Lady C did make amends towards the end, even arranging a dinner date in China Town!
"I think people are afraid of her and once I got the lashing of her tongue I was afraid of her too and stepped back. What we went through with the storm brought us together and we were all somewhat cohesive by the fire last night," he explained.
He added: "I don't ever want to get in a fight with her and knowing that her opinion can't be changed on anything leaves me in a great place with her. I keep my thoughts to myself and I will be safe. She loves food and I love food so I trust her and I'm going on a date with her."