The rapper married Britain's Got Talent star Alesha in 2005 but they split a year later when he had an affair with Javine Hylton. Harvey and Javine later had a daughter, Angel, ogether.
Speaking to housemate Jasmine Lennard on last night's show, he said their separation hadn't been nasty: "People read all these things like there was some bit war happening. There's never been one cross word or beef.
"I got off lightly. She could have ripped me apart and took me to zero but she dealt with it graciously."
Harvey went on to say that he was Alesha's "world" and she trusted him whole-heartedly and never looked at another man while they were together.
"I regret the pain that I caused, of course I do," he continued. "I lied to God because we got married in a church... you're giving someone your bond and trust and you destroy it."
Harvey, who had to don several colourful lycra suits throughout the episode to raise money in the shopping task, added: "What makes it worse is that every time I get serious with a girl they go back to that moment because it's documented, isn't it?"
Jasmine then told him she doesn't think he would have behaved the same way if there had been children involved.
Celebrity Big Brother airs nightly on Channel 5
Read more: Natalie Cassidy's Celeb BB column: I'm torn between Jasmine and Rhian
Read more: Celebrity Big Brother: Is it ever good for your career?