In reality, however, Spencer's a really nice guy and admits to us: "I fall in love quite easily because I'm a romantic at heart.
"I'm very Disney by nature. I fall into relationships quickly, despite the fact that I'm a bloody useless boyfriend."
"People think I'm this womanising cheat. I actually have a heart of gold and I'm really good to friends and family. But fidelity has always been a problem of mine.
"I'm like a dog in the sense that dogs make mistakes, but they come back and get patted because they are sweet. That's just like me and Jamie [Laing] – we've decided that he's a labrador and I'm a pug."
Most recently, Spencer's been linked to Stephanie Pratt from The Hills. However, he says he's now single after a whirlwind few months with the glamorous reality star.
"She's one of the nicest people I've ever met. I know her brother, Spencer, and agreed to show her the hot spots when she came to London. We really hit it off. We began seeing each other in a romantic light, so I suggested she stayed at mine."
She's now gone back to the States, leaving Spencer free to... erm... make the most of his time!
He adds: "I've only been getting this sort of attention – not just from women, I mean in general - for a couple of years now. It's fun! I want to settle down eventually, maybe in my late 30s. It's lovely to meet great girls and have monogamous relationships, but I trip up because it's not what I want at this time in my life.
Spencer's been busy penning his autobiography, Confessions of a Chelsea Boy, in which he talks openly about everything from drug-taking to the women he's loved.
He also opens up about the tragic death of his brother Mike, who died after climbing Everest when Spencer was just 10 years old.
"Obviously he was doing something very dangerous and the family understood that. But it was very shocking," adds Spencer.
For the full interview with Spencer, grab your copy of Reveal magazine - it's out now.
Spencer's book, Confessions of a Chelsea Boy, is out now.