Ben, who is currently working as a news correspondent for NBC, punched the thug in the face after he approached him in London's Oxford Street.
The British star has since relayed the incident on Twitter, but is said to be unhurt and not distressed by what happened.
Ben - who was wearing a suit at the time - told his followers: "Dear Mr Mugger. You picked on the wrong man. Enjoy the black eye. Love Ben."
His spokesperson has since clarified: "He felt the mugger try to grab the bag out of his left hand. And it was just on instinct that he whirled round and gave him a whack with his right hand."
Ben is married and has two children. He found fame in 2000 after starring in BBC reality series Castaway, before going on to present shows such as Countryfile and Animal Park.