The Sons of Anarchy actor told Entertainment Weekly that he felt guilty for walking away from the erotic movie when he'd already built up a friendship with the rest of the cast and crew.
He said: "It was very, very painful for me, that whole process, just very unpleasant. I felt like I let myself down. I let those people down on Fifty Shades that I'd grown to really love and respect. It's just a really unpleasant period of time for me."
Charlie, 35, told the magazine he'd become overwhelmed with work commitments: "You see it all the time in Hollywood. There's a tendency to make hay while the sun shines and to capitalise on every second once somebody gets hot and people just get burnt out. That's what I started to do."
The actor pulled out of playing Christian Grey in Fifty Shades back in October 2013. At the time, it was said the reason was scheduling conflicts with his TV show Sons of Anarchy.
Northern Irish actor Jamie Dornan was then cast opposite Dakota Johnson in the movie, which went on to break box office records and pull in more than half a billion dollars worldwide.
Watch a scene from the Fifty Shades movie below: