Although, he isn't ruling out a return completely!
Ben's admission came during an appearance on ITV's Good Morning Britain, where he had been comparing working on a soap to a film over in Hollywood.
The 25-year-old's exit from EastEnders aired in February last year, and since then, Ben has swapped Albert Square for the US.
As well as starring in X Men: Apocalypse as Angel - a mutant with bird like wings, he has two films coming out next year.
Explaining how different it has been working on a movie set, Ben told hosts Ben Shepherd and Susanna Reid: "The main difference I found was just having so much more time in film. You'd do a page of dialogue a day opposed to like 12 scenes on EastEnders. It's such a hard craft and you learn so much from doing it, it's just relentless. It's hard to get your best work out when you're doing that kind of structure.
"I actually got a lot more attention on the street from being on EastEnders, you're in their homes everyday essentially, they feel like they know you."
Richard Arnold then asked that all important question: whether Ben would be keen on returning to EastEnders at all.
Ben replied: "Never say never, never say never. Who knows? Not in the imminent future."
That will no doubt have EastEnders fans talking, as many are convinced Ben will return as Peter Beale on the soap.
His comeback was supposedly alluded to on the show recently in a scene with Jacqueline Jossa's character Lauren Branning. Lauren and Peter had left Walford together to go to New Zealand, although Lauren has now returned with Peter's baby.
And instead of having Peter by her side, she came back with his brother Stephen Beale.
In October, fans saw Lauren plan to send a video message to Peter of their son Louis - suggesting he is still alive and in New Zealand - and many viewers took that as a hint Peter isn't completely out of the picture.
Taking to Twitter, one user commented: "Can Peter just come back already #EastEnders." Another said: "TELL US WHAT IS GOING ON WITH PETER!! I'M DESPERATE TO KNOW AND I CAN'T TAKE MUCH MORE OF THIS SECRECY!! #EastEnders."
They're not getting any update from Ben, though!
It was first announced Ben was leaving EastEnders in November 2014, with Peter's exit playing out onscreen three months later.
Follow us on Twitter @RevealMag for all your celeb, fashion and beauty news!
Fancy winning some GREAT prizes? Check out Reveal Competitions here!