The record exec was a judge for the first two seasons of Simon Cowell's now-defunct US talent contest, serving alongside the likes of Nicole Scherzinger, Demi Lovato and Britney Spears.
But in a new interview with Hollywood Reporter, he admitted that after the first series, he wasn't enjoying the experience anymore, partly because he was still running a record label while being a judge, and also because they faced competition from The Voice.
"It's the worst thing I've ever done," he said.
"The first season, yes, I had a great time. It was a little bit of a vacation. I'd been making music, working at running labels for a long time, and it was a little bit of, 'Let me just have some fun.' But the second season I was fully engaged trying run a label, and it wasn't fun anymore."
X Factor USA was cancelled after three series.
X Factor's most shocking moments:
For exclusive competitions and amazing prizes visit Reveal's Facebook page NOW!