Cheryl, who has rejoined the UK show's judging panel this year, has spoken about her lows in a heartfelt interview with ELLE UK - and agreed with Simon's decision to drop her.
Recalling the 2011 sacking, she said: "I can talk about this because I'm OK about it now. But I went through hell and it literally drove me mad.
Now I can say that Simon was right to get rid of me. I wasn't well in the head when that was going on, so it wasn't going to work. But he wasn't right not to tell me to my face."
In the interview, which features in the August edition of the glossy magazine, Cheryl also admits that she was suffering up until the beginning of last year.
"It only hit me at the beginning of 2013 that I'd completely lost myself," Cheryl said. "I'd look in a mirror and think, 'Who the hell is she?'
"Something was seriously wrong: it felt like a nervous breakdown."
Thankfully, Cheryl is now back in a great place and, along with her judging job on The X Factor, she's also making her musical comeback.
On 20 July, Cheryl will release her new single 'Crazy Stupid Love', which features Tinie Tempah, and her album will go on sale in November.
Recently, Cheryl announced the name of her fourth studio album after deciding to call it Only Human.
Away from her career, Girls Aloud star Cheryl is loved-up with new French boyfriend Jean-Bernard Fernandez-Versini and was treated to a romantic surprise for her 31st birthday on Sunday.
Cheryl's full interview appears in the August issue of ELLE UK, on sale today (3 July). Also available as digital edition.