The girlband were sent home by groups mentor Louis at Judges' Houses in a decision that sparked massive controversy among viewers who had thought the ladies were a shoo-in for live shows.
Even Louis admitted it had been his "hardest decision."
But Four of Diamonds unexpectedly got their chance to return for live shows when Brooks Way left the competition following allegations about Josh Brooks' personal life.
"Obviously it's under unfortunate circumstances that we're here but we still feel so lucky," said the girls in a new interview with Reveal magazine, out now.
"When we first met Louis, he asked us, 'Are you mad at me? Have you forgiven me yet?' We were like, 'Erm… we're not sure Louis!' We all sat down and had a really long chat. Louis is very honest. He even admitted he usually hates working with girl bands as they don't usually get on but says he has a good feeling abut us."
Louis' been very positive about Caroline, 19, Lauren, 22, Yasmin, 17, and Sophia, 19, saying he enjoys working with them as they are all friends and each member can sing.
He told The Sun: "I don't particularly like working with girlbands because it's very hard to get a girlband that actually all get on. These four seem like they are really good friends... I'm really glad I have them now."
On this week's Xtra Factor, Louis even said they could have a career as big as Little Mix, who are the first and only group to ever win X Factor.
Four of Diamonds made their live show debut during last weekend's Motown themed second performance show, and made it through to week three which is Diva themed.
They tweeted on Monday they'd just chosen their as-yet-unannounced song for the show and said Little Mix – who performed on the Sunday results show - had given them good advice.
"They were really lovely & gave us loads of advice on how to make the most of our time on the show," the band tweeted.
Pick up Reveal magazine now for more from Four of Diamonds including what Brooks Way said to them and how they got together after auditioning as solo artists.