The girl group made it through to Louis Walsh's Judges' Houses and have now been called back to the competition in the wake of Brooks Way's departure.
Louis says he acknowledged that fans were disappointed to see Four Of Diamonds sent home before the Live Shows, which helped him make his decision when replacing Brooks Way.
In a statement, he says: "I was really impressed with the girls at Judges' Houses but only had three spots to fill for the Live Shows.
"At the time a lot of viewers were disappointed that I didn't put the girls through, now that a place has opened up in my category, I've listened to the fans and am delighted to be bringing the girls back.
"They're a great girl group and I'm hoping will take me all the way to the final."
Shortly before the first Live Show episode aired, news broke that Brooks Way - made up of twins Josh and Kyle - had been 'suspended pending an investigation'. Josh, 17, has been accused of attacking and sending threatening messages to an ex-girlfriend. He denies the claims.
On Monday it was confirmed the duo had "mutually agreed" to leave the ITV1 show. An X Factor spokesperson said: "Following the allegations relating to Josh Brooks, we have decided by mutual consent that Brooks Way should leave the competition."
In his own statement, Josh has also said: "Due to events in my personal life I've decided that it's best not to be on the show at this time. While not everything that has been claimed is true I would like to apologise to everyone involved."
Four Of Diamonds will now join 5AM in Louis' Groups category. Louis lost his act Bratavio following the first live sing off of 2016.
Speaking about their return, Four Of Diamonds say: "We are so excited to be back on The X Factor! We're really looking forward to performing live, working hard and showing everyone what we're made of."
Four Of Diamonds is made up of Caroline, 19, Lauren, 21, Yasmin, 17, and 19-year-old Sophia.
The X Factor, Saturday and Sunday at 8pm on ITV1.