The group - made up of best friends Mikey Bromley, Charlie George, Parisa Tarhomani and Betsy-Blue English - were sent home from the ITV1 show on Saturday night (22 November) after Simon Cowell announced there would be a shock double elimination.
As a result, Only The Young were shown the door without the chance to fight for survival in a sing-off. Meanwhile, Stevi Ritchie was eliminated the following night after a sing-off with Andrea Faustini.
Speaking exclusively to Reveal today, the band say they felt a sing-off could have saved them from the chop.
Mikey says: "I wanted to do a sing-off. It was frustrating. I feel that if we had done one, we'd still be in the competition.
"We had planned to do 'I Won't Give Up' by Jason Mraz and had devised a really good take on it. We hadn't done a ballad before, so it would've been nice to show the judges and the public that we can do that sort of stuff too."
Throughout the competition - and before he was eliminated last night - Overs singer Stevi had caused controversy for staying put.
So were the band annoyed to leave before him - albeit 24 hours?
When asked, the group exclaim "No!" in unison, before Mikey replies: "Stevi got really upset when we went out. I said, 'Stevi, don't be upset – the public kept you in and you got more votes than us'."
Parisa adds: "We personally think Stevi was great on Saturday. We watched him backstage and were so proud of him. I don't think anyone else can pull off a pharaoh's outfit while being covered in honey and feathers. Plus, his sing-off was amazing. If he'd have won, good on him."
Only The Young were mentored by Louis Walsh throughout this year's X Factor and the band said he was "gutted" when they were voted out on Saturday.
"He said his goodbyes to us and said he is going to miss us," Betsy-Blue reveals. "He just kept saying, 'You guys need a record deal!' He said he doesn't have any worries about us. He said he knows we're going to well."
She adds: "Also, Cheryl invited me and Parisa into her dressing room to say goodbye. She said the same as Louis - that she's going to miss us."
Only The Young have big plans for 2015, with their main aim being securing a record deal. And they are already well prepared!
"We started an album before the show and can't wait to get it released. We've got a good sound," Mikey reveals.
Parisa adds: "There's a couple of covers on there that we love, and that we performed on The X Factor, but the majority is music we have written ourselves."
Betsy-Blue explains: "Each of us have our own genre of music that we like and it will all come together on the album."
The band will kick off next year with 'The X Factor Live Tour 2015' and they are very much looking forward to hitting the road together.
"We cannot wait for the tour," Parisa gushes. "The best part of the show is performing every weekend, so the fact we're going to be doing it every night is so exciting. I love that I'll be touring with my three best mates too. There'll be lots of pranks!"
The band currently live together at Parisa's parents' house but are hoping to move into their own place in London after Christmas.
But do they ever fight?
"We get on each other's nerves all the time, but we have a brother and sister relationship," Parisa explains. "We bicker all the time but we don't fall out over anything serious. We've always made up again within minutes."
Although they are positive about the future, it's clear Only The Young are sad that they didn't make it all the way. But if they had their time again, would they have done anything differently?
"Personally, I would worry less," Charlie reveals. "Everyone is focused on getting through each week but, looking back, I think I needed to embrace it a little bit more and relax.
"I wish I'd got excited to the point that my head hurt, rather than hold back that final little bit."
Mikey adds: "I think I would try to make that last performance a bit more entertaining. Maybe it wasn't entertaining enough? Who knows!"
Were you sad to see Only The Young go? Tell us below, as the best comment from across the website each week wins £50.