The teacher and part-time singer, 23, auditioned for the judges in London this summer, and his audition was aired on Sunday night's episode of the ITV show. He performed an original song called 'Stay' and sailed through to bootcamp.
But Joe then shocked viewers but announcing via social media that he had already walked away from the show by the time the audition aired.
Now, a rep for X Factor has confirmed his departure to Reveal, telling us: "Further to his arena audition, Joe informed producers that he no longer wanted to continue in the competition. Subsequently he did not attend bootcamp."
Ahead of the episode airing, Joe had been quoted as saying on social media: "So mid-week I received a completely random email from the X Factor team. Although I made the choice to not continue in the programme because I felt it was the best thing for my career at the time, they have decided to air my audition.
"SO I PROMISE I DIDN'T LIE ABOUT ANYTHING. But they have chosen to air my audition even though I chose to leave."
He added this morning: "So last night was a crazy. But it's an amazing feeling to have all these new people that I can share my music with :) ..... and it feels even more crazy that I'm waking this morning to go and teach!! I have been working my arse off for nearly 2 years gigging everywhere and anywhere and it feels like something has fallen my way and I just hope you can keep enjoying my stuff."
X Factor auditions continue Saturday at 8pm on ITV.