The controversial contestant, who freely admits to being one of the weaker singers on the show, divides opinion up and down the country as he remains in the competition while talented singers like Ella Henderson are sent home.
But the 24-year-old seems to have had enough criticism and has gone on a lengthy Twitter rant, telling trolls their comments are "disgraceful".
"Don't think people understand some of the stuff that I get sent day in and day out. Some of you should be locked away for your comments," the tirade began.
"Regarding my sexuality, it's absolutely disgraceful. Imagine that was your son or brother that was receiving those messages enough is enough."
In a brutally honest series of messages, the Essex lad told followers his mother is "seriously ill", that he had "never come from money" but had refused to use these things as a "sob story" to keep him in the show.
The rant from the normally cheerful contestant may have been sparked by alleged comments made by judge Gary Barlow on Tuesday night.
According to an unconfirmed report in The Sun, Gary was less than complimentary when he quizzed the audience at a Mencap charity concert on what they thought about Rylan. (We've contacted X Factor for a comment).
Rylan responded with tweets that have since been deleted (see below), apparently calling Gary out on his alleged comments: "I'd rather be "absolute s***" than just stand there and sing the same old stuff every week on the spot....... #boreme. @GBarlowOfficial."
With the final looming in the not too distant future and acts like James Arthur finding themselves in the bottom two, this tension is likely to come to a head on Saturday night's show.
Do you think Rylan get too much abuse? Or should he have been sent home a long time ago? Let us know in the comments below and you could win our Comment of the Week prize of £50!
Read more: Is X Factor's Christopher Maloney a big phoney? We thrash it out!
Read more: Ella Henderson leaves X Factor over James Arthur in shock results