But Rylan's now told The Sun newspaper that he never thought of his job co-hosting with Matt Edmondson as lasting more than one series.
Rylan said: "Personally when I signed up to do Xtra Factor I looked at it as a one year thing. We've both got our other jobs that we love and always will have hopefully.
"But we've both been looking at this as just a one season thing so wanted to enjoy it and take it for what it is and if we are going to go out we want to go out on a high."
But he did joke: "If they offer us a couple million then f**k it!"
It's true that Rylan is super busy: he hosts Big Brother spinoff show Bit On The Side, recently fronted his own late night talk show, has a regular showbiz slot on This Morning, made a cameo in the Absolutely Fabulous film, and just penned an autobiography.
And in an interview with The Telegraph earlier this summer, Rylan admitted he still has ambitions to check off his list!
"I always wanted to be an architect. I'd like my own design company in future, but it wouldn't be the normal run-of-the-mill endorsement. I'd also love to design a home range where everything complements everything. I've got a real eye for detail and my dream range would include home aromatherapy, vases, artificial flowers, photo frames, candle holders, candles, bedding, curtains, the lot.
"Would I be hands-on? Anything I do, I'm the one who designs it, makes it and signs it off. If anything has my name on it, be assured it's me that's done it from start to finish without anyone else getting involved."
Rylan and Matt were brought onto Xtra Factor this year as a replacement for Rochelle Humes and Melvin Odoom.