Nicole Scherzinger will be jetting to Caribbean island Antigua, Gary Barlow is heading to New York, Sharon Osbourne is going to Los Angeles while Louis Walsh will be flying to the south of France, according to reports.
It has also been said that Nicole is hoping to be paired with Mary J Blige while Gary has apparently called upon former X Factor star and Xtra Factor host Olly Murs for his help.
Louis has asked for two celebrity helpers this year - Westlife's Shane Filan and Sharon's daughter Kelly Osbourne, claims The Sun, while Sharon is yet to finalise who her guest will be.
An X Factor spokesperson said the details of Judges' Houses have not been confirmed, however a source told the newspaper: "Once again, bosses have pulled out all the stops and have chosen some glamorous locations."
Last year, Gary took the Over 28s to Boughton House in Northamptonshire and was joined by former judge Cheryl Cole. Louis Walsh paired up with Sharon and they took the Groups to Las Vegas.
Tulisa Contostavlos and Tinie Tempah flew with the Girls to St Lucia, while Nicole Scherzinger and Ne-Yo took the Boys to Dubai.
Meanwhile, it has also been claimed that ITV bosses have "begged" head honcho Simon Cowell to return to the show.
Following the 2010 series, he left to judge the American version and, although he has returned to the UK for Britain's Got Talent, there are no signs of him coming back across the pond for The X Factor too.
Max Clifford, Simon's spokesman, was quoted in The Guardian saying: "Obviously, everybody to do with the show wants Simon because he gives it an extra quality, we understand that, but with his commitments over there he simply can't be in two places at once."
It has been claimed that Simon earns £100,000 per episode of US X Factor.
Would you like to see Simon return to The X Factor? Tell us below, as the best comment from across the website each week wins £50!