Peter, 58 - who also does the voiceover for Britain's Got Talent and Live at the Apollo - announced his exit via Twitter on Tuesday afternoon (28 July).
He said: "After the best 11 years with the X Factor, it's time for me to check out and cut loose."
Who else is going to announce the judges' names with so much gusto?!
A spokesperson for X Factor confirmed: "We can confirm Peter Dickson has left the X Factor after 11 fantastic years. Everyone involved in the show would like to thank him for his sterling service for us over the years and we all wish him well in the future."
Peter, who once said that his favourite name to pronounce was contestant Rachel Adedeji, said in July last year that Cheryl's name change had left him a little tongue tied.
Peter's exit follows on from the departure of Dermot O'Leary as main host after eight years, who has been replaced by Olly Murs and Caroline Flack. In addition to this, Rochelle Humes and Melvin Odoom are now the new Xtra Factor hosts, having replaced Sarah-Jane Crawford.
After the best 11 years with the X Factor, it's time for me to check out and cut loose.— Peter Dickson (@peterdickson) July 28, 2015
Mel B and Louis Walsh have also been replaced on the judging panel by Rita Ora and Nick Grimshaw. They join Cheryl Fernandez-Versini and Simon Cowell.
Following last week's auditions, X Factor is now at the Boot Camp stage.
For exclusive competitions and amazing prizes visit Reveal's Facebook page NOW!
X Factor's most shocking moments: