The handsome actor told ITV's This Morning that after years of "complaining" about his looks being the focus of people's attention, he was now quite happy to be branded the most fancied man in Britain!
"I was pleased, what can I say?!" he laughed.
"At my age it's quite nice to be objectified having complained about it for years, thank you very much! I was very happy."
Rufus, 49, admitted he was quite surprised by the popularity of ITV's Victoria, which starred Jenna Coleman as the iconic Queen Victoria to his Prime Minister Melbourne.
"I was in America when it happened so I wished I was here because it took me by surprise," Rufus said. "I suppose if you have such a nice time doing something, you think it would be too good to be true if people really like it as well. I think it came across how much we had fun."
Rufus said he fell in love with Lord M (as did we all!) after reading the script, saying: "It was just such a great part. When I got sent the script, I was playing a Nazi at the time [in The Man In The High Castle], there in my SS uniform, and I read this part and just liked him so much.
"The relationship between [Lord M and Victoria] seemed so classic, interesting," he added. "I kind of fell for him."
Sadly, it seems unlikely Rufus will be appearing in the now-confirmed second series of Victoria, as his relationship with the Queen effectively ended in series one, and she's now happily married to Prince Albert.
Rufus said: "As they say, fact is a stubborn thing, not long after that he kind of aged really, and was out of office. There was a little bit of crossover when occasionally she kept writing to him, they put a stop to that. Something kind of happens but there isn't much you can do beyond that."
You can, however, get your Rufus fix in the second series of The Man In The High Castle, which is released to Amazon from 16th December.
The first series of Victoria is out on DVD now.
ITV has already confirmed a second series of Victoria has been commissioned and both Jenna Coleman and Tom Hughes (Prince Albert) will return. The series averaged 7.7 million viewers, making it the top rated ITV drama of the year.
Screenwriter Daisy Goodwin said: "Even though she reigned in the 19th Century, Victoria is a heroine for our times. In the next series she faces the very modern dilemma of how to juggle children with her husband and her job. As Victoria will discover, it's hard to be a wife, a mother and ruler of the most powerful nation on earth!"
Queen Victoria, born Alexandrina, ruled from 1837 until her death in 1901. At 63 years, seven months and two days, she was the UK's longest reigning monarch until September 2015, when great-great-granddaughter Queen Elizabeth II overtook her.
Like this? Don't miss out! Make sure you're following Reveal across Twitter @RevealMag / Facebook @Revealmagazine / Instagram @revealmagofficial