The husband and wife duo – who have both left Strictly now – told ITV's Loose Women on Tuesday that dancers aren't always happy with who they're paired with but, of course, can't show that on camera.
"It's live and you genuinely don't know who you're going to get," said James, talking about the live launch show where pros and celebrities are paired up. "[So if you're not happy] you can't show that. You're still jumping around but inside you're going, 'Oh no.'"
He then gave an example of being disappointed with a partner.
"I can say the year I got Pamela Stephenson I was slightly gutted because she was a grandma, she was slightly older. I'm a dancer and I know I don't dance the same as I did ten years ago. That's a fact of life, unfortunately.
"Would I prefer someone at 20 who has loads of dance training? Of course I would. It makes my job a lot easier. But in hindsight, when I started training her, I realised this woman was amazing and she made it to the final."
James and Ola both said it was a "shame" that negative comments they've made about Strictly make more headlines than the happy ones.
"When you do an interview, they take all the negative stuff, never the positive stuff, which is quite a shame for us because we love Strictly," said James. "We have a lot of friends on there and it's a juggernaut of a show, it's amazing, people love it, everyone loves it, so do we."
During the interview, the couple also spoke about the early days of their relationship, where Ola was 17 years old when she moved to the UK to be James' dance partner, after he flew to Poland to meet her.
"[My parents] were terrified obviously not understanding James either at that time. I think the dream was so strong for me that I wanted to do what was best for my dancing career. So I just said to my mum and dad 'you have to give me that chance because that chance may never come' and I'm so glad they did or I wouldn't be in the UK or have met James."
James added: "When I went to Poland it was Ola's last chance to potentially fulfill her dream of eventually being a professional dancer. She moved over and a year later we started dating. I made her wait a year!"