The former footballer raised eyebrows among viewers when he was seen telling former X Factor contestant Jake he couldn't work out why Jake was in the jungle or what his talent was...
While some viewers agreed it was banter, others thought Jimmy had taken it too far.
And it now seems that the interaction with Jake may have contributed to Jimmy being the first celebrity voted out of the jungle. We saw him exiting the show last night - despite being the early favourite to win.
Speaking on ITV's Good Morning Britain today, Jimmy said he understood how the conversation between himself and Jake may have come across on TV, but insisted it was very different in camp.
"It was totally tongue in cheek. The way it came across on TV is totally not how it was in camp. Saying that, I can be a bit of a nightmare. The best person to ask about this is Jake. We're great friends and will be for a long time. And he gave as good as he got as well! I can see how it could have come across harsh in reality for the punters to watch."
Jimmy said he didn't know why he'd been eliminated first, but added: "I've seen some clips there and how it can portray you in the show, but when you're there 24/7 things are so different."
The ex-athlete also spoke with Ant & Dec on the main I'm A Celebrity show, where he was again quizzed about his "banter" with Jake and whether it went too far.
"I get what you're saying," answered Jimmy, before going on to explain: "I sort of had my lines where I could go [with the contestants] but with Jake and Foggy, I let them have it full blast. Well, I thought I could anyway!"
I'm A Celebrity airs nightly on ITV.