Louis' been quite vocal this series about how much he enjoys working with Four of Diamonds as, in his words, they can all sing and all get along.
But on Sunday night's Xtra Factor, Simon couldn't help asking: "Who were you inferring [about] last night on the show when you were talking about 'it's nice to work with a group who can actually sing?'"
Louis replied: "I was talking about a lot of girl groups, Simon, worldwide. A lot of girl groups who maybe have one main singer, you know what I mean?"
The Irish judge then pointed towards his fellow judge Nicole Scherzinger and said: "Nicole was the lead singer in Pussycat Dolls."
But Simon replied: "I think it was disrespectful bearing in mind how many groups have done well on the show whether it's Little Mix, One Direction or JLS. It was kind of inferring that they weren't very good."
Louis insisted: "No, I wasn't inferring that."
"Well then," said Simon. "You can apologise."
Louis and Simon's acts went against each other in last night's sing off which could perhaps explain the tension: it was Simon's artist Gifty Louise who was sent home after the other judges all chose to save Louis' band Four of Diamonds.
Simon admitted it was probably his fault that Gifty was in the bottom three because he'd chosen her song, 'In Love With A Monster' by Fifth Harmony, and it was too gimmicky for her.
Louis, meanwhile, vowed Four of Diamonds will be back with a bang next week, after two weeks in a row in the sing off. "I think the girls have turned a corner now and they're starting to look like a group, and next week you are going to see an amazing improvement. Honestly, they are the nicest people I've worked with."
X Factor continues Saturday at 8pm on ITV.