Olly had been due to be Ben's best man at the nuptials in 2009, however he stepped down after learning he had made it to the penultimate stage of the ITV reality show.
Days later, Ben furiously sold his story to a national Sunday newspaper and branded his twin a "self-obsessed sellout". The brothers have not spoken to each other ever since.
Lifting the lid on their argument, Olly has now explained how Ben's reaction left him stunned given how big of an opportunity The X Factor was for him.
Speaking to the Radio Times, he said: "For 25 years of my life, I always came first in his life and he always came first in mine. I loved my brother to bits. Anything he needed or he wanted I'd do for him.
"But at that point in my life I was like, as much as I love him, I have to do this for myself, I have to change my life. I never expected him to react the way he did."
Olly came in second place to Joe McElderry in the sixth series of The X Factor and has gone on to launch a highly successful career in the spotlight.
Along with his music, Olly has presented on shows including The Xtra Factor and will this Christmas host his own one-off special, A Night With Olly Murs.
Last week, Olly announced the details of his 2015 tour, which will follow the release of his new album Never Been Better.
Olly is currently on the road promoting his latest music and appeared on BBC Breakfast this morning (25 November).
Watch the video for Olly's new single 'Wrapped Up' featuring Travie McCoy: