The 'Mysterious Girl' singer was devastated when his beloved sibling Andrew died in December 2012 at the age of 54 following a battle with cancer.
He's now recounted that period of his life in new book, Between Us.
Speaking on ITV's This Morning, he said: "[Losing my brother] tore my family apart. We have not 'till this day - nearly four years later - sat down as a family and talked about it, because it is literally still so raw.
"I was nearly two weeks late delivering [the book] because that was the only chapter I couldn't do. My brothers won't read it. They said, 'Bro I hope you don't mind, but we don't really want to read it.'"
Peter, 43, said the loss of his brother "plummeted" him back to the dark place he'd been in the past, battling depression and panic attacks.
He'd been with then-girlfriend Emily MacDonagh for less than six months at the time and admitted he told Emily she was welcome to leave him.
"I thought there's no way she's [Emily] going to stay with me. She would talk to me and I'd walk into the other room and shut the door. For her, I shut her out completely. And I said to her 'you've got two options… you can either go and do what you have to do, let me get through this, and when I get through this if you want to talk, let's talk. Or you can stick by me.'
"And I really thought she'd say, 'you know what, I need some time out' but she said, 'there isn't two options, there's only one and I'm staying with you'.
"This happened in the honeymoon period, the first six months when everything is supposed to be incredible. If you can get through that… there will be things in the future. I hope one day she needs me and I'll be there for her because she stood by me completely."
Peter added that he wasn't speaking out to gain sympathy, he just wanted to show people that you can go through a dark time and come out the other side.
"I don't want people thinking, 'Oh poor guy.' The upside and positive is I came through it. Millions of families go through it every day, not just anxiety and those things, but losing a family member. You do just sort of learn to live without."
Fancy winning some GREAT prizes? Check out Reveal Competitions here!