The former Blue member slammed the ex-Dollar singer for making an intentionally light-hearted comment about suicide.
Simon said: "He annoyed me when he said he could have topped himself after spending a night in a cave. I've lost friends that way - so it upset me.
"But I expect him to be in the final."
The 29-year-old - who compared his time in the camp to "living in a pressure cooker" - insisted he was "relieved" to leave the jungle and doubted he would have survived much longer on the show.
He added: "I'm relieved to be out. I was surrounded by massive personalities.
"I didn't think I'd last long because I didn't do many trials. But it was nice to be a part of it, I enjoyed the mental part."
The other three remaining contestants are former 'EastEnders' star Joe Swash, 'Star Trek' actor George Takei and tennis ace Martina Navratilova.
Comments: Our rules
We want our comments to be a lively and valuable part of our community - a place where readers can debate and engage with the most important local issues. The ability to comment on our stories is a privilege, not a right, however, and that privilege may be withdrawn if it is abused or misused.
Please report any comments that break our rules.
Read the rules here