James played Phil Banks - Will's uncle - in the 90s American sitcom and died earlier this week reportedly from complications following open heart surgery.
Alfonso, who played his on-screen son Carlton, has described James as his "second father" in a touching tribute to the actor.
He said: "I'm deeply saddened to say that James Avery has passed away. He was a second father to me. I will miss him greatly."
Jada Pinkett Smith, who is married to lead Fresh Prince star Will, took to Facebook to pay her respects to James.
"I was hoping to start the new year differently but we have lost yet another friend. James Avery who we all lovingly know as Uncle Phil from The Fresh Prince has passed," she told her fans.
"Our condolences to aunt Florence (his mother), Miss Barbara (his wife) and all those who loved him. Rest in peace James."
James played Uncle Phil Banks from 1990 to 1996. Tatyana Ali, who starred as his on-screen daughter Ashley, also tweeted a message of condolence.
"James was my teacher, my protector and the most magnificant actor. We're feeling his loss very deeply. He'll always be a part of me," she said.