Gilbert, who posted a letter to Cavs fans on the team's website shortly after James announced his plans to sign with the Heat, told The Associated Press late Thursday night that it's "accountability time" for James.
"He has gotten a free pass," Gilbert said in a phone interview with The AP. "People have covered up for (James) for way too long. Tonight we saw who he really is."
Gilbert feels James quit on the Cavs during their second-round series against the Boston Celtics, who rallied from a 2-1 deficit to eliminate Cleveland.
"He quit," Gilbert said. "Not just in Game 5, but in Games 2, 4 and 6. Watch the tape. The Boston series was unlike anything in the history of sports for a superstar."
The Cavaliers were beaten by 32 points in Game 5. During the game, James appeared distracted and uninterested, often glaring at Cleveland's coaches as the Cavs tried to foul to get back into the game in the second half. James also made some puzzling postgame comments, saying he had "spoiled" people with his play over seven seasons.
Gilbert also said he believes James quit on the Cavs in Game 6 of their series in 2009 against Orlando.
"Go back and look at the tape," he said. "How many shots did he take?"
Gilbert, who has owned the Cavs for five years, said he was most disappointed by James' behavior in the months leading up to the superstar's announcement that he is going to Miami to play with Dwyane Wade and Chris Bosh. Gilbert said James never returned a single phone message or text since the end of the season and that the Cavs were not informed of James' decision until he went on the air.
Gilbert said Rich Paul, one of James' business partners, called the Cavs moments before the announcement.
"LeBron James needs to go to another team with two superstars already so he can win a championship," Gilbert said. "We will win a championship before (the Heat) do."
Gilbert said he now wishes he had done some things differently with James, who spent seven seasons with the Cavs.
"It's not about him leaving," Gilbert said. "It's the disrespect. It's time for people to hold these athletes accountable for their actions. Is this the way you raise your children? I've been holding this all in for a long time."
In Miami, Wade said he was stunned by Gilbert's comments.
"I think I'm happy that I have the owner that I have here in Miami," Wade told The Associated Press late Thursday night. "I'm happy Micky Arison is my owner. I couldn't believe it. I'm speechless. It's very unfortunate and I think it makes LeBron that much better about his decision.
"We knew 'Bron would take some backlash," Wade added in his interview with AP. "I told him he's a strong man for it."
AP Sports Writer Tim Reynolds in Miami contributed to this report.