A retired U.S. lieutenant general who made comments denigrating Islam withdrew Monday from speaking at a West Point prayer breakfast after a veterans' advocacy group asked the Army chief of staff to rescind the invitation.

VoteVets.org told Gen. Raymond Odierno in a letter that allowing retired Lt. Gen. William G. Boykin to speak at the U.S. Military Academy next week would be contrary to Army values and disrespectful to Muslim cadets.

Late Monday afternoon, West Point issued a brief statement saying Boykin had decided to withdraw speaking at the Feb. 8 event and that another speaker would be lined up in his place.

Boykin, a former senior military intelligence officer, had been criticized for speeches he made at evangelical Christian churches beginning in January 2002. He said that America's enemy was Satan, that God had put President George W. Bush in the White House and that one Muslim Somali warlord was an idol-worshipper.

Boykin later issued a written statement apologizing and said he didn't mean to insult Islam. But VoteVets.org said Monday that Boykin has continued to make denigrating comments about Islam since his 2007 retirement.

"These remarks are incompatible with the Army values, and a person who is incompatible with Army values should not address the cadets of the United States Military Academy," VoteVets chairman Jon Soltz said in a letter written with the group's vice chairman.

Army public affairs didn't immediately comment. West Point's Lt. Col. Sherri Reed said cadets are "purposefully exposed to different perspectives and cultures" during their four years at the academy.

In a statement issued earlier Monday, Reed noted that Christian, Jewish and Muslim cadets would be participating in the prayer breakfast, and she had expressed confidence that Boykin's speech would "be in keeping with the broad range of ideas normally considered by our cadets."

Boykin has continued to attract controversy since his retirement. The Council on American-Islamic Relations and People for the American Way had asked officials in Ocean City, Md., to rescind an invitation to speak at a prayer breakfast last week. Boykin attended and spoke about his faith.

CAIR also asked West Point officials to retract Boykin's invitation

"It gives Islamophobes a platform at the nation's most prestigious military academy. And I doubt that they would invite a KKK speaker and claim that they want to expose the students to a variety of opinions," said CAIR national executive director Nihad Awad.

Boykin didn't return a call seeking comment or respond to an email sent to his account at Hampden-Sydney College in Virginia where he is a visiting professor.

A Pentagon investigation concluded that Boykin violated regulations by failing to make clear he was not speaking in an official capacity when he made nearly two dozen church speeches beginning in January 2002. It also found that Boykin, who made most speeches wearing his uniform, didn't get prior clearance for the remarks.