Seems some people are falling out of love with former "Love Boat" star Fred Grandy in the wake of a controversy over comments he and his wife have made about radical Islam.

Grandy, an ex-congressman best known for his TV role as Gopher, just lost a speaking gig in Minnesota after the group he was scheduled to address decided it would be too controversial to keep him on the program. The organization, the Minnesota Association for Financial Professionals, was apparently concerned he might talk about Muslims.

The engagement was nixed shortly after Grandy had a high-profile fallout with the Washington, D.C., radio station where he had hosted a talk show since 2003. Grandy is contractually forbidden from talking about the circumstances of that split, but his wife claims the radio station pressured the couple to stop talking so much about radical Islam, and so he quit.

That decision hadn't affected his side job as a public speaker on the lecture circuit -- until Tuesday, when the Minnesota group informed his agent that an upcoming conference address in April was off.

"It just made sense for us to not have him be a speaker," said Tracy Howard, president of the organization's board of directors.

The event in question was a conference in St. Paul, the state's capital, which has a large Muslim population. Howard made clear that the group was concerned about the issue of radical Islam popping up.

"We felt as a board, given the situation that had occurred with the radio program and some of the comments that had been made by his wife and some of the circumstances of his departure ... that it was in the best interest of our association and our members ... that we seek a different speaker," she said. "We just felt that in the interest of our members that we needed to be sensitive to the issue."

Grandy's wife Catherine said the decision was "silly," since her husband never talks about Islam during paid speaking engagements. She said the two of them have lined up some non-paid, casual events to talk about the issue, but that his paid engagements have always focused on public service, drawing on his background as president of Goodwill Industries.

"He usually does the same speech. ... It's a do-good-for-your-neighbor kind of speech," she said. "He would never bring his issues to a paid speech. That would be foolish."

Still, she said the Minnesota group called Fred Grandy's representative and expressed concerns about "violence" if Grandy were to speak -- even though her husband apparently assured the group earlier in the month that he would not discuss the topic of Islam at the event. His name was still on the conference speaker's schedule as of early Thursday morning.

Catherine Mann-Grandy said the Minnesota organization would still owe her husband part of the $9,000 speaking fee because it canceled so close to the date of the conference. She questioned why the group would be so concerned about his appearance that they would be willing to lose money over it.

Fred Grandy wrote in a column Tuesday that he seems to be "too toxic for the greater Twin City area" due to his critical comments about the Muslim Brotherhood and Shariah law -- though he described his public speeches as "pretty vanilla stuff with no partisan edge."

So far, nobody else has canceled on Grandy.

Howard suggested the Minnesota organization was worried not so much about what he might say but about the "perception" that would accompany him being there.

"It really was primarily because we felt that the situation that occurred was fairly controversial, and there are extreme views on either side," Howard said, adding that the group wanted to "distance ourself" from the controversy.

Though both Fred and Catherine started devoting more segments to the subject of Muslim influence in America over the past year, the controversy Howard was referring to started about a month ago. Catherine had gone on air at WMAL to rattle off a litany of Islam-related charges, allegedly prompting an intervention from management. According to Mann-Grandy, her husband was told to tone down the Islam remarks in the future and to keep Catherine off the air. In response, he resigned.

WMAL's management later challenged the claim that the station tried to limit what Grandy could talk about.