As the fallout from Phil Robertson's comments on homosexuality continues to rain down, some say its time to put a fork in "Duck Dynasty," while others think the hit show has a few quacks left in it.

On Wednesday, following intense criticism from gay rights groups and others, A&E announced that the patriarch of the duck-hunting family had been placed on "indefinite hiatus" following his comments to GQ magazine.

A source close to the show tells FOX411 that even if Robertson apologized, at this point, A&E is done with him.

But can the ratings juggernaut really go on without its central character, the man who presides over the family and leads the dinner time prayers?

"All shows run their course. But this still has at least another season or two in it," the insider said. "Granted, the overall feeling of uninhibited love that this country has had for the show and brand will likely start to turn a bit."

More On This...

According to our source, given that "Duck Dynasty" boasts such a big cast, losing a cast member won't prevent them from moving ahead with the popular series, even if it ends up being fashioned as more of a "spin-off." But the creative team still faces some major problems as Phil is ultimately the leader of the family and the source of many "words of wisdom" parlayed to the grandkids. He is also the founder of the multi-million dollar Duck Commander business the show features, and thus his absence is likely going to hurt.

Some think these problems – paired with Robertson’s comments -- are insurmountable.

"Phil Robertson's rambling and widely offensive remarks in GQ magazine are likely the last quacks we're going to hear from 'Duck Dynasty.' A&E simply cannot afford to continue airing the program without inciting protests not only from LGBT groups, but also from groups supporting African American and rural interests," crisis communications expert Gene Grabowski, VP of Levick Strategic Communications, said. "What's more, few advertisers will risk supporting 'Duck Dynasty' because of these devastating remarks and renewed focus on a show that invites ridicule of stereotypically rural families."

And while some have mulled the idea that perhaps "Duck Dynasty" could move to another network home if Phil isn't welcomed back into the fold, another key player in the reality production arena argues otherwise.

"There isn't a network out there that would do things differently, with all their major corporate conglomerate ties, they'd all follow a similar course as A&E has," said the source. "It's likely the beginning of the end for 'Duck Dynasty.'”

This isn't the first time A&E has had to contend with remarks made by one of its stars.

In 2007, the network halted production of "Dog the Bounty Hunter" after audio tapes revealed head honcho Duane Chapman repeatedly using the N-word. The media exposure resulted in Chapman issuing a public apology. Three months later, when the outrage had simmered, the network announced the program would return.

Many fans want to see Robertson similarly reinstated. A&E is facing threats of audience boycotts and outrage from devoted fans of the show since it made the decision to suspend Robertson, with many calling it an act against freedom of speech and those who hold traditional Christian values. There are also 14 million "Duck Dynasty" lovers who tune in every week to hear the Christian family do their thing - a huge audience to possibly upset.

"A&E has an enormous piece of business with 'Duck Dynasty' between the show itself and all of the licensing and merchandising. They did the right thing by acting fast in support of equal rights, but they also will inevitably invest quite a bit of resources in protecting the asset," observed our insider. "But whatever happens going forth, for a long-lead interview with a print magazine [like the one Robertson gave GQ], they should also have a PR executive there to supervise. Reality TV personalities are not trained in media and can't always be expected to say the right thing at the right time."

While Robertson did not apologize for his comments, he did explain them in a statement: "I myself am a product of the 60s; I centered my life around sex, drugs and rock and roll until I hit rock bottom and accepted Jesus as my Savior. My mission today is to go forth and tell people about why I follow Christ and also what the bible teaches, and part of that teaching is that women and men are meant to be together. However, I would never treat anyone with disrespect just because they are different from me. We are all created by the Almighty and like Him, I love all of humanity. We would all be better off if we loved God and loved each other."

Follow the writer of this story @holliesmckay