Deep in the rolling farmland of western Massachusetts, Bill Cosby's compound has for decades provided quiet refuge for the comedian and his family.

The typically low-key Cosbys have been known to spend holidays and significant parts of their summers in town, situated more than 100 miles from Boston and a popular tourist stop for its locally made arts and crafts and picturesque landmarks. And most people in this normally reserved New England community are content to leave them alone.

But as allegations about years of sexual misconduct continue to reverberate around Cosby, it has become increasingly difficult for locals to ignore their most famous resident, even if they try.

In downtown Shelburne Falls, a village center about 15 minutes from the Cosby compound, some shop owners say they're weary of answering questions about the 78-year-old comedian, who they say never frequented their shops anyway.

"It's sad when your town's No. 1 celebrity-type person ends up going down into a situation like this," says Michael Eller, co-owner of the Sawyer News store, one recent weekday afternoon. "But there's nothing we can do about it. He lives over there doing whatever he does. He doesn't really interact with the town."

More than two dozen women have accused Cosby of sexually assaulting them, sometimes after they had been drugged. Two lawsuits are pending: a defamation suit in Massachusetts and a civil sexual-assault case in California. Cosby, who has never been charged with a crime, has acknowledged having sex with many women and obtaining quaaludes to use during sex. But he says any contact was consensual.

Although many Shelburne Falls residents are still reluctant to pass judgment, they nonetheless find themselves pulled into the debate about Cosby as a growing chorus of women raises more allegations against him.

"I think a lot of people had sympathy for him early on, but it's just a tidal wave right now," says Sidney Anderson, whose family has owned The Baker Pharmacy and its old fashioned soda fountain, for three generations. "There obviously has to be something there."

Others say they feel sorry for Cosby's family, noting they've contributed many positive things to the community since moving to the area in the 1970s, from buying acres of rural land for preservation to speaking out against a controversial natural gas pipeline project through the region.

"I view it as a private matter," says Town Clerk and Selectman Joseph Judd, who has personally known the Cosby family for years. "I can't imagine a finer family, the way they've raised their children, the way they've lived. There's really nothing negative to say."

Most say they wouldn't act any differently if they encountered Cosby.

"I'd probably still say hi," says Justin Nichols, the master roaster at Shelburne Falls Coffee Roasters, a cafe outside downtown that Cosby favors and even donated an espresso machine to once. "Not that I feel bad for him. I'm just a nice guy. I wouldn't want to make anything awkward."

Earlier this month, a half dozen cars were parked outside the Cosby compound and laughter and light banter emanated from the property, which includes a historic 15-room farmhouse and a modern but rustic-style guest house on 21 acres assessed at roughly $2.7 million. But low-slung gates all around the estate bore a stern warning: "If You Are Not Invited, DO NOT Pass Through These Gates."

A neighbor rumbling past in an old rusty tractor says he doesn't pay much heed to the comings and goings of the family, which declined requests for comment through a longtime spokesman.