Neighbors of a 51-year-old Pennsylvania man who was found with 12 girls in his home say they're angry that Lee Kaplan was able to keep the girls – ranging in age from 6 months to 18 years – under wraps for so long before police finally acted.

Kaplan is charged with statutory sexual assault, aggravated indecent assault, corruption of a minor and unlawful endangerment.

The oldest girl was allegedly “given” to Kaplan by her Lancaster County parents as a prize for helping the family out of financial troubles, police said. The girl was just 14 at the time. Kaplan is accused of fathering a 3-year-old and 6-month-old child with the girl. It’s unclear how or if the other girls are related.

“[The children were] so sad and fearful every time I see them. That’s what made me call,” neighbor Jen Betz said. “I’ve been telling my husband for years, ‘Something isn’t right, something isn’t right.’”

But police said, though they were called to the Feasterville house numerous times during the past several years, they were never tipped off to anything criminal and didn't have a warrant before this month's arrest.

"The neighbors that called two years ago - I don't know what you're referring to - what kind of call did they call, that they saw Amish people? That's the calls that we got, I mean we didn't get child-abuse calls there, so if it was a child-abuse call, we would have responded, naturally, but that's not what we got," Lower Southampton Public Safety Director Robert Hoopes told Philly.com.

Kaplan reportedly did his best to protect the group's privacy. The girls slept on air mattresses in Kaplan’s basement and were home-schooled in a residence with boarded-up windows. They rarely went outside, and when they did, they played in a garden with high weeds. Neighbors told Philly.com the oldest girl held Kaplan's hand.

"I'm saying if you go to the house - he wouldn't even let people go in - even if the police were coming, he would turn the lights out, stuff like that," Hoopes told Philly.com.

That didn’t stop neighbors from having their own suspicions, however.

“You knew something was wrong,” Bob Greenfield said. “It makes you feel bad. If I had said something a while ago, they would have come earlier.”

Neighbor Kevin Rihl said he went to police with his concerns about a year ago, however. But apparently nothing was done at the time.

"How long does it take when the evidence is beating you in the face that something's wrong here?" Rihl told Philly.com.

Daniel Stoltzfus and Savilla Stoltzfus, the parents of the girl gifted to Kaplan, also face charges. The Stoltzfus' have said they're the parents of nine other children in the house -- all except the 6-month-old and 3-year-old -- however, authorities said they've found no documents to back up that claim.

The Associated Press contributed to this report.