Andrew Ahlering, the former county employee behind the campaign to relieve Baca of his post, said he had gathered just 40 of the nearly 400,000 signatures needed by December to get a recall measure on the ballot.
Ahlering, who accused the Los Angeles sheriff of giving Hilton preferential treatment by allowing her to return home after serving less than four days of a 23-day jail sentence, also said he had raised less than $100 for the campaign.
But he told the Los Angeles Times on Monday that his effort would get a second wind from Nicole Richie's having spent only 82 minutes in jail for driving under the influence.
The recall petition accuses Baca of giving favorable treatment to celebrities and of "gross mismanagement of the largest jail system in the United States."
A Baca spokesman said Ahlering's difficulty generating enthusiasm for the campaign is evidence the sheriff has the public's support.
"The sheriff has said when the hysteria dies down about the Hilton incarceration people will come to see he made the right decision, and his interest is in the safety of the people of Los Angeles County," sheriff's spokesman Steve Whitmore said.
Hilton, 26, Richie's co-star on TV's "The Simple Life," walked out of jail after a three-week stay in which she was briefly released to home confinement and then sent back screaming to a lockup. The hotel heiress-socialite was jailed for a probation violation after pleading no contest to alcohol-related reckless driving.