Recently, U.S. Attorney General Lynch released the “Roadmap to Reentry,” a set of principles to reduce recidivism through federal reentry reforms. The roadmap is a laudable beginning, but it stops short. Beyond individualized plans for those in the Bureau of Prisons, the roadmap provides no specific answers regarding the difficult terrain returning citizens must traverse in their life after release. Beyond the gates await 44,000 documented barriers that restrict where those with criminal records can live and study, what kind of work they can do and whether they can vote or access credit. These restrictions are so omnipresent that they constitute a form of perpetual punishment, a “second prison.”

The second prison is not a small problem. More than 600,000 Americans are released from custody each year, and 65 million people—or one in four—have some form of criminal record. Even if a person’s arrest relates to a minor offense in the past, a criminal record will continue to have serious, ongoing consequences for them and their families. In communities where many people have a criminal record, the cumulative effects are devastating; one out of 13 African Americans of voting age is unable to cast a ballot due to criminal disenfranchisement. Nationally, the second prison costs as much as $65 billion in lost economic output each year.

We need comprehensive national action to help address the second prison. This should be done thoughtfully; some ongoing restrictions on some people are necessary and appropriate. But many other aspects of the second prison are arbitrary and counter-productive. To reduce recidivism and address decades of over-incarceration, we must do more than invest in people’s reentry while they are incarcerated. We must make sure they have access to opportunities once they are prepared to contribute to the community instead of taking from it.

One important part of the puzzle is to make sure that fewer people wind up with a criminal record in the first place. Over-criminalization of some activities, including nonviolent drug offenses and regulatory infractions committed without criminal intent, have helped increase the number of people in the second prison. In 2007, an employee at a retirement home diverted sewage from blocked-up toilets into a storm drain to prevent flooding. He wound up with a year’s probation—and a criminal record—for unwittingly violating the Clean Water Act. Surely cases like these do nothing to ensure justice or protect public safety. Reforms are needed to ensure that sentencing corresponds to the seriousness of the offense, the defendant’s intent and sentences imposed for similar crime in other cases.

We should also do away with restrictions on returning citizens that do not actually protect the public. It is appropriate, for example, to make sure that people convicted of violent sexual offenses are not working in contexts that would endanger potential victims. But who is safer because someone who once grew cannabis cannot become a certified ASL interpreter, as is the case in Minnesota? Who is protected by laws that block a mother‘s access to temporary food assistance because of an old, drug-related conviction? We should hold on to restrictions that are proven to protect the innocent, but we should jettison those that senselessly prevent people who want to start over from succeeding.

Finally, as a society, we need to challenge the stereotypes that allowed the second prison to emerge in the first place. Fear and stigma make employers, landlords and governments leery of extending second chances to people with a criminal record. However, studies have found that, after a period of time, people with a criminal record are scarcely more likely to be arrested than a member of the general population. A Harvard sociologist studied the service records of people who enlisted in the Army under a waiver that allowed them in despite a felony conviction. Those with a felony waiver were, on average, no more likely to get kicked out than their peers, and they were actually promoted faster and to higher levels. At the very least, we must acknowledge that people with a criminal record have dignity, worth and potential equal to any of their neighbors. When we keep them forever on the margins, we deny these attributes and miss out on their unique contributions to society.

The Roadmap to Reentry is a good beginning. It will help prepare people to make the difficult transition from prison to the community. But the investment of time and energy in these reforms will ultimately be poor if the road does not lead somewhere, to a fair chance to achieve economic mobility and to participate in civic life. Having completed their sentences, the growing number of Americans who live with a criminal record should not face obstacles that so vastly outnumber their opportunities to build better futures for themselves, their families and their communities.