James Little, 41, was in Rikers Island for allegedly assaulting his girlfriend when he was released on March 28 over concerns he could contract the virus, the New York Post reported.
On Tuesday, he walked into an Apple Bank in Manhattan's Gramercy Park wearing a mask and gloves and handed the teller a note demanding cash, police told the newspaper.
"Give me money," the alleged note read.
Little was arrested the next day and linked him to two other bank heists -- one on Jan. 23 and the other on Dec. 26 of last year. He left with $1,000 in each robbery, police said.
The New York City Department of Corrections did not immediately respond to a Fox News request for comment Friday.
Little served 20 years in prison for a murder in Coney Island when he was 15 years old, police told the Post. He was released on parole in 2016, they said.
Last month, New York Mayor Bill de Blasio said at least 300 nonviolent inmates convicted of nonviolent crimes would be released from the jail complex. He said several inmates had shown symptoms of COVID-19 after coming into contact with one who contracted the virus.