The violence sweeping Baltimore has begun to scare away visitors, especially among the lucrative business traveler sector.

Tom Noonan, president and CEO of Visit Baltimore, confirmed to FoxNews.com that the Door and Hardware Institute has cancelled its CoNEXTions convention, scheduled to be held at the Baltimore Convention Center April 29-May 1.  The group had booked 2,500 hotel nights in the city for its more than 2,000 attendees.

“At the end of the day, they felt that the current situation left them no alternative,” said Noonan.

In addition, the American Heart Association has cancelled its meeting scheduled at the Hilton Baltimore from April 29-May 1, said Noonan.

These are among the 29 conventions booked this year that were expected to draw an estimated 206,000 people to Charm City, generating an economic impact of $138 million.

It’s too early to know the economic toll the unrest will have on the city’s growing tourism industry, but some have suggested it could be in the millions -- even billions

Over the last 20 years, city officials successfully lured in visitors to the Baltimore Convention Center and other attractions such as Oriole Park at Camden Yards, Sports Legends Museum at Camden Yards, M&T Bank Stadium, Power Plant Live!, and the Reginald F. Lewis Museum of Maryland African-American History.

But on Tuesday, many of those hotspots stood empty. The Baltimore Aquarium remained closed and all events there have been canceled or rescheduled. The Baltimore Orioles postponed a second straight game against the Chicago White Sox on Tuesday due to the riots.

According to officials, a minimum of 20 police officers have  been injured as a result of unrest sparked by the death of Freddy Gray who died while in police custody.  

Now officials are working overtime to build in new security around Inner Harbor, where many tourist sites are located. The Maryland National Guard has been brought in, and there is a citywide curfew from 10 p.m. today until 5 a.m. that will last for one week or longer if need be, Baltimore Mayor Stephanie Rawlings-Blake said. 

Yet, some groups are braving the unrest.  The Food Safety Summit, which opened Tuesday at the convention center, will go on as planned.

“We’re continuing to serve as a resource for other groups that are planning meetings in the city over the next 30-60 days,” said Noonan.