Updated

Clay Allison literally built his New Jersey nursery from the ground up, but now he fears his 80-hour workweeks are no match for an economy stuck in neutral and government regulation that seems aimed at stunting the growth of small business.

Allison fears his nursery could join some 200,000 small businesses that have failed since 2008, and, as he and others cling to their livelihoods, President Obama's claim the entrepreneurs owe their success to others rings hollow. In particular, Allison bristled at Obama's declaration that small businesses succeed because the government invested in roads and bridges.

[pullquote]

"He's absolutely clueless," said Allison, who lives in the rural western New Jersey county of Sussex. "It's so foolish to think that building roads and bridges increases commerce. It makes no sense. Infrastructure improvements are the result of business and profits, not the other way around."

Fourteen years ago, Allison bought 250 acres of farmland and started the Beaver Brook Nursery with nothing but seeds and sweat. The trees and shrubs he planted and cultivated were sold to landscapers and now beautify homes throughout the Garden State. But when the real estate market fell in 2008, so did demand for his leafy inventory. He had to let all his employees go, and now hangs on, mainly because his wife has a job and he lets hunters use his property.

"When [the economy] collapsed, people stopped landscaping," Allison told FoxNews.com. "Five years ago, when things were good, I thought in a year or two I would be in a position to retire."

Obama's words, delivered two weeks ago at a campaign stop in Roanoke, Va., came as he pushed for tax increases on those making $250,000 or more, a tax critics say will impact small business owners' ability to hire.

“If you got a business, you didn’t build that. Somebody else made that happen,” Obama said. There was a great teacher somewhere in your life. Somebody helped to create this unbelievable American system that we have that allowed you to thrive. Somebody invested in roads and bridges. If you've got a business, you didn't build that. Somebody else made that happen."

A recent Gallup Poll found that 59 percent of business owners disapproved of the commander-in-chief’s comments, and that figure includes Dan Withrow, who employees 15 workers at CSS Distribution, in Louisville, Ky.

“He has a tainted sense of what works and what doesn’t,” Withrow said. “We have experienced growth, but that was through hard work. Many small business owners I know are scared to death and sitting on their hands, waiting to see what happens.”

Withrow said rising taxes and new health care mandates are preventing him from expanding.

“I will have to pay an extra $30,000 this year. I could have hired another worker,” he said. “I’ve tried to soften the blow, but it still hurts me and my employees.”

Withrow's brother, Jim Withrow, is also an entrepreneur, but he fears his catering business is being killed by government regulation. The company, Splendid Fare, did lucrative business with the pharmaceutical industry before a law was passed that limited drugmakers to spending just $100 on on food per doctor, per year. Pharmaceutical companies often try to pitch their products to physicians, sometimes over meals.

“I’m probably down two-thirds in sales this year,” Jim Withrow told FoxNews.com. “It’s kicking us in the rear end. We use to average a 150 people a day I have to actively trying to find a buyer for my company.”

“And the government has been more of a hindrance that anything when I started my business. I didn’t have any help," he said. "I’m proud of what I did and don’t want to give it up. I just don’t see it rebounding to what it was in 2006.”