Wisconsin Gov. Scott Walker took heat from both sides of the aisle on Monday for saying it's "legitimate" to discuss building a fence along the U.S.-Canada border. 

The Republican presidential candidate made the comments in an extended interview with NBC's "Meet the Press" posted online a day earlier. Asked about the possibility of a northern border wall, Walker said people in New Hampshire have brought that up with him.

“They have raised some very legitimate concerns, including some law enforcement folks that brought that up to me at one of our town hall meetings about a week and a half ago,” Walker said. “So that is a legitimate issue for us to look at.”

At first, the jabs came from Democrats. But soon, Walker’s Republican rivals started to pile on. 

“There have been a lot of dumb ideas put out,” Sen. Rand Paul, R-Ky., said in an interview with Boston Herald Radio. “One, that the Mexicans will pay for a wall, [which] was probably the dumbest of dumb ideas. But putting a wall up between us and Canada is sort of a ridiculous notion."

Democratic Vermont Sen. Patrick Leahy, who was born and raised near the Canadian border, called building a wall to keep Canadians out “crazy.”

“Election season always brings out crazy ideas, but this is one of the craziest,” he said in a written statement on Monday. “If Marcelle’s (his wife's) parents were still here they would say the same thing, but in French.” Leahy married Marcelle Pomerleau in 1962. Her family is French-Canadian.

Leahy also said Walker "simply must be unaware of the economic prosperity that commerce across the northern border brings to the United States." 

Walker, though, defended his statement on Twitter, as one of former Florida Gov. Jeb Bush’s political backers used the incident to slam Walker on another thorny topic -- the Iran nuclear deal. 

Mike Murphy is head of the Right to Rise super PAC, which supports Bush’s candidacy. 

Walker tweeted: 

The Walker campaign has not responded to a request for comment from FoxNews.com.