Decades ago, Senator Patrick Leahy remembers, neighbors in his home state of Vermont would drive into Canada, and from there do what they couldn’t do in the United States – fly to Cuba.

His Canadian friends would ask “What is wrong with you?” – referring to Americans generally and the national prohibition against traveling to Cuba.

When he got to the U.S. Senate in 1975, Patrick Leahy said, he encountered with the same question from leaders from around the world.

“I thought the United States was isolating itself through the embargo,” Leahy, a Democrat, said in an interview with Fox News Latino. “Europeans and Latin Americans said to me, ‘How do you feel with the U.S. treating Cuba as if it were some major threat to you?’”

Now Leahy has emerged as the central figure behind President Barack Obama’s stunning announcement last December that the United States and Cuba were going to resume diplomatic relations after more than half a century of being sworn enemies.

The normalization of relations was, for Leahy, the crowning jewel of a long-time mission that gained momentum and urgency with the detention of Alan Gross, a former U.S. government contractor who was charged with counter-revolutionary activities for distributing computer equipment inside Cuba, which is prohibited.

[Cuban officials] made very clear that they appreciated what I did [to return Elian Gonzalez] and felt they could talk with me...The first time I met with Raúl Castro, he started out talking about my involvement with Elián González.

— Sen. Patrick Leahy, D-Vt.

Leahy met with Gross in the Cuban jail and also met with Cuban officials on several occasions to negotiate for his release.

But beyond Gross, Leahy and other U.S. officials engaged in clandestine talks with high-level Cuban authorities about broader issues, such as convicted Cuban spies jailed in the United States whom Cuba wanted released and returned, and the lifting of the trade embargo.

It wasn’t the first time Leahy and Cuban officials had discussed the embargo and engaged in the delicate waltz of trying to move toward a truce that factions in both countries vehemently opposed.

Some of the encounters were casual and even, on the surface at least, rather trivial.

Leahy was in Cuba in 1999 when the Baltimore Orioles played an exhibition game against a Cuban national game in Havana – the first time that a professional American team played on the island since the embargo started.

The Senator told FNL that he was moved when the mostly Cuban crowd in the stadium applauded after the playing of Cuban and American national anthems.

“It was striking,” he said. “People came over to shake hands with us after the game.”

Leahy held a press conference after the game at which he hailed the U.S.-Cuba exchange and pushed for an end to the embargo. Later he had dinner with then-President Fidel Castro.

The senator brought up the subject of the embargo with Castro, saying that it gave him a scapegoat for his economic and political failures.

Castro appeared to ignore his comment, he said, and said he preferred to speak with Leahy’s wife.

“He said, 'She’s a nurse, so she’s the humanitarian here,'” Leahy recalled.

After returning to the U.S., Leahy followed through on a promise to send Castro Vermont’s famous Ben and Jerry’s ice cream. Informed that the shipment would violate the U.S. embargo, Leahy instead sent Castro coolers of the ice cream packed in dry ice via a Cuban baseball team that visited the United States to play the Orioles.

Months later, he received a letter from Castro praising the ice cream. A bottle of Cuban rum accompanied the letter, which said, “This product from my state is very good.”

Fidel Castro sent him letters annually until he fell ill, around 2006.

It's that kind of exchange, Leahy maintains, that in many small doses over time melts away the iceberg between two nations.

Except when another ice storm hits – and freezes everything over again.

In 2000, that came by way of a skinny little boy who appeared off the coast of Florida floating in an inner tube – Elián González.

Elián, then 6 years old, survived a harrowing trip across shark-infested waters that his mother had decided to risk in order to flee Cuba. She drowned, and the United States and Cuba engaged in a pitched battle about whether to return Elián to his father, who was a Communist Party loyalist, or keep him in the U.S., where his mother's relatives fought to keep him.

Leahy played a pivotal role in pushing for Elián's return, taking part in talks with high-level Cuban officials and vowing to help send the boy back to his father so long as the Castro regime did not make a spectacle of having brought the U.S. to its knees.

"My involvement wasn't as public" as it was with Alan Gross, Leahy said. "My reaction was that you had a little boy who saw his mother drown. He has a father who loves him. What difference does it make whether he's in Cuba or anywhere else? He should be with his father. It was about family values."

Cuban officials, Leahy said, "made very clear that they appreciated what I did and felt they could talk with me."

The talks in the last year about the release of Alan Gross and restoring diplomatic relations took place in a variety of places, including Canada and even the Vatican, with Pope Francis acting as something of a broker between the nations.

"The first time I met with Raúl Castro," Leahy said, "he started out talking about my involvement with Elián González."

Obama has lifted many the travel and trade restrictions, but only Congress can end the embargo completely. Many Republicans – and some Democrats – have vowed not to let that happen, saying the Obama administration has made a deal with the devil.

Critics of the resumed diplomatic relations say the U.S. is making too many concessions to an oppressive regime that not only has not promised democratic reform, but has vowed that it will not change its core governing principles.

Raúl Castro has played Leahy and Obama, the critics' thinking goes, and all the tourists from around the world who have for years visited Cuba have not made a difference in the oppressive control the regime has over its people, and the repercussions that people know they will face if they push for freedom.

Leahy balks at the criticism.

“I’m not naïve enough to think that if I go down there, everything will change,” he said. “Will we agree on the dissidents? Of course not. Will we agree on the amount of debate [allowed] and free speech in Cuba? No.”

At its most basic level, he said, the change in relations between the two nations will begin as organic.

“When I started going down there I met with government officials, but I also met with the people in the street, in the stores,” Leahy said. “I was struck by the beauty of some of the buildings and the surroundings, but also I was struck by the poverty and the deprivation of the people, the reaction of people who thought they were being watched every minute.”

“I came to the conclusion that we had to have a change,” he said.

“I did not like the idea that the U.S. could tell me, as a U.S. citizen, what countries I could go to,” Leahy said. “It’s one thing if a country like North Korea says, ‘We won’t let you in.’ But American can go anywhere except for a small island nation just 90 miles away.”

Leahy recently co-sponsored bipartisan legislation in the Senate lifting travel restrictions to Cuba.

It is not likely to get far, especially if colleagues like Cuban-Americans Sen. Marco Rubio (R-Fla.), Sen. Ted Cruz (R-Texas) and Sen. Bob Menendez (D-N.J.) have anything to say about it.

They have vowed to block any effort to confirm whomever Obama nominates as U.S. ambassador to Cuba.

“That would be cutting off our nose to spite our face,” Leady said. “We have an ambassador in China, and we disagree with them on a lot of things. We have an ambassador in Middle Eastern countries with which we have disagreements.”

Leahy believes it is just a matter of time before the embargo is lifted.

Public opinion, he says, is leaning toward favoring an end to the embargo, and toward full diplomatic relations with Cuba.