While no meeting has been formally planned between Cuban President Raul Castro and President Barack Obama at this weekend’s Summit of the Americas in Panama, White House officials say spontaneous discussions – even perhaps a substantive meeting – may occur.

It will be the first time that Cuba, which the United States previously has banned from the summit, will participate in the event. And it would be the first time that Obama and Castro exchange words in person since they greeted each other at the 2013 memorial service in South Africa for former President Nelson Mandela.

“I’m sure that President Obama will be interacting with President Castro at the summit events as leaders gather on the margins of those events,” said Ben Rhodes, deputy national security adviser in a telephonic press briefing on Tuesday.

Many Latin American countries had said they would boycott the summit, which is sponsored by the Organization of American States, if Cuba continued to be banned from it.

The White House officials admitted the United States has long been dogged by criticism because of this.

“Every time we showed up at the Summit of the Americas, the question was why Cuba wasn’t invited,” said Rhodes, a key player in the talks to restore relations with Cuba.

“In the hemisphere, it made no sense that the United States consistently made the decision to isolate ourselves from the rest of the Americas because we were clinging to a [Cuba] policy that wasn’t working,” Rhodes said. “We were not joined in imposing sanctions [against Cuba] by any other country because no other country agreed with our approach.”

Rhodes said high-level discussions are taking place between the U.S. and Cuban governments.

One priority the Cuban government has publicly pushed since Castro and Obama announced in December a restoration of diplomatic relations has been its removal from the U.S. State Department’s terrorism watch list, which also includes Syria, Iran and Sudan.

Rhodes said the State Department is nearly finished with its review – ordered by Obama after the diplomatic relations agreement in December – of whether Cuba should be removed from the list, and that Secretary of State John Kerry will forward the agency’s recommendation to Obama.

The process requires that Cuba vow not to have links with terrorism. Obama would then send Congress an initiative to remove Cuba from the list.

But that is likely to run up against strong opposition, given that the House and Senate both have a Republican majority that generally objects to restoring ties with Cuba while Raul Castro is in power and while there are no tangible steps toward democratic reforms.

Republican leaders in Congress already have said they will not allow for the confirmation of a U.S. ambassador to Cuba.

Cuba was put on the list in 1982.

The nations on the list, the State Department website says, “have repeatedly provided support for acts of international terrorism.”

Being put on the list results in actions by the U.S. such as “restrictions on U.S. foreign assistance; a ban on defense exports and sales; certain controls over exports of dual use items; and miscellaneous financial and other restrictions,” according to the State Department.

The State Department says that Cuba has given refuge to members of Basque Fatherland and Liberty (ETA) and the Revolutionary Armed Forces of Colombia (FARC).

The agency also noted that "the Cuban government continued to harbor fugitives wanted in the United States. The Cuban government also provided support such as housing, food ration books, and medical care for these individuals."

Members of Congress such as Rep. Ileana Ros-Lehtinen, a Florida Republican, and Senators Marco Rubio, also a Florida Republican, and Robert Menendez, a New Jersey Democrat, have said Cuba's relations with the governments of countries such as Iran and Syria require that it continue to be on the terrorism watch list.

Rhodes said he hoped that the steps toward restoring diplomatic relations with Cuba would lead to “having more constructive engagement in the hemisphere.”

The officials said among topics they’ll focus on at the summit are hemispheric efforts to promote global economic competitiveness, broader college student exchanges between the United States and Latin America, migration, U.S. immigration reform, clean energy programs, expanding broadband Internet.