With a cordial evening handshake, President Obama and Cuban President Raul Castro moved Friday toward a historic meeting on the sidelines of the Summit of the Americas.

The meeting between the U.S. and Cuba Saturday with the leadership of the Western Hemisphere gathered around them could signal substantial progress. Both sides are still working through issues that would lead to the opening of embassies in Washington and Havana, the first state of a new diplomatic relationship.

The first clues of an improved relationship between Obama and Castro came Friday evening as they both arrived at a Panama City convention center for the summit’s opening ceremonies. A reporter for a Venezuelan TV network posted video online showing the two greeting each other comfortably with multiple handshakes and extended small talk, while U.N. Secretary-General Ban Ki-moon and Cuban Foreign Minister Bruno Rodriguez looked on.

A White House official said the interaction was informal and there was not a substantive conversation between the two leaders. Both men are expected to speak further Saturday – the first extended conversation between leaders of the U.S. and Cuba in more than 50 years.

Obama, speaking to a meeting of civil society groups, cast the move to end hostile relations as a triumph for the Cuban people.

"As the United States begins a new chapter in our relationship with Cuba, we hope it will create an environment that improves the lives of the Cuban people," he told the gathering, which included Cuban dissidents. "Not because it is imposed by us, the United States, but through the talent and ingenuity and aspirations, and the conversations among Cubans from all walks of life so they can decide what the best course is for their prosperity."

The White House was coy over the status of the State Department's recommendation to remove Cuba from the U.S. list of state sponsors of terror. Removal is a top issue with Castro because it would not only eliminate Cuba's status as a pariah, but also ease Cuba's ability to conduct simple financial transactions.

There is still is speculation that an announcement over the terror list could come soon.

The U.S.-Cuban outreach entered a new, accelerated stage in recent days, with Obama speaking with Castro by phone Wednesday and U.S. Secretary of State John Kerry holding a lengthy meeting with Cuban Foreign Minister Bruno Rodriguez late Thursday.

The Cuban foreign ministry issued a brief account of the Kerry-Rodriguez meeting, saying they met for nearly three hours in a "respectful and constructive atmosphere."

It was the highest-level, face-to-face contact between officials from the two countries since the Dec. 17 announcement that Washington and Havana would move to restore diplomatic relations that were severed in 1961.

Even as Washington talked up the historic shift toward Cuba, leftist leaders in Latin America took shots at the U.S. in solidarity with Venezuela.

Barely off the plane, President Nicolas Maduro of Venezuela laid a wreath at a monument to victims of the U.S. invasion of Panama in 1989. To shouts of "Maduro, stick it to the Yankee," he vowed to personally ask Obama to apologize to Panama and compensate victims of what he called a "massacre" that left more than 500 people dead during the invasion.

"Never again a U.S. invasion in Latin America," Maduro said.

President Evo Morales of Bolivia said he backs Maduro's drive to end U.S. intervention in the region.

The flurry of diplomacy around the summit was recognition of the historic nature of the new relationship intended to end five decades of American presidents either isolating or working to overthrow Fidel Castro's government. Officials hoped to make the most of the exchange between the two men.

Obama still plans to meet with 15 Latin American activists, including two Cubans who have challenged the Cuban government. The White House identified the Cubans as Laritza Diversent, a human rights lawyer and independent journalist, and Manuel Cuesta Morua, a leader of a centrist opposition group. A large contingent of pro-Castro Cubans who were supposed to participate in a larger civil society forum left shortly before Obama spoke to protest the inclusion of Cuban dissidents.

Removing Cuba from the list of state sponsors of terror would be a major milestone and likely generate controversy in the U.S., given the political repercussions of any Cuba opening. The sensitivities over Cuba are especially acute in Florida, a key presidential battleground, and they are likely to ignite vigorous debate among the presidential candidates.

Congress would have 45 days from the day Obama removes Cuba from the list to issue a resolution of disapproval, but the margin of passage would have to be veto-proof.

Rep. Jeff Duncan, the chairman of a House panel on the Western Hemisphere, criticized taking Cuba off the list. And Sen. Lindsey Graham, who is among Republicans considering a presidential bid, decried the expected Obama-Castro meeting and called the Cuban leader an "entrenched dictator."

The Associated Press contributed to this report