President Obama said Sunday he's not concerned with the politics of shaking hands with Venezuelan President Hugo Chavez and is more interested in expanding a policy he described during his presidential campaign of extending an open hand to nations hostile to the U.S.

Obama received a book from the Venezuelan president on Saturday after greeting him on Friday evening during the weekend Summit of the Americas. Chavez told the Obama administration that he would like to send an ambassador to the U.S. in exchange for an ambassador in Venezuela. The U.S. suspended diplomatic ties last September.

U.S. officials responded that they need to see more from Chavez before moving forward but were pleased by the sentiment. 

Obama did not make note that the book offered by Chavez, "Open Veins of Latin America: Five Centuries of the Pillage of a Continent," by Uruguayan journalist Eduardo Galeano, blames foreign interests like the United States for exploiting Latin America for centuries.

"It was a nice gesture to give me a book. I am a reader," Obama said during a solo press conference at the end of a the summit held in the Caribbean nation of Trinidad and Tobago

"We had this debate throughout the campaign. I mean the whole notion was somehow that if we showed courtesy or opened up dialogue with governments that had previously been hostile to us that would show weakness. And the American people didn't buy it and there's a reason they didn't buy it -- because it didn't make sense," Obama said.

Obama credited Chavez, who has been slowly instituting constitutional changes that will allow him to become president for life in his country, with successfully stealing the limelight during the weekend summit. But he said meeting with the Venezuelan leader, who has made "inflammatory" anti-American comments, isn't going to break the United States.

"There have been instances where we've seen Venezuela interfere with some of the countries that surround Venezuela in ways that cause concern. On the other hand, Venezuela is a country whose defense budget is one-six hundredth of the United States. They own Citgo, the oil company. It's unlikely that as a consequence of me shaking hands or that having a polite conversation with Mr. Chavez that we endanger the strategic interests of the United States," Obama said.

Back in Washington, D.C., former CIA Director Michael Hayden was cautious about any changes in U.S. relations with Venezuela.

"Here's a case where I would watch for behavior, not for rhetoric, and the behavior of President Chavez over the past years has been downright horrendous -- both internationally and with regard to what he's done internally inside Venezuela," Hayden told "FOX News Sunday."

Indeed, independent Venezuelan reporters who normally cover Chavez said the president has a tendency to act "statesmanlike" when the world is watching, but has traditionally reverted back to his normal defiant, socialist and overtly-bombastic rhetoric when describing the U.S. after his return home. Chavez's mercurial rhetoric also has been described as a wildly swinging pendulum, that ranges from one extreme to the other often within the same speech.

On Saturday evening, Jeffrey Davidow, a special assistant to the president who spearheaded logistics for Obama's attendance at the summit, told FOX News that Chavez's bid for publicity should not be read as a breakthrough or even a hint of one in strained U.S.-Venezuelan relations.

"I think the Venezuelan leader is always wanting to make a splash, that's what he does and he does it pretty well. He's captured the interest of the press," Davidow said. "I think the fact that our president shook his hand and smiled doesn't constitute a new relationship."

Obama spoke Sunday after skipping an official photo also ignored by Bolivian President Evo Morales, who accused the United States of somehow being involved in an attempted assassination. Obama rejected that accusation and said the United States has nothing to do with the alleged plot.

"I am absolutely opposed and condemn any efforts at violent overthrows of democratically elected governments," Obama said. "That is not the policy of the American government, that is not how the American people expect their government" to act.

"The United States has a history in this region that is not always appreciated by the perspective of some," Obama added . 

Obama tried to put a positive spin on the weekend summit, where he was often overshadowed by criticism by Morales and Nicaragua's Daniel Ortega as well as "overtures" by Chavez and Cuba's Raul Castro.

During his attendance at the summit, Obama endured a 50-minute diatribe from Ortega that lashed out at a century of what he called terroristic U.S. aggression in Central America and included a rambling denunciation of the U.S.-imposed isolation of Cuba's communist government.

Obama sat mostly unmoved during the speech but at times jotted notes. Asked afterward what he thought about Ortega's speech, he answered: "It was 50 minutes long. That's what I thought."

Obama did receive positive signals from Cuba's communist leader last week. Castro said that he was open to talks on "everything." The remarks followed Obama's lifting restrictions on travel and remittances to Cuba by Cuban Americans. 

Cuba is not a participant in the Summit of the Americas because it is not a democracy.

Speaking Sunday, Obama said that he supported a change in U.S. policy toward Cuba, noting that he doesn't want to continue a policy of isolation against Cuba created "before I was born."

He said changes won't take place overnight, an he wants to see Cuba release political prisoners, afford greater freedoms and stop skimming money sent from families in the United States to relatives in Cuba.

Back in Washington, Sen. Lindsey Graham, R-S.C., told "FOX News Sunday" that Cuba needs to make the next move. 

"Release the prisoners and we'll talk to you. ... Put up or shut up," Graham said.

"I think we're taking the right steps, and I think the ball is now clearly in Cuba's court," added Sen. Claire McCaskill, D-Mo., who appeared with Graham. "They need to respond and say what they're willing to do."

Obama said the American hemisphere is ready to launch "a new era of partnership" in which progress will be measured by actions.

"The test for all of us is not simply words, but also deeds," Obama said at the start of a press conference in which he highlighted initiatives developed during the 34-nation talks. 

He added that the potential for progress will be proved in part through a demonstration of respect for democracy and human rights.

Obama stressed that the U.S. is a willing partner, "inclined to listen and not just talk," in trying to advance national interests.

"We recognize that other countries have good ideas, too, and we want to hear them," he said, adding that the fact that an idea comes "from a small country, like Costa Rica," should not diminish its potential benefit.

FOX News' Major Garrett contributed to this report.