Woods, whose image was tarnished by a sex scandal, was the top earner for the eighth consecutive year but his winnings and endorsements dropped 31 percent in the 2011 survey to $62.3 million while Mickelson remained steady at $61.2 million.
Federer kept his top position among non-U.S. athletes despite slipping 14.5 percent to $52.8 million, third overall, while WBO welterweight champion Pacquiao shot 38 percent higher to $52.5 million for fourth overall.
Spanish Formula One driver Fernando Alonso ($45 million), Barcelona's Lionel Messi ($43.8 million) and Real Madrid's Cristiano Ronaldo ($38.8 million) rounded out the top five on the international list.
The magazine based its figures on salary, winnings, bonuses, endorsements and appearance fees.
(Reporting by Gene Cherry in Raleigh; Editing by Frank Pingue)