"This harassment has included police raids, arbitrary arrests, ill-treatment and extortion, as well as destruction of informal housing," the London-based human rights group said in a statement.
"The requirements under the 'FIFA by-laws' which create extensive exclusion zones for informal economic activity are seen as particularly prejudicial," Amnesty said.
Police spokesman Vish Naidoo defended the action by police and local authorities, saying Amnesty was trying to link the World Cup to a perennial problem in South Africa, where millions live in grinding poverty and shanty towns.
"Just because the World Cup is happening, must we ignore laws and by-laws that are being broken?" he said.
Amnesty also said the deployment of huge numbers of police at World Cup-related sites would mean fewer officers in poorer neighborhoods, where the largest part of South Africa's notoriously high levels of crime occur.
It added that it was worried about police getting trigger-happy in their zeal to protect the 350,000 foreign visitors who are expected for the month-long tournament, which kicks off on June 11.
(Reporting by Ed Cropley)
(Editing by Jon Bramley)
Receive your weekly recap of all the happenings around the world of sports.