The head of the British Bankers' Association says financial firms are planning to leave London within weeks because of uncertainty about the U.K.'s exit from the European Union.

Anthony Browne says banks fear EU politicians will erect trade barriers with Britain in a bid to undermine the City, London's financial district. They also fear U.K.-based financial firms will lose the right to conduct business across the bloc.

Browne wrote in Sunday's Observer newspaper that bankers' "hands are quivering over the relocate button." He said "many smaller banks plan to start relocations before Christmas; bigger banks are expected to start in the first quarter of next year."

Browne said tariffs would hurt both Britain and the EU, but that economic arguments might lose out amid hardening political rhetoric from both sides.