Australia's prime minister has told an Asia-Pacific regional conference that terrorism is a modern form of treason and the region's greatest security challenge.

Prime Minister Tony Abbott gave an opening address on Thursday to delegates from about 30 countries attending a conference in Sydney on countering terrorism propaganda and enhancing regional cooperation.

Abbott says the work of the conference was to share experiences "so that all of us can learn to be more effective in grappling with the biggest security challenge of our time."

He says Australia is "looking at what can be done to deal with Australian citizens who have betrayed our country by fighting with terrorists, this modern form of treason."