The men, who've been released from police custody, range from 18 to 22, and face second-degree rape charges. Allen George, a commander with the Seventh Judicial Circuit Major Crimes Task Force, said they are from Anniston, Jacksonville, Centre and Guntersville, Ala.
Their names were not released. It was unclear if they were students at the university.
The arrests stem from reports of six statutory rapes earlier this year, WAFF-TV reported.
George said he became aware of other incidents that took place on campus after the Jacksonville Police Department asked him to investigate one of the complaints.
The assaults are said to have occurred at halls and a parking garage. Authorities are not looking for a serial rapist and students are not in danger, George said.
In a statement, the school said it was aware of the investigation, which is ongoing.
"The university is fully cooperating," the statement read. "No more statements will be made at this time due to privacy laws. There is no risk to our students."
The Associated Press contributed to this report.