Mfume announced his candidacy at a news conference in Baltimore.
"It is with great pride and deep humility that I announce to you today my candidacy for the Senate of the United States," Mfume said.
"I can't be bought. I won't be intimidated. I don't know how to quit," he said as his supporters applauded. Mfume also issued a three-page statement.
Mfume, 56, left the Baltimore-based National Association for the Advancement of Colored People (search) last year after serving nine years as its chief. Before that, he served five terms as a U.S. congressman from Maryland.
Sarbanes, a 72-year-old Democrat, announced Friday that he would not seek a sixth term, noting that he would be 80 when the term ended.
"It was just the right time," Sarbanes said. "We think we've served long and well and honorably, and we're very comfortable with this decision."
Maryland Democratic Party spokesman Derek Walker said he expects the party will field many candidates for the office and called Mfume's announcement "step one in what will be an embarrassment of wealth" for the party next year.
Other Democratic names have been offered as possible successors to Sarbanes, including House Minority Whip Steny Hoyer (search), U.S. Rep. Albert Wynn (search) and Montgomery County Executive Doug Duncan (search).
The heavily Democratic state voted for John Kerry in the 2004 presidential election and has two Democratic senators. Six of its eight representatives are Democrats.
However, Gov. Robert Ehrlich is a Republican and his African-American lieutenant, Michael Steele, is considered a rising star in both the state and the GOP.
Mfume has been credited with helping restore the NAACP's image. When he first took the helm, the historic civil rights organization was grappling with an embarrassing sex scandal involving its outgoing president Benjamin Chavis, bitter internal strife and a crippling $3.2 million budget deficit.
When Mfume left, the organization had had a budget surplus for eight consecutive years and enjoyed an growing endowment fund. Membership was a half million spread across 48 states.
Mfume, born Frizzell Gray, is a Baltimore native whose adopted West African name translates to "conquering son of kings." He began his career as a dashiki-clad radio talk-show host and political activist in the 1970s.
He transformed himself into one of the nation's foremost civil rights leaders with a reputation for bridge-building, and started his political career in the Baltimore City Council before moving to Congress, where he led the Congressional Black Caucus.
The Associated Press contributed to this report.