The Twins announced the move Monday, saying Terry Ryan has agreed to assume the role of interim general manager.
Smith had been general manager since September 13, 2007, when Ryan announced his resignation after 13 seasons.
"We are fortunate and grateful for Terry Ryan's willingness to step in as interim general manager," Twins owner and chief executive Jim Pohlad said in a statement.
While praising Smith's desire to win and lauding his commitment and talent, Pohlad said he differed with the outgoing GM "in the scope and approach that was required" for the Twins to get better.
Smith has been a part of the organization since 1986 and Pohlad said the team intends to discuss an ongoing role for him within the club.
The Twins captured back-to-back AL Central titles under Smith after losing a sudden-death tiebreaker in 2008 to the Chicago White Sox, but they were just 63-99 this past season to finish with the worst record in the AL.
Smith is credited with a significant role in opening the team's new stadium, Target Field, last season.
"Working for the Minnesota Twins for the past 26 seasons has been the greatest professional privilege I could have ever imagined," Smith said in a statement.
Ryan was originally hired as GM in 1994 and helped the Twins win four division titles from 2002-06. He has spent more than 28 years with the organization and will lead the search for a permanent replacement.