The team fired Caldwell on Tuesday, the second major decision in what is expected to be a long offseason shake-up. Team owner Jim Irsay fired team vice chairman Bill Polian and his son, Chris, the team's general manager, the day after the season ended.
It took a little longer for the decision on Caldwell, who had just finished his third and worst season as head coach of the Colts, who stumbled to a 2-14 finish without injured quarterback Peyton Manning. Caldwell ends his Colts' tenure 26-22 overall.
Caldwell won his first 14 games, an NFL record for a rookie head coach, and became only the fifth first-year coach to take his team to the Super Bowl.
But with fans complaining about game management and clamoring for a change since midseason, Irsay didn't have much choice.
With Manning, the Colts won a league-record 115 regular-season games over the previous decade, tying the league mark for most consecutive playoff appearances (nine), winning two AFC titles and one Super Bowl trophy, the Colts lost their first 13 games in 2011, then won twice in five days and nearly lost the No. 1 draft pick, too.
Without Manning, Indy started 0-8 and was the heavy favorite to win the Andrew Luck sweepstakes at midseason. Caldwell's team lost the next five games, too, before finally winning two straight to avoid becoming the second 0-16 team in league history.
A season-ending loss at Jacksonville, officially gave the Colts the top pick, which is expected to be used on Luck.
Players never gave up on Caldwell and many cited their preference to keep playing for him next season. Manning was one of Caldwell's supporters, calling the coach that helped him win a record-setting four MVP Awards a "friend."
But the disastrous 2011 season was too much for Caldwell to overcome after winning AFC South titles in each of his first two seasons in Indy.
After overhauling the front office, Irsay last week hired 39-year-old Ryan Grigson as his new general manager, then wanted to wait until Grigson had time to evaluate Caldwell's performance.
The decision came Tuesday, setting off the second major search of the month.