NFL

Ravens' Harbaugh not thinking about staff changes

September 20, 2015; Oakland, CA, USA; Baltimore Ravens defensive coordinator Dean Pees before the game against the Oakland Raiders at O.co Coliseum. The Raiders defeated the Ravens 37-33. Mandatory Credit: Kyle Terada-USA TODAY Sports

September 20, 2015; Oakland, CA, USA; Baltimore Ravens defensive coordinator Dean Pees before the game against the Oakland Raiders at O.co Coliseum. The Raiders defeated the Ravens 37-33. Mandatory Credit: Kyle Terada-USA TODAY Sports

Baltimore Ravens head coach John Harbaugh hears the grumblings about his team's defensive failures, but he's not planning on shaking up his coaching staff.

The Ravens' defense ranks 24th in the league after surrendering 457 passing yards to Josh McCown in a 33-30 loss to the Cleveland Browns. The unit has given up fourth-quarter leads in three of its four losses, leading many fans to point the finger at defensive coordinator Dean Pees.

"The coaches are doing a good job," Harbaugh said via the team's official website. "We've just got to collectively find a way to finish games and get the job done. It's tough challenges that we're facing, but we've got just the men for the job right here."

More from FoxSports

For the second straight year, Pees has had to overcome a number of injuries to his defense. He lost star pass rusher Terrell Suggs for the season in Week 1, and Elvis Dumervil was forced from Sunday's game in the first half with a groin injury. The secondary has taken a beating just as it did in 2014. Lardarius Webb (thigh), Will Davis (torn ACL), and Kyle Arrington (concussion) all left the loss to the Browns with injuries.

Harbaugh refused to use the injuries as an excuse, saying the coaches have to do a better job of elevating the backups' play, just as they did last year when the Ravens advanced to the divisional round of the playoffs.

"I don't doubt our character," Harbaugh said. "We don't need to play harder, we need to play better. We don't need to coach harder, we need to coach better. We've got to find a way to make the difference. It's making plays, it's calling plays that give guys a chance to make plays in critical situations to get you over the hump."

Baltimore is 1-4 for the first time in franchise history. The next test will come against a team with the same record in San Francisco on Sunday.