The Saints were beaten 41-36 on Saturday by the Seattle Seahawks, a team that won the NFC West with a 7-9 record and was widely considered to be the weakest team in the playoffs.
New Orleans, Super Bowl winners a year ago, were without injured running backs Pierre Thomas and Chris Ivory and were missing Reggie Bush for the middle chunk of the season.
The injury bug continued Saturday.
"The runners handled it pretty well, it wasn't until the second half that we got hampered with Reggie (Bush) going down and Julius (Jones) going down," he said.
"It has been one of those seasons we had a number of injuries but that wasn't the difference today,"
"In the end we just weren't good enough in a number of areas, I think it was obvious to see.
"We didn't do a good enough job in the kicking game overall, our return game really struggled, we gave up too many big plays defensively," he said.
Payton was also critical of the way his offense stalled during a spell when the Seahawks were scoring with ease.
"It is disappointing, it starts with us, myself and the rest of the coaches," he said. "We had effort but it was not enough to win the game. It's disappointing to get into the post-season and finish with a loss to start.
"There will be a lot of things we will look at. We didn't play to our best and Seattle made the big plays when they needed to.
"There were some momentum swings. We felt we had it early and then it shifted back to them. We just weren't able to make the plays you need to win in the playoffs and beat a team on the road like that."
(Writing by Simon Evans in Miami; Editing by Steve Ginsburg)
Receive your weekly recap of all the happenings around the world of sports.