It sounds cruel, but let's face it: Except for the occasional highlight like "Vanilla Sky" or "My Brave Face," for the past 20 years, Paul McCartney (search)'s catalogue has been pretty barren.

So the former Beatle wanted to make his new solo album, "Chaos and Creation in the Backyard" (search), stand out.

"I decided to lay it on the line for myself and challenge myself and say, 'You're going to make a good album here.' It was a good motivator," he told The Associated Press.

Time magazine breathlessly declared "Chaos" to be McCartney's first album that matters since the Beatles (search) broke up 35 years ago. But it's simply unlike anything he's done before, a quiet disc with complicated emotional shadings — the album that generations of critics who derided his sunny, silly love songs have been asking him to make.

He'll never be mistaken for Nine Inch Nails. But the heartache of "Too Much Rain" and smoldering anger of "Riding to Vanity Fair" are unusual for McCartney. When the 63-year-old struggles for the notes in the "Blackbird" successor "Jenny Wren," he even sounds fragile.

"Even though I'm essentially an optimist, an enthusiast, like anyone else I have down moments in my life," he said. "You just can't help it. Life throws them at you.

"In the past I may have written tongue-in-cheek, like `Maxwell's Silver Hammer,' and dealt with matters of fate in a kind of comical, parody manner. It just so happens in this batch of songs I would look at these subjects and thought it was good for writing. If it's good enough to take to your psychiatrist, it's good enough to make a song of."

McCartney also was pushed by the blunt Nigel Godrich, a producer known for his work with Radiohead and Beck.

His method was to force the music legend out of his comfort zone. McCartney brought his touring band in to record; after two weeks Godrich dispatched them. Much like he did with his very first solo album, McCartney played virtually every instrument himself — on "Friends to Go" alone, he's credited on the grand piano, acoustic/bass/electric guitars, harpsichord, drums, tambourine, flugelhorn, melodica and shakers.

Producer and artist particularly clashed on "Riding to Vanity Fair," which McCartney brought in as a fast song and Godrich kept trying to slow down.

"There were one or two moments on the album when I had to think to myself, `You know, I could just fire this guy,'" McCartney said.

Widening his emotional palette doesn't necessarily belittle his optimism, McCartney said. He's not disowning anything. He spoke of talking with Bruce Springsteen a few years ago at a Rock and Roll Hall of Fame induction where the Boss admitted he didn't really get "Silly Love Songs" at the time it came out in 1976. He feels differently now that he's a family man.

"It took a little nerve to hold on, knowing that people were going to take a cheap shot," McCartney said.

Obviously he never set out to log many years of lackluster recordings.

"You're not really aware of that," he said. "You can maybe get a little complacent, or you're not hitting a good patch, or you can think it's great and it isn't. There are a multitude of reasons why."

For a man who seemingly tumbled out of bed every morning of his youth with a brilliant melody, the struggles were painful to listen to. Think "Freedom." What was once effortless seemed forced.

Writing songs isn't necessarily harder for him as he gets older, McCartney said. And for whatever reasons — time, a happy remarriage and new fatherhood — he feels he's writing better than he has in a long time.

"I still have this deep love for melody in particular and writing songs," he said. "It isn't any more difficult. Obviously what made it easier then was writing with John (Lennon). He was such a great collaborator. The two of us were on fire every time we sat down to write.

"If he was stuck, I knew that I could help him out and vice versa. We normally sat down for three hours and bingo, a pretty good song came out. We never had a dry session. Every time we sat down, we came out with a song."

That happened up to the end; Lennon even asked for advice on "The Ballad of John and Yoko," he said. "We're not stupid," McCartney said. "We knew a good thing."

Yet it put in place the essential dilemma of his solo years. McCartney seems to intrinsically understand the value of a strong collaborator, but what can compare when you've had a partnership for the ages?

He enjoyed, for example, a brief songwriting collaboration with Elvis Costello that produced some good music ("My Brave Face"). But "you do something like that and it makes it even more obvious that there's no replacing John for me and no replacing me for him."

McCartney spoke by telephone from a car driving to band rehearsals in Miami for his American tour. He's long past the period where he felt he had to prove himself post-Beatles so, twistedly, he avoided the band's work in concert.

Now the whole catalogue is up for grabs, and it's easy to find songs he's never played live before — like the voice-shredding "Helter Skelter," which he brought out for the Live 8 concert.

One last query as the car pulled up: Has he ever thought over the years, I've put a pretty good songbook together, maybe it's just time to let it be?

"Pardon the pun," he said. "The trouble is, I like it too much. If I was asked to retire tomorrow, if I was forcibly removed from my contract, I'd just do it for fun."