Arlen Specter (search) has fought off challenges to his Pennsylvania Senate seat for nearly 24 years, and despite claims by some opponents that this is definitely the year for change, recent polls and Specter's supporters suggest the four-term senator isn't going anywhere.

"There is a group of people — more so in Washington — who have never liked Sen. Specter and every year they come up with another reason why," said Chris Nicholas, Specter campaign spokesman.

"But he is Pennsylvania's only four-term senator and in 2004, and he will be their only fifth-term senator," Nicholas added.

Conservatives don't buy that Specter is guaranteed re-election — in fact, they smell vulnerability. Republican Rep. Pat Toomey (search) is mounting an aggressive, well-funded primary against Specter, and is using Specter's tendency to side with senate Democrats on abortion, tax and tort issues to call for his ouster.

"He's never had a primary opponent who had the ability to run a credible race," Toomey told Foxnews.com. "The highest funded primary opponent raised a half-a-million dollars. I've already raised $2.5 million and there's a lot more in the pipeline."

Specter has had little trouble fighting off Republican challengers since his first primary in 1980. According to the most recent poll by the Morning Call/Muhlenberg College Institute of Public Opinion, that streak is continuing. Specter leads Toomey 52 to 25 percent among registered Republican voters.

But Specter has had his fair share of tough challenges, barely surviving the 1992 election he won 49 percent to 46 percent. In that race, feminist  groups targeted him for fiercely supporting the 1991 confirmation of conservative U.S. Supreme Court Justice Clarence Thomas (search).

While the left has attacked Specter for being too conservative on women's issues, the right now complains he's too liberal on those matters.

"He's the poster child for NOW (National Organization for Women) and NARAL (National Abortion and Reproductive Rights Action League)," said Connie Mackey of the Family Research Council (search).

Mackey added that Specter is beholden to liberal interests on abortion, stem cell research — of which he is a supporter — and other social issues that are likely to be debated by the courts.

Those issues  — many likely to face future court action — are of grave concern to conservatives who not only remember Specter's vote against conservative judge Robert Bork (search), who lost his confirmation to the Supreme Court in 1987, but also fear he will be a liability as chairman of the Senate Judiciary Committee.

If Specter wins re-election and Republicans maintain the majority in the Senate, he is expected to take the seat currently held by Sen. Orrin Hatch, whose term as chairman expires at the end of 2004.

Many conservatives say they expect no help from Specter on the ongoing Democratic filibuster of President Bush's judicial nominees or on a controversial constitutional amendment that would  define marriage as a union only between a man and a woman.

"I think most of the family groups understand he has not been much of a friend," said Richard Lessner, executive director of the American Conservative Union (search), whose chairman David Keene nonetheless supports Specter.

Lessner pointed out that Specter has a lifetime average rating of 47 from the ACU — that's less than the 60-point rating the American Civil Liberties Union gave the senator in 2002. Keene has said he will support Specter because of Specter's role in the Thomas confirmation, his support for a flat tax and Keene's personal regard for him.

Supporters of the 73-year-old senator also note that the White House has publicly shown support for Specter's re-election as well as his expected judiciary chairmanship.

"If the White House has no problem with this, why should these conservatives?" asked Sarah Chamberlain Resnick, executive director of the Republican Mainstreet Partnership (search), a coalition of moderate GOP members of Congress. Resnick said her group expects to put more than $1 million — or as much as it takes — into defending Specter in 2004.

Sources on Capitol Hill say that several Republican members of the judiciary panel do not agree with Specter’s more moderate approach, but they are unlikely to break ranks against him when the time comes to confirm him to the chairmanship.

One unnamed congressional aide predicted solidarity with Specter among Republicans on Capitol Hill.

"He's a team player and he works hard," said the aide. "And when he's on your side and he's on your team, he's a great asset."

But Steve Moore, president of the fiscally conservative Club for Growth (search), said his group is supporting Toomey, who has long been considered a fiscal conservative, and is putting more money behind him than any other race this year. The Club for Growth has already spent $650,000 on pro-Toomey ads and plans to spend upwards of $1.5 million before the April 27 primary.

"We view this as the most important political race outside of the presidential race," Moore said. "We've got a potential superstar in Pat Toomey, against one of the most liberal Republicans in the entire Congress."

Nicholas has accused Toomey, who has vowed to keep his three-term limits pledge regardless of the Senate primary outcome, of only recently adopting a strong pro-life, socially conservative stance for the benefit of distancing himself more sharply from Specter.

"He's been flip-flopping on his flip-flops," Nicholas said.

"They've been mischaracterizing my record from the beginning," Toomey responded. "It's always been consistently conservative across the board."

Specter's supporters say Pennsylvanians aren't looking for a senator like Toomey, and add that Specter has been able to secure millions of dollars for the state from special projects. He also carries a lot of clout for the state.

Stuart Rothenberg, editor of the Rothenberg Political Report (search), said he thinks Toomey is a "credible candidate," who benefits from the shifting away of the GOP power base from Specter's home district of Philadelphia to more rural areas like Toomey's district of Lehigh Valley. But, Toomey is still a long shot, Rothenberg said.

"With the money and political dice he has, and the appeal he has, he should get at least 40 percent," Rothenberg said. "I'm not hearing anyone who thinks Toomey is going to win."