How Can You Tell When Polls Are Biased?

This is a partial transcript of "Special Report With Brit Hume," May 17, 2005, that has been edited for clarity.

Watch "Special Report With Brit Hume" weeknights at 6 p.m. ET

BRIT HUME, HOST: News reports keep telling us that public opinion is running against the Republicans in their campaign to end filibusters against judicial candidates. And two recent polls, at least, would appear to bear that out.

But do they really do that? For a closer look at the recent polls, I’m joined by Daron Shaw, professor of political science at the University of Texas, part of whose work is the study of polling. Daron Shaw is also a member of the FOX News election night decision team.

Daron, welcome. Glad to see you.

DARON SHAW, UNIVERSITY OF TEXAS: Good to see you, too, Brit.

HUME: Let me walk through just a handful of recent polls. First, a couple of weeks back, there was one in The Washington Post, which was the lead story in the paper. It made headlines about how the administration’s position, or the Republicans’ position, was not supported by the public.

"Question: Would you support or oppose changing Senate rules to make it easier for the Republicans to confirm Bush’s judicial nominees?" Sixty- six to twenty-six percent of people said they would oppose that.

Now, the Pew poll, only Monday — that’s the Pew Research Center — came out said — had a poll that asked the question, "Republicans won the last election so President Bush should be able to appoint anyone he wants to the federal courts if a majority of senators agree." Well, 53 percent of the people agreed with that.

But when the question was put another way, "The minority party ought to be able to block some of the judges they feel strongly about because judges are appointed to the federal courts for life terms," that did even better; 62 percent agreed with that proposition, which would seam to be the opposite, and 30 percent disagreed.

And finally, we have a poll from the somewhat-controversial Rasmussen polling system, which uses automated phone calls, but in large numbers. And he says — his poll question was, "Senate rules should be changed so that a vote must be taken on every person the president nominates to become a judge," and that wins 57-25 percent.

Now, let’s go back through these, Daron. What about this Washington Post poll, "Would you support or oppose changing Senate rules" — this is going back now to the Post poll — "Would you support or oppose changing Senate rules to make it easier for the Republicans to confirm Bush’s judicial nominees?"

Is that a good, well-designed question? If so, why? If not, why not?

SHAW: All right. Well, let’s sort of start with the premise that people aren’t very interested, aren’t very involved, aren’t very engaged in politics, which I think everybody who studies public opinion would concede. That’s one of the first things we teach in Politics 101.

Given that, you know, every pollster sort of confronts a simple question when you start out. What sorts of information do you want to introduce in your polling frame? Because that information is going to be used by people to construct opinions on the spot, when they respond to a pollster.

HUME: So very often what you’re doing is you’re asking people what their opinion is on something, and until that moment, they might not have had an opinion about it.

SHAW: Absolutely. And people are nice, so they’re going to try to grab whatever information they can off the top of their heads or from the question frame to construct an opinion to, you know, satisfy the pollster and do what they think is the right thing to do.

The Washington Post poll, I think, has two kinds of conceptual problems. And this is not meant as a criticism of the Post generally. But in this particular instance, first of all, they have this sort of partisan invocation. Right, so you know if you invoke President Bush and the Republicans, you know, you’re going to split people along party lines.

Now, that’s not necessarily a bad thing. This is a partisan issue. But you know you’re going to sort of lose people’s abstract thoughts on filibustering and, you know, democratic norms when you do that.

The second thing that they do is they explicitly mention, do you want to change the rules of the Senate so Bush and the Republicans can kind of get their way? If people are sort of presented with that information, it’s not too surprising that they’re going to decidedly sort of slide towards opposing that particular issue.

HUME: And in fact, of course, Daron, if the rules were changed, it wouldn’t just make it easier for Republicans to confirm Bush’s judicial nominees. It would make it easier for Democrats, if they were in the majority, to confirm a Democratic president’s nominees.

SHAW: Right. But there’s no context to the notion of changing the rules. There’s simply — you know, the Republicans want to change the rules.

HUME: Right. All right, now, we proceed to the one from the Pew Center that said, "The Republicans won the last election, so the president should be able to appoint anyone he wants to the federal courts, if a majority of the senators agreed."

And that was posed alongside a question that said, "The minority party ought to be able to block some of the judges they feel strongly about because judges are appointed to the federal courts for life terms." What’s your assessment of those two questions?

SHAW: Right. Well, I love what they tried to do here, which is they basically sort of went into the game saying, "You know, people aren’t going to know much about this. So let’s give them the Republican option, or you know, the Republican spin and the Democratic spin and see how they play out." The trouble I have is that — and they get predictable results, which is the Republican spin works positively for the Republican position, Democrats, it works well for the Democratic position.

HUME: On the other hand, they don’t refer to the Democrats by name. They refer to them only as the minority party, right?

SHAW: Exactly, "minority party." And in talking about the Republicans, you know, they basically sort of present a Republican — a particular Republican option that is certainly not the way the RNC would frame this, because what they say is that Bush can nominate anyone he wants.

And the use of the word "anyone," I think, is sort of provocative. I mean, it leads people to believe he could nominate Elmer Fudd, when in fact, you know, Bush’s judicial nominees, so far as I know, have not only been judged qualified but I think well-qualified by the ABA.

HUME: It’s a combination of qualified or well qualified. They’ve all been screened by the ABA and found qualified, at least, if not well qualified.

SHAW: Right. Right. So this notion of "anyone" — you know, Bush can nominate anyone, I think, kind of artificially suppresses support for the Republican position, compared to the Democrats.

HUME: And the Democrats are seen as blocking only some of the judges they feel strongly about, when in fact, of course, their position is they ought to be able to block anyone they want with a filibuster, right? And they claim that’s not it. But under the rules, they could, right?

SHAW: Right. And I think your point about — there’s identification of the Democratic position as the minority party, not the Democratic position.

HUME: Last question, real quick. The Rasmussen poll, "Rules should be changed, the vote must be taken, every person the president nominates, that wins," is that a better question?

SHAW: You know...

(CROSSTALK)

SHAW: Yes, I have some problems with the Rasmussen methodology, but I kind of like that question. I think that’s a reasonable way to ask it.

HUME: All right. Daron Shaw, great to have you. Thanks for your help on this.

SHAW: Thanks, you, too.

Content and Programming Copyright 2005 Fox News Network, L.L.C. ALL RIGHTS RESERVED. Transcription Copyright 2005 eMediaMillWorks, Inc. (f/k/a Federal Document Clearing House, Inc.), which takes sole responsibility for the accuracy of the transcription. ALL RIGHTS RESERVED. No license is granted to the user of this material except for the user's personal or internal use and, in such case, only one copy may be printed, nor shall user use any material for commercial purposes or in any fashion that may infringe upon Fox News Network, L.L.C.'s and eMediaMillWorks, Inc.'s copyrights or other proprietary rights or interests in the material. This is not a legal transcript for purposes of litigation.