A Must-Read On Poll Methodology

Zogby explains some of the arcana to the great unwashed (like me).

His gist: it is not an 11 point Bush lead, as Time and Newsweek have it, but Bush has done well in the last few weeks, had a great convention, and Kerry is "on the ropes".

The key is the weighting of Dems, Reps, and Independents - Zogby thinks that his weightings are right, and that Newsweek has the wrong mix of Reps and Dems.

Comments

[Ripclawe]

Newsweek sampling is completely out of whack considering they have weighed heavily demos since mondale. It's a setup poll so when they reverse the sample, "KERRY! The New comeback kid!" makes the cover.

[Andrew X]

USA Todays's call of a seven point difference (52 / 45)sounds about right to me.

As a rabid ANTI-conspiricist, one of the few plots 'n schemes that seems to resonate with me is that TimeWeek literally decided to skew their polls PRO-Bush by talking to more Republicans.

Reason being they see the dire straights of Kamp Kerry, and hope to give Bush an 11 point lead, that they can then report as an 8 point lead, then a 6 point lead, then a 3 point lead, as that "great closer" Kerry sops up the momentum.

'Course Kerry's gotta do SOMETHING in the next sixty days to help. (Ed: "John, give us a hand here. We can't put you in the Oval by OURSELVES, f'r cripes sake!" Heh.)

Now, anti-conspiricist that I am, maybe MY tinfoil hat needs replacing. But the media is so appallingly in the the tank for Kerry (ask Evan Thomas) that I gotta wonder.

[Andrew X]

Yikes. See Rip?? Look what they've done to us!

[TM]

Re: the Comeback Kid Conspiracy theory - that was my first thought as well (and it may become my third thought).

But I am currently dwelling on the theory (which I got around to posting earlier as "Perception Drives Reality") that the media really would have preferred to minimize the Bush bounce. Both sides are trying to raise money for their 527s right now, and the Dem effort is hurt by the appearance that money spent on Kerry is wasted. (I dismiss Rep overconfidence as a concern right now, although by mid-October, maybe).

That said, I also thought the TIME poll was high, and I don't have a better idea as to why they would do that.

[David Hunt]

Not so fast - I believe the Gallup poll, showing a 7 point lead is probably about right - and Zogby has been very hard on Bush and Republicans for a couple of years now - typically showing Bush's numbers 3-5 points down from the consensus of other polls - and Zogby completely missed the Republican sweep in 2002.

Yes, the Newsweek poll did oversample Republicans, but crunch the numbers yourself - they also weighted the survey, most likely to over-correct for that. If you just crunch the raw data, Bush would be up over 16 points for the total two-day survey - some kind of weighting got that down to 11 points - and that was likely correcting for the higher number of Republicans in the poll - Zogby does not mention this.

Hard to tell about Time, because they didn't release all their numbers and methodology.

[susan]

Bush is ahead 3 or 4 points. Kerry could still easily win.
Stop gloating, wake up and work harder.

[Martin Robins]

Yes, I think Bush is about 4 points ahead. Zogby is useless; according to them, Kerry is winning the EC vote by nearly 100.

[Jos Bleau]

Don't know who's ahead in th polls but do know that the media environment in this race has changed.

Sorry I don't have the exact cite, but over the weekend the New York Times ran a piece about the Russian school terrorist attack that quoted a Russian citizen praising the US appraoch to the war on terror. Quote was allowed to run without refutation or 'clarification'.

First time the admin approach to the war on terror has been praised by a subject without rebuttal in the Paper Of Record, I think.

The Times they are a'changin' ...

[Gerry]

Mr. Maguire,

Zogby currently has Bush leading by two points, but Senator Kerry leading just about everywhere. Yet he writes that Kerry is on the ropes? One of two things is probably true: either he does not believe what he just wrote, or he does not believe his own numbers.

I just wrote an article a few days ago about the various polls and about controlling for partisan breakdown in them. Some pollsters (Zogby, Rasmussen) do. Others (Gallup, Pew) do not. When you do not control for partisanship, you can get odd results like the Time poll from the week of the RNC, or the Newsweek polls from both the RNC and DNC.

However, when you do control for partisan identification, you can be very wrong if your assumptions are wrong. For examples of this, see Zogby's polling in 2002 which was awful across the board, or Rasmussen's 2000 results.

This year, Zogby is using 39% Democrats and 35% Republicans. Why? Because that is what the turnout was in 2000 according to the VNS exit polls. If the turnout this year is different, his polls have no chance of being right. My article details what the partisanship details have been since 1976 in Presidential elections.

[Greg F]

This year, Zogby is using 39% Democrats and 35% Republicans. Why? Because that is what the turnout was in 2000 according to the VNS exit polls.

This is so wrong wrong wrong. For statistical validity the sample must be random. From a pure statistical point of view there is one of two things wrong with the polls. Either their party affiliation percentages are wrong or their sampling is not random, you can’t have it both ways. By default a random sample would select an appropriate percentage from each of the 3 groups. If the pollsters argue the percentages are wrong then they are also telling you they are not random and therefore not valid. You cannot use historical data to correct for this, it is a case of garbage in garbage out.

[GT]

Tom,

In the end, and as we learned in 2000, it's not the national poll that matters but the swing states.

There are three polls (Gallup, Rasmussen, and Zogby) that have specifically looked at battleground states and they all have (or had, since this changes daily) Kerry ahead.

And this is right after the convention.

Gallup has Kerry ahead by 5 among RV. I don't know what they have for LV in swing states and there is a big methodological debate as to wheter LV or RV are better at this stage.

Zogby's poll came out yesterday for the WSJ and has Kerry over 300 in the EC.

Rasmussen has a daily tracking poll for battleground states, available for premium subscribers. Yesterday's data had Kerry up by 1.2%. I don't know about today.

[Jeff]

Good Lord--and I thought the left wing had become increasingly paranoid and conspiracist. Nice to see that the "liberal media's" current polling that places Bush 11% up is actually a plot--to help Kerry!

Get a grip.

[Gerry]

"For statistical validity the sample must be random."

Yes, and no.

This technique is called stratification. In many ways, for political polling it amounts to combining three polls (one of Republicans, one of Democrats, one of others) in a weighted model. If each strata has its sample chosen randomly (but within the definition) then it is still valid, although the combined results can only be as accurate as the proscribed turnout model.

[Cecil Turner]

This article probably ought to be required reading for interpreting Zogby polls. The gist is that he's more into opinion shaping than most pollsters, and on this subject he's clearly not disinterested. I'm reluctant to claim his post-9/11 performance decrement is a result of bias, but nor am I willing to discount it as a possibility.

And the complaint on the weighting issue is that it's necessarily an estimate. From the same article:

"There's no known distribution of party identification in the sense that we think of a known distribution of sex or race. All we know comes from other survey data, so it has to be an estimate by definition."

[GT]

Although no pollster is perfect and they all have their misses Zogby has one fo the best records of all and was one of only two polling organizations to get the 2000 election right.

[Jim Glass]

"... it's not the national poll that matters but the swing states ... There are three polls (Gallup, Rasmussen, and Zogby) that have specifically looked at battleground states and they all have (or had, since this changes daily) Kerry ahead."

My gosh, the Democrats aren't planning to win the election with a minority of the popular vote, are they?

What a kefuffle that would cause!

[GT]

Wouldn't it be fun if that happened?

[Greg F]

"This technique is called stratification."

No, a strata is a distinct population (i.e. Democrats, Republicans, and independents). When your sample is non-random, stratification is used to determine weighting to compensate for the non-random nature of the data, (correct for party affiliation percentage), which is my biggest problem with the polls. All of the pollsters are assuming the samples are non-random, which seems reasonable, but with wide variations. For example, Zogby uses the 2000 and 1996 elections to arrive at his weighting. I believe his weighting is highly suspect as a lot has changed since 9/11. This table of data of liberal/conservative identification indicates a significant shift in attitudes, and most probably party affiliation, between 2000 and 2002. Examining the data, there is a fairly abrupt change from historical levels between 2000 and 2002. If we group “slightly conservative'/ 'slightly liberal'/ 'middle of the road'/ 'haven’t thought” as a group, we find a significant drop from historical low of 71% to 63%. IOW a fairly large percentage have left the middle. The voters identifying themselves as conservative jumps from 18% in 2000 to 25% in 2002, while liberals go from 11% to 14%. It would seem reasonable to me that the Republicans would experience an increase in the percentage of party affiliation relative to the Democrats. As a disclaimer I will concede that this change could have been temporary and voter identification may have returned to pre 9/11 levels, we just don’t know. I don’t usually trust the polls and this year I trust them less.

"In many ways, for political polling it amounts to combining three polls (one of Republicans, one of Democrats, one of others) in a weighted model."

Combining data from different studies/ polls ect. is usually done by a meta analysis.

[Kenneth]

When Ted Koppell recently defiled himself and the entire U.S. media it left me feeling strange. I was never a conspiracist but seeing Koppell so weak...so emasculated in his rationalization over the Vietnam piece...it left me nauseated. I don't say that sarcastically but in a literal and forboding sense. For the first time in my life I saw the liberal puppet-master actually pulling the strings on live television. It was creepy.