How to grade the polls

Byline: | Category: 2012 | Posted at: Tuesday, 6 November 2012

Let me begin by saying that whomever calls it closest doesn’t necessarily win.  Sure, I trumpet Rasmussen’s 2008 prediction, but that is simply to demonstrate to those who think that he is a partisan hack that he has a pretty decent track record and cannot be discounted for partisan leanings without being guilty of partisanship yourself.

Before we can answer who grades this election right, we have to first revisit a little basic statistics.  Let’s first assume that we have populations large enough as to approximate infinity (we do).  Theoretically then, if we want to release polls within a 90% confidence interval (most are 90 or 95 percent, but often they don’t say, so I assume the lower percent), then we need a representative sample of about 1,200 voters to get a margine of error of about 3%.  That means that if you gather a 100 different representative samples of 1,200 respondents each, on average, 90 of them will give you the right answer to within a range of about +/- 3 points.

Whoever then gets closest, even if they do everything right in terms of gathering their sample, is as much a game of chance as skill.  If the final result is a 50-50 ties,  Acme Polling’s November 6th estimate of a 2-point Obama lead is not necessarily worse than Pollco’s prediction that same day of a tie.  Let’s look at those two polling companies and see who better tracked the ultimate result.

pollco_2.jpg

So who did the better job polling this race?  If you go by final result, you would say Pollco because their last poll predicted the ultimate result of a tied 50-50 race.  However, if you look at each poll over the last ten days, you see that Pollco average an Obama lead of 6 points while Acme Polling’s average was a tie.  It just so happened that on the final day of polling Pollco got a lucky break with their sample, while Acme Polling got a sample that gave them a result within their margin of error.  It just happened to be the one-day-in-ten when Pollco was outside their four-point margin of error.  However, Pollco’s methodology overall was clearly biased in Obama’s favor.  (In statistics ”bias” is a neutral term that has nothing to do with ideology.)

When it comes to determining who did the best job of estimating today’s election, we’re going to have to look at not just the overall result, but their track record as well.  If Romney wins by five points, for example, then there is an argument to be made that Gallup did a better job than Rasmussen, because for a longer period of time Gallup was closer to the final result, even though both organizations will have missed it by four.  If Obama wins by three, then we’re probably looking at giving the laurels to IBD/TIPP. 

Bottom line:  you can’t just look at the final number; you have to consider the track record to know which polling company did the best job.

Share this post:

One Response to “How to grade the polls”

  1. Lorenz Gude Says:

    Right now at 10:30 Eastern it looks to me that Obama will win solidly. We get an election like 2004 with the incumbent winning without much drama. I have to say Obama did enough between the debates and Benghazi to turn it into a 1980 style win for Romney. But evidently Romney didn’t have the horsepower to win. He looked better to me as the elections season wore on so I am mildly surprised that he doesn’t seem to be doing well at all. It looks like we have avoided a 2000 style election which could have been the worst outcome depriving either candidate of legitimacy. My only other observation is that it looks to like we are now going to have 16 years of poor government from Washington. The states are facing up to the fiscal crisis and by 2016 it should be much more obvious that there is ‘no more cake’. I suspect the electorate just voted for more cake, but I think they will be disappointed. Then we will see what emerges from the American people.