Milwaukee J-S Poll

The Milwaukee Journal-Sentinel has a poll of Wisconsin out this afternoon.  The top line has the race going from an Obama 1-point lead two weeks ago to an 8-point lead today.  That’s a shift that defies common sense. 

However, that doesn’t mean that they did the poll wrong.  Let me explain.

A poll of about a 1,200 respondents in its sample will have a margin of error of about 3% at the 90% confidence interval.  If the actual race is Obama 45 – Romney 45, and you select a hundred different random samples from the population, you would expect that 90% of those samples to give you a result of somewhere between Romney 42 and Romney 48.  The other ten polls will be outliers of more than the margin of error of 3 percentage points.

I don’t see anything necessarily wrong in the demographices of sample polled.  Nor did I observe anything done differently from the poll the J-S did two weeks ago.  However, with little reason to believe that the race moved 8-points in Barack Obama’s direction, it is likely that either the earlier poll or this one is an outlier.  Given where everyone else is showing the state of the race in Wisconsin, it is probably this most recent poll that is the outlier. 

But my standard state poll caution applies:  Be very wary of media and university polls of states that are not routinely performed time and time again.  Polling at the state level can be very screwy and there are a lot of one-off polls performed with shady methodologies.

Finally, there is one odd thing about this poll.  It shows a 100% likelihood of voting even as there are 51 respondents who report being unregistered.  Now I believe that Wisconsin has same-day registration, so that might explain the unregistered but likely respondents.  However, that wouldn’t explain the other issue.  See the table below:

Either the J-S includes only those who say that they are absolutely certain to vote plus those who report that they have early voted in their likely voter sample, or some segment of this likely voter sample is lying about their intent to vote.  Anyone know?

Obama opens widest lead

Proof of Barack Obama’s last-minute surge came from National Journal’s latest poll today.  He leads by 5 points, his widest margin of any national poll since the first debate.  And for only the second time in that same period the incumbent has hit the magic 50%

Of course, there’s this:

In its likely-voter model, the Congressional Connection Poll projected that the 2012 electorate will be virtually unchanged from 2008, with Democrats holding an 8 percentage-point advantage among voters (compared with 7 points last time) and whites representing 73 percent of voters (compared to 74 percent last time).

It was actually 76% if you accept the Census figures instead of the exit polls.  Nonetheless, now Nate Silver isn’t out alone on his limb. 

But if you really believe that youth are more likely to vote this year, that there will be no dropoff in minority participation, and that after four years of Barack Obama as President, Republicans are as unenthused about Mitt Romney as they were about John McCain, then this poll probably gives you some comfort.

UPDATE:

The RCP Average is now tied.  A few grains of salt:

1.  The National Journal poll is now in the mix.  It is an outlier by four points from any of the other polls in terms of its spread.  It shows the second lowest Romney portion of the vote and the highest percentage that Obama has received of any poll in a couple weeks.  Without this poll in the mix, instead of the race tied at 47.6, it would be Romney 47.9 to Obama 47.3.  That’s why it’s a good thing to remove outliers if you’re going to do an average of polls.  Remember, even when you do polling correctly, you still get it wrong one time out of ten.  Removing outliers removes their influence from the result.

2.  The National Journal poll of 713 likely voters is a relatively small sample for a national poll.  Even more troubling is that it comes from a pool of 1,010 adults.  That means that 71% of the adult population contacted was queried for this “likely voter” poll.  Since 1972, turnout has never been that high.  In 2008 only about 62% of the voting age population showed up to vote.  Usually turnout is in the high-50s.  71% is unreasonably high.

3.  The poll was conducted from the 25th through the 28th.  Three of those four nights were weekends, and that skews the results.  In fact, with polling conducted over four days, I’m having trouble understanding why so few respondents. 

4.  Finally, the last night of polling was the 28th of October, hours before Sandy hit the coast.  From here on out, I’m not sure what to think about national polling, given the effects of the storm on such a large swath of a very populated portion of the nation.  I suspect that it is going to give us screwy samples for another day or two.

Polling: gospel or heresy?

Here are a couple interesting stories from two well-respected poll watchers.

Jay Cost looks at the polls in Ohio and thinks that he sees a bimodal distribution of their results.  Some polls (Rasmussen, Gravis, ARG) consistently show a very tight race.  Other polls (CBS, NBC, PPP) continue to show a sizable Obama lead.  In a normal distribution of polling results we would expect to see polls clustered about the mean, with each polling group sitting on boths sides of the average.  Here there are virtually no polls showing an Obama lead of two points, which is the approximate arithmetic mean.

The aptly named Sean Trende explores the divergence between the state polls and the national polls and concludes that they can’t both be right.  National polls show a Romney lead of approximately one point, but the state polls continue to show Obama leads in many of the swing states.  Trende constructed a model using the average of state polls to try and come up with the national result.  He couldn’t make the numbers work.

Both Cost and Trende are observing something I have been pointing out for weeks.  How you build your sample predetermines your result.  If you assume that 2012 turnout is going to closely match 2008, then you probably have a sample showing a slight Obama lead.  But if you assume a return to a more historically normal turnout, then your poll will like have Mitt Romney with a comfortable margin.

This just goes to show what I like to say about polling:  “It is an educated guess heaped upon conjecture piled atop assumptions filtered through subjectivity and complicated by lies.”  It would be a heresy to mathematics to treat polling as gospel truth.

MORE:

AJ Strata and Ricochet have their takes too.

The trend tells the story

Quinnipiac is out with a poll of three key states that it did for CBS and the New York Times.  Here’s a data point out of Virginia that just leaves you scratching your head:  Mitt Romney leads among the 35% of the electorate that calls itself independent by 21 points but somehow trails overall by two points.  Really?

All election season long, Quinnipiac has produced Obama leads, so large and outlying that Mark Halperin told the MSNBC audience that he was skeptical of this poll.

However, keep in mind that I keep saying to ignore the top line results, but to look at the trends. Even with a bad sampling methodology–which I believe Quinnipiac has–over time the trend is still likely to appear. That’s why these results should alarm Obama’s camp (emphasis added):

The Race in Florida
President Obama continues to enjoy a double-digit lead among women, 53 to 43 percent, but that lead has been almost cut in half since last month. Romney has increased his advantage with men from three points last month to nine points now, 52 to 43 percent.

The Race in Virginia
In Virginia, the gender gap has narrowed slightly but it is still significant: President Obama holds a 10-point lead among women, 53 to 43 percent, while Romney holds a 9-point lead among men, 52 to 43 percent. . . Among independents in Virginia, Romney has surged from a two-point deficit two weeks ago to a 21-point lead today, 57 to 36 percent. He has also increased his lead slightly among voters in military households.

There is still one item of good news for Obama in this poll.  In Ohio the race hasn’t much changed.

When to be skeptical

So you see a news report about a horribly obtuse advertisement employed by one of the campaigns.  Should you immediately accept the report?  Or should you withhold judgment until you know more?

First there’s the story about how the advertisement was received:  “A friend of ours in northern Virginia recieved this flier delivered to her front door this morning.”  The use of the “friend” is a ubiquitous element in most urban legends.

Then there’s the timing:  she received the flier “this morning.”  As in 30 October.  As in yesterday, less than 18 hours after the storm’s eye made its way onto land.

Then there’s the flyer itself:  A grainy black and white photo in low resolution. 

Then there’s the attribution to a person and organization hated by the other side:  “Our friend says the flier says it was produced by Americans for Tax Reform. That’s the anti-tax group run by Grover Norquist, a leading conservative movement leader and prominent Obama critic.”  But where on the ad does it say that?  If it says that on the flyer, it’s not obvious from the picture you have. 

If your world view is that the other side is not just wrong, but . . . evil . . . then you probably accept the report of the political advertisement without a second thought.  But if you believe that those Americans you disagree with are wrong but are essentially good people, you probably wait to reserve judgment about the veracity of the report.

If you went with the second option above, you were correct.

“An ATR spokesman comments, ‘the photo you have is of a photocopy of a piece of mail we sent out in September.  Someone is either trying to be cute or deliberately trying to mislead.”

It’s not even the same flyer.  The background is the same, but the words in the alleged flyer apparently have been photoshopped over the real ATR advertisement.  (Correction: it is the second page of the original flyer:  but what you see in the real flyer is the standard bulk rate mailing header.  That is blacked out in the Houston Chronicle’s report.  If that had been in the original picture it would have been a dead giveaway that the flyer was not received “yesterday” as the report said.

If you chose to be skeptical, congratulations; you are well-suited to wading through our nation’s sometimes confusing political minefield.  If not, as a consolation prize, perhaps you have employment options with the Houston Chronicle or MSNBC.

UPDATE: The below is now posted by the Houston Chronicle at the orginal link:

UPDATE: We received this response from John Kartch, spokesman for Americans for Tax Reform:

“I think someone is trying to mislead you.  We’re circulating no such flyer.  ATR sent out a  mail piece opposing President Obama’s policies using a storm analogy way back in September.  Sounds like someone is being dishonest.”

NOTE: We’re planning to post a separate item explaining ATR’s response on this site.

MORE:  Here’s the link to their new post.  Rick Dunham, the author of the original post, does not offer a mea culpa for having been snookered by his “friend,” but there is enough in this new post that it’s a tacit admission of error.

Throwing cold water at your excitement

Drudge reports breathlessly: 

GALLUP: OBAMA’S EARLY-VOTE LEAD DROPS 22 PTS FROM 2008…

Before you get too excited, look at the internals:

Those over 50 are more than twice as likely to have early voted than those under 50.  So by that metric alone, if there weren’t a Republican lead, Mitt Romney would be in serious trouble. 

Look also at the geographic breakdown.  More people report having already voted in the South and West where Mitt Romney will probably do significantly better than Barack Obama.  But in the East only 4% of voters report having already voted.  That might be because the two largest states in the region–New York and Pennsylvania–don’t have early voting.

This is an example of a sample that is not representative of the national voting population.  So, as Glenn would warn you, don’t get cocky, kid!

MORE:

I’ve received some emails asking me to elaborate on the above. 

First.  The states that conduct early voting are not the same from year to year.  Comparing 2012 early voting results with those from 2008 is skewed by the differing mix of early voting states from year to year.  To make a retail comparison, you can’t just look at all sales and conclude that a company is doing better because sales are going up.  It may be the result of expansion instead.  If one wants to know how a company is performing from year to year, one has to compare same-store-sales. 

Second.  Even when you compare same-store-sales, there can be exogenous factors that greatly influence the results.  Using retail again as an example, the greatest predictor of November sales is the date on which Thanksgiving falls.  Since the holiday is the fourth Thursday of the month, that means that the next day (the unofficial start of the Christmas shopping season) can come as early as the 23rd or as late as the 29th of the month.  You can’t directly compare even same-store-sales from one November with eight holiday shopping days to another November that has only two.  Getting back to elections, those states that have early voting, don’t always have the same number of early voting days from year to year.  Furthermore, there is evidence to indicate that the propensity to early vote increases with the number of years that a state has had early voting as an option.  That again shifts the population from year to year.

Finally, there is another bit of semi-related news out of Gallup.  They have suspended national polling for at least two days.  That seems a smart thing to do when you have some states with 15 to 25 percent of the population without power and the remaining population with better things to do than to answer the phone.  Getting a representative sample under those conditions was going to be even more difficult with Sandy than it already is. 

This is going to drive poll watchers nuts.  There is no precedent for an event like this.  Polling samples will be skewed for several days, if not a week.  Polling results might change due to the President’s performance in the wake of the storm.  Early voting patterns will be disrupted.  None of this is modelable–at least not with a model that is testable against the past.

My suggestion?  Go vote.  And then sit back and see how it turns out next Tuesday night.

Out on a leftward limb

One thing that you have to admire about Nate Silver is that he isn’t afraid to go out on a limb.  As an example of that, the New York Times political soothsayer currently projects Barack Obama winning the popular vote by 1.7%. 

That would place him well on the left side of most current polling.  The below histogram shows the distribution of the spread between the two candidates in each of the ten polls that make up today’s RCP Average.

There are five polls showing a Romney lead, three showing an Obama lead, and two ties.  The RCP Average has Romney winning the national vote by 0.9%.  (Since many polls use a 90% confidence interval, and that means that the remaining 10% of the time, a poll will be outside the margin of error, I’ve isolated the highest and lowest polls from this sample.  Even by eliminating the effects of those outliers, that still makes the adjusted average a Romney lead of .75%.)

Nate Silver’s projection that the spread will be 1.7% in Obama’s favor is a full 2.6 points larger than the RCP average.  That is quite an outlying position.

However, remember that I keep saying to ignore the spread and to look instead at the incumbent’s level of support.  By that metric, Silver’s estimate is even more extreme.  Below is the histogram of those same polls and where they show Obama’s numbers right now.  The RCP average of Obama’s support sits currently at 46.7%.  But Nate Silver projects the President to finish with 50.4%, nearly four points higher than today.

Now, to be fair to Silver, there are still undecided voters, some of whom will break in the President’s direction.  If we assume that about one-percent of the electorate is going to vote for a third-party candidate (an assumption Silver makes as well), then the current RCP average leaves 4.7% of voters still undecided.  However, for Barack Obama to go from 46.7% today to the 50.4% that Nate Silver projects a week from now, it would require that he get 79% of the remaining undecided vote.  That’s certainly going out on a limb.

Either nearly every public poll we currently see is vastly over-reporting Republican strength (a result opposite of what occurred just four years ago) and Nate Silver is the greatest prognosticator of our time, or Nate Silver will be the one eating crow on election night.  In a week we will see which one it is.

RELATED:

Dylan Byers is skeptical as well.  Read the comments.  I’ll sum up most of them in three words:  appeal to authority.

MORE:

This is shaping up into a battle of old versus new.  Long time polling stalwart Gallup reports this huge bombshell of news

“Romney currently leads Obama 52% to 45% among voters who say they have already cast their ballots.” 

If that is the case, Mitt Romney is crushing this race.  On the other hand, Gallup was 20th of 23 polling organizations analyzed four years ago, while Nate Silver, who was born a mere three years before George Gallup died in his 83rd year, got 49 out of 50 state races right.  Needless to say, both pollsters know that they have a lot riding on this race.  Will this be a case of a new type of methodology winning out over the old?  Or will the neophyte get his comeuppance?

Does Ohio really matter?

Gripped by the memory of trench warfare less than two decades before, both France and Germany built a line of heavily armed fortifications stretching north along their shared boundary from the point where both countries met the Swiss border.  The German Westwall (often called the Siegfried Line after the name of its WWI predecessor) was not nearly as strong as its French opposite: the unbreachable Maginot Line.  But in the end it didn’t matter, all because of a little place called Belgium.

Conventional political wisdom holds that Ohio is, as the state’s slogan goes, “the heart of it all.”  Since 2004 both parties’ candidates have put more money, people, and resources there than anywhere else in the nation.  As such, both sides know every bit of its ground.  It is political trench warfare in the Buckeye State this year, just as it was four and eight years ago. 

The use of new technology has improved the efficiency of campaigning.  And since Ohio is so important, new technologies are employed there first.  Micro-targeting produces micro-gains.  That may give one side or the other a few inches, or in political terms–households, but since Ohio is the juiciest electoral prize within one percent of the nation’s political 50-yard line, every inch helps. 

But new technologies are expensive.  They can’t be employed everywhere at once.  Even more dear is the cadidate’s time.  In the last days of a race, trailing candidates camp out in Ohio, because it becomes the must-have electoral prize that a popular vote losing candidate cannot hope to win without.

Ever since Al Gore dedicated insufficient resources to Ohio, no subsequent presidential candidate has dared to make that mistake.  As a result, both John Kerry and John McCain made good showings there even as they lost the overall vote.  Kerry lost nationally by 2.4%, but fell short in Ohio by just 0.3% less.  Four years later, John McCain, while losing by 7.3%, trimmed his losses in the Buckeye State to only 4.6%.  Ohio is on the critical path to securing an electoral victory if one fears being on the trailing side in the popular vote.  That is the reason why the last two losing candidates lost by smaller amounts in Ohio:  it was where they made their last stand.

However, on the side of the popular vote leader, Ohio isn’t necessarily key.  That is because usually within a few fractions of a percentage of it are other states with enough electoral votes to win.  While Bush took Ohio with a slim majority of only 118,000 votes, a switch one-tenth that size would have given him Wisconsin’s 10 electoral votes and would have made Ohio unnecessary.  Had he secured a slightly larger popular lead than he had, it was likely that the Badger State would have gone his way.  

The more time and resources the trailing candidate devotes to Ohio, the more vulnerable he becomes in other states.  That’s what happened four years ago, and that is what confused me about the state of the race then.  Fixated on Ohio, I ignored its flanks.  As I pointed out on Friday, even if John McCain had taken Ohio and every other state to its right (Florida, Indiana, and North Carolina), he was still going to lose the race.  Either one of Colorado or Virginia was going to make Ohio moot.  Behind in the polls overal, McCain could not have defended them all.

Take a look at an updated version of the chart I showed the other day. 

Florida is now off the list of contested states.  Of the states that Romney has to win, only Virginia and Colorado remain.  Every other state in the red box is now a must win for Obama.  When you’ve lost the popular vote advantage, no matter how strong you are between Lake Erie and the banks of the Ohio, you have no margin for error anywhere else in the land.

Sean Trende said to cautiously optimistic Republicans like myself, “Guys, the national polls are interesting, but until polls start showing Romney ahead in NV/IA, WI, or OH, he’s losing.”  That’s true, but only to an extent.  The deeper one candidate’s penetration of the popular vote, the wider becomes the political front.  With another point or more of separation, even Michigan gets added to that list.  Only one of them has to switch. 

The bottom line is that if Mitt Romney’s popular vote margin is large enough, no matter what happens in Ohio, some other state is going to play Belgium to Obama’s Maginot Line. 

Part II: Obama optimism: why do they still believe?

Recently I was asked by a friend why Intrade showed more than a 60% chance of re-election in spite of polls giving Romney a lead.  Yesterday I pointed to just a few of the numerous data points and trends that show that Barack Obama is on the way to a popular vote defeat and that such a loss places him in no better than a 50-50 race to win 270 electoral votes.  Today I’m going to discuss why his supporters (and the markets) refuse to believe that he could lose.

It’s not just a river in Egypt

I know all about denial.  I lived it four years ago.  Then, I made a wildly optimistic prediction that John McCain would lose the popular vote, but eke out an electoral college win.  Sure, the polls showed him down nationally, but there were indicators that his ground game in traditional bellwethers Missouri and Ohio was beating the national trend.  And if he could win there, a Republican couldn’t possibly lose reliable red states like Indiana, North Carolina, and Virginia.  I ignored the signals that were flashing–such as when McCain ran a mindless celebrity ad containing a message that resonated only with his hardest core supporters.  I should have seen the sophomoric appeal for what it was: an admission  that the base was seriously unenthused.

From inside the bubble of misplaced self-confidence, Democrats probably don’t recognize the similarity between what I was thinking then and what they believe now.  Complacency is the bitter fruit of victory, while circumpsection is dessert served after a meal of crow.  It likely will take a presidential loss before Democrats confront the reality that Minnesota and Wisconsin have been sliding away from them for years, just as I ignored long-term indicators that Colorado and Nevada were trending left. 

It’s not bias if it’s correct

Democrats have no monopoly on the propensity to blame bias for reporting that is contrary to their world view.  I can’t count the number of times this election cycle that I have heard Republicans claim that polls showing an Obama lead, were media fabrications with the purposeful intent to manipulate the electorate.  However, Democrats do seem to take cries of bias to a whole new level even when there is independent evidence that reports of bias are unfounded. 

Take polling done by Rasmussen as an example. A Fordham University study of 2008 presidential polls found that Rasmussen was tied for first with the most accurate prediction of the final outcome of the 23 polling organizations they analyzed.  In spite of this recent presidential polling success, a number of prominent Democrats contemptuously ignore what Rasmussen currently says.  Electoral-vote.com goes so far as to run two aggregator models: one with, and one without, Rasmussen polls.  Liberal scold Andrew Sullivan agrees, “I generally remove Rasmussen from the poll of polls, because they are so openly biased in their sample.”  It’s one thing to see a poll from Rasmussen and to decry its bias; but it’s something else entirely to remove Rasmussen from your field of view.

All the news that’s fit to print

Is there a more arrogant slogan than that which runs on the masthead of the New York Times?  Nonetheless, many of its readers apparently believe it is true.  Blind faith in the omniscience of a chosen media outlet is far from a recent phenomenon.  More than a century ago, the most famous editorial ever written contained the line from Virginia’s father that, “If you see it in the [New York] Sun, it is so.” 

Today’s New York Times, because of its effect on what other press outlets report, is in an especially influential position when it chooses what isn’t fit to print.  If one wants, one can walk through life purposefully censoring exposure to contrary opinions like might be contained in reports by Rasmussen, Fox News, and the Drudge Report.  However, because the major networks and most newspapers share a liveral basis, it is much more difficult for a conservative to limit oneself only to reports from a conservative press.  So it’s not that conservatives are necessarily more broad-minded that they get greater exposure to contrary points of view, it’s that there is little chance for conservatives to avoid them.  The reverse feat would be difficult to pull off.  As John Stuart Mill reminded us, “He who knows only of his own side of the case, knows little of that.”

Pauline Kael’s oft-misquoted remark that she “only [knew] one person who voted for Nixon,” in spite of the fact that Nixon enjoyed the largest popular vote margin of any presidential candidate in the last 75 years, is an attitude typical of a liberal who has lived a blinkered way of life.  Pauline Kael wasn’t just an ordinary liberal, but was a prominent columnist for New Yorker magazine.  Meanwhile, two blocks away, columnists at the New York Times enjoy even more vaunted positions of isolation.

Krugmania

There is something about a New York Times byline that seems to turn a man of accomplishment into a god–at least in the eyes of many of his readers. 

Paul Krugman is a Nobel laureate.  He left his influential position in Princeton University’s economics department for a loftier perch at the New York Times.  An unabashed Keynesian, he laments that over the last four years the federal government hasn’t spent nearly enough to pull the economy out of recession.  One can attack Krugman’s economic prescriptions (especially as many of them run contrary to the advice he gave when there was a Republican in the White House), even as one acknowledges the gravity of his economic view. 

If a Times writer is not careful, that god-like adulation can go to one’s head.  It can cause one to take shortcuts and to pretend that expertise extends to other areas as well.  When Krugman steps out of the field of economics and into politics, which is more often than not these days, he is especially careless and obtuse.  This widely circulated two-part post by internet satirist “Iowahawk” is probably the best demonstration of how completely wrong Paul Krugman can be.  So wrong, in fact, that it should beg for skepticism in the minds of any who hear Krugman speak.  Nor is it just the occasional Iowa auto body repairman who recognizes that Krugman oft wanders across the line between data and imagination.  Upon the occasion of his retirement from the Times, former ombudsman Daniel Okrent wrote that, “Paul Krugman has the disturbing habit of shaping, slicing and selectively citing numbers in a fashion that pleases his acolytes but leaves him open to substantive assaults.” 

[See update below] Cautions like that don’t stop people who should know better.  Atlantic columnist Clive Crook just this week gushed:

“I don’t know about you, but when confusion arises in disputes of this kind, I always turn to Krugman for dispassionate adjudication. . . Thank heavens we have Krugman to lean against the trend, and go where the evidence takes him without partisan preference.”

Bunk.  What Crook really means, is that Krugman takes him where Crook’s partisan preference already wants to be.  Crook is not alone.  Paul Krugman benefits from having a readership that largely turns to him to confirm what they already believe.  Amplified by the Atlantic and others, Krugmania rolls on.  

Nate Silver is another man of accomplishment now at the Times.  He too has many readers who, willfully or accidentally, are ignorant of other points of view.  So Silver’s confirmational pronouncements pleases his acolytes.  Because his prognostications are buttressed by the supposed authority of the Grey Lady, they are absorbed verbatim by credulous readers who believe that if you read it in the Times, it is so.

People, not pencils

Because I just compared Nate Silver to Paul Krugman (a juxtaposition that is terribly unfair to the sabremetrician), one might get the impression that I don’t like Nate Silver and don’t find his analyses informative.  That would be incorrect.  I’ve read him since before the 2008 election and admire him for his data savvy superior to my own. 

But where I do find fault, is his certitude.  I don’t mean that attitudinally.  As I am sometimes accused of being arrogant myself, it would be unfair for me to hold against others a similar trait.  But I do criticize Silver for the certitude implied in the exactness of his results.  When I read on 538.com that Obama has a 76.6% chance of winning Ohio, I am reminded of the joke about the old docent at the natural history museum.  He points proudly to a fossil and says, “These bones are two-million seven years old.”  The astonished visitor asks the man how he knows the age so precisely.  To which the old man responds, “The exhibit was two-million years old when I started working here seven years ago.”  

AJStrata, a statistician who works with Global Positioning Systems, says it well.  He cautions readers to be wary of “ridiculously precise models using data that has enormous error bars.  Where Keplerian physics is well understood and can be modeled precisely enough that the incertitude of the samples [one] measures makes little difference to the outcome, in politics, “the dynamics of what is being sampled (the electorate) is very poorly understood.”

The progressive is particularly prone to being fooled by the supposed certainty of expertise.  It is an ideology that believes that if the right power can be placed into the hands of experts capable of understanding and controlling the complex, then the world will be a better place.  If one already accepts that the value of currency, the nation’s entire health care network, and even the planet’s climate are not so complicated that a smart expert with good intentions could predict and control outcomes, then certainly Nate Silver, a billiant and well-intentioned man, is capable of solving a minor problem of math down to the tenth of a point.

However, while polling involves math, it is not math.  The math within it bears little resemblance to the certainty of Keplerian physics.  Keep in mind that I am a fan of polling and have done it professionally before.  But having said that, I urge the reader to remember that polling is an educated guess heaped upon conjecture piled atop assumptions filtered through subjectivity and complicated by lies.  To treat it as more certain than that is to regard humanity with disdain.  For polling is nothing more than a gauge of human interactions and humans are complicated things.

Economics is another gauge of human interactions.  One of the best economics essays of the last hundred years is, I, Pencil, a commentary indirectly attacking the arrogance of men who believe that they can predict and control even seemingly simple things.  “Pick me up,” says the pencil.  “Not much meets the eye—there’s some wood, lacquer, the printed labeling, graphite lead, a bit of metal, and an eraser.”  Simple, right? Economist Leonard Read then goes on to describe the complexity of making even the most ordinary object in the world. 

“Now contemplate all the saws and trucks and rope and the countless other gear used in harvesting and carting the cedar logs to the railroad siding. Think of all the persons and the numberless skills that went into their fabrication: the mining of ore, the making of steel and its refinement into saws, axes, motors; the growing of hemp and bringing it through all the stages to heavy and strong rope; the logging camps with their beds and mess halls, the cookery and the raising of all the foods. Why, untold thousands of persons had a hand in every cup of coffee the loggers drink!”

And that describes only the logs.  From there they are shipped and milled, before being filled with graphite and other chemicals, painted, and then given a tiny eraser on top.  Conceivably millions of people have a hand in the making of a pencil.  And yet, without any central control whatsoever, somehow, that old reliable #2 is always available wherever I choose to shop.  

There have been many command economies that have thought that they could control events far more complex than producing pencils.  Most no longer exist.  Unfortunately for the progressive experts, humanity is the greatest source of complexity in the world, and it is humanity itself that the experts wish to predict and to control. 

So what?

Yesterday’s post, along with all of the above from today, helps to explain why I think that Democratic optimism in an Obama re-election is unfounded.  The lack of circumspection that oft follows victory, the easy escape from other views, and the progressive’s obedience to expertise make Barack Obama’s supporters more optimistic than they should be.  Even while Barack Obama is still close enough to win, it should be clear by now that he is in an undesirable position for an incument.  However, this doesn’t address how, if I’m right, Intrade could be so wrong. 

For that, we have to look at Thursday’s ABC/Washington Post poll*.  Even as this poll indicated that Mitt Romney held on to a three-point lead and sat at the magic number of 50%, it also found that among Obama’s supporters, more than 90% of those polled expected Obama to win.  Among Romney’s supporters barely 70% thought that their man would prevail.   It is an incongruous result when a majority expects the winning candidate to be the one with a minority of support.  However, Intrade, like polling, is simply a gauge of human emotions.  And those emotions are influenced by expert opinion.  Far  too many Democrats have faith in their experts’ opinions and enough Republicans fear that those experts are right.

* I have misplaced the link to the internals from that poll.  Rather than expect you to trust me, I’ll provide this corroborating story.  On Friday’s Anderson Cooper show, CNN National Correspondent Gary Tuchman noted that the closing messages from both campaigns is curiouly identical: 

“Romney’s strategists have the concern that the perceived lack of their candidate’s competitiveness could ultimately lead to less enthusiasm, and therefore, lower turnout.  So the message being emphasized to the GOP base is that Romney can win.  And that makes the message from Obama strategists all the more interesting.  In order to increase Democratic enthusiasm and turnout, they have the very same message:  ‘Romney can win this.’”

UPDATE:  Thanks to several readers for pointing out that Clive Crook was being ironic when he gushed over Paul Krugman.  Irony is best when it is believable, but sometimes it can be a little too good.

Part I: Obama optimism: why it is unfounded

A smart friend wondered today why Intrade still expects an Obama win even when the national poll numbers are slipping away from him.  It’s a good question that I’ll attempt to answer in two parts.  The first part is about why Obama optimism is unfounded.  (The second part is here.)

First let’s look at the Cook Partisan Voting Index in the chart below.  I’ve arranged each state numerically by PVI from most Democratic Washington DC to GOP stronghold Utah.  The third column is that state’s electoral votes.  The last column is the running total of EV.  (As an aside, since Colorado is the only state with a PVI of zero and its 9 electoral votes include the one that get either candidate to 270, this is why it is highly unlikely that a spread in the popular vote by much more than a point will present a situation where the overall vote winner doesn’t also win at least 270 electoral votes.) 

Compare where the last two weeks of the election was fought four years ago with where it is being contested today.  Most of the remaining swing states in 2012 are on the Democratic side of Colorado, which serves kind of like a 50-yard line marker.  One could quibble at the margin over this list of states.  For example, four years ago McCain went into Pennsylvania late in the race when he saw that Ohio was shutting down on him and this year there is some indication that the race is about to extend into Michigan and Minnesota.  However, there’s no question that the current race has shifted up the scale of states in the favor of Republicans.  Four states where Obama marched strongly into the opponent’s territory four years ago (Arizona, Georgia, Indiana, and Missouri) are on nobody’s list of possible Obama wins this time. 

Consider also what the PVI means.  It is the amount by which a state is more or less Republican than the national average.  If for example, Barack Obama wins nationally by two points, we should expect him to get most states with a PVI of less than two.  If current national polls (according to the RCP average) are accurate, then Mitt Romney leads by about one point.  That means that if every state follows its partisan leanings, then Iowa and Nevada are on the bubble to flip to the Republican side.  That leaves Barack Obama to fight an upstream battle that requires him to keep those two states, plus either Colorado, which is one point more Republican than the spread, or Ohio, which is one point more Republican still. 

As an example, imagine that in 2008 Barack Obama had narrowly lost Ohio, and that every state shifted right by 4.6% (the amount by which he secured the Buckeye State).  That would have kept Florida, North Carolina, and Indiana in McCain’s column.  But still, it wouldn’t have mattered.  Obama would have won Colorado and Virginia and a total of 291 electoral votes.

Yes, Ohio probably goes with the ultimate winner, but because the list of remaining contested states has shifted north from four years ago, that means that Mitt Romney does indeed have more paths to victory available to him.  In fact, if yesterday’s polls are accurate, Romney has a lead of three points.  Sure, Obama may still win Ohio through the brute strength of his ground game there, but it probably won’t help.  To win overall, he will also have to hold onto another 61 electoral votes from the 7 Democratic-leaning states that that have a PVI greater than the national margin of 3 points.  That’s a lot of territory to defend.

***

One of the primary sources of Democratic optimism springs from Nate Silver, a former sabremetrician and blogger.  Four years ago he built a complex mathematical model that correctly predicted 49 of 50 state outcomes in the presidential race.  From his higher perch now at the New York Times, he confidently predicts that Obama has a 73.1% chance of winning and that he is likely to take about 294 electoral votes.  Read the comments on blogs both right and left; Democrats wield Silver’s predictions like a crucifix in front of a vampire of contrary poll results.  The headline in today’s (London) Telegraph, for example, adoringly calls him “the geeky statistician who is singlehandedly dismantling the myth of Mitt-mentum.” 

The first crack in Silver’s statistical prognostications appeared in 2010 when his early projections significantly undercounted Republican gains in the House.  Eleven days before the election he predicted that there was a 70% chance that Republicans would gain less than 60 seats.  They won 63.  That alone should be enough to remind observors that there shouldn’t be so much certitude about Silver’s 70% predictions a week and-a-half away from a vote.

But there’s another problem with Silver’s model; and it’s a problem that a sabremetrician should most studiously attempt to avoid:  It is based on the wrong statistic.

Sabremetrics is a portmanteau derived from a nearly 20 40-year old group known as the Society for American Baseball Research.  SABR sits at the intersection of mathematics and sports and its overwhelming desire is the search for the perfect measure of success.  These are the guys that found that instead of tracking batting averages or runs batted in, a far more accurate predictor of offensive baseball success is OPS: on-base percentage plus slugging.  (Moneyball, the story of Billy Beane’s use of sabremetric-like statistics to create a winning ball club in a small market team, is a great read on this subject.)

Silver’s model relies heavily on one metric:  the spread between candidates–and especially on the spread in state level polling.  There is a problem with both parts of that and with the model itself.

First, the model.  Most good time-series mathematical modeling is validated against past events in order to predict future unknowns.  Furthermore, greater weight is given to more recent events when verifying the model.  That is usually the smart way to model a problem–except, that is, when the recent event with the greatest weight happens to be an historical outlier.  The 2008 presidential election was an outlier.  It was the first election since 1952 when there was neither an incumbent president nor a sitting vice president on the ballot.  Since it was a contest unencumbered by incumbency, late-breaking undecideds were not predisposed by external factors to break one way or the other.  Going into election day, the RCP average showed about a 7-point lead for Obama over McCain and that’s the way it ended up on election day.  In other words, late-breakers broke to each side in about the same proportion as the decided portion of the electorate.  However, when there is an incumbent on the ballot, it is uncommon for him to get the late-breaking vote.  Look at the the chart from 2004 below:

After John Kerry gained points at George W. Bush’s expense during the first debate, and after John Kerry gave some of those points away when he made a stupid third-debate remark about Dick Cheney’s daughter being a lesbian, we see a distinct pattern over the last two weeks of the race.  Bush’s numbers are stuck.  Meanwhile, the challenger John Kerry saw significant gains from his depths four days after the last debate.  In the last two weeks of the 2012 race we should expect to see a similar pattern.  Why?  Why not.  So far we have seen a similar pattern between the Bush-Kerry race and the Obama-Romney race all the way up to this point.  Here’s the same chart, but with both the 2004 and 2012 races superimposed.  They are almost identical in shape for both incumbent and challenger. 

Furthermore, the pattern of this race bears no resemblance at all to what occurred in 2008.  Therefore, there is nothing up to this point that would lead us to believe that 2008 is a good predictor for today.

Nate Silver’s model tells you where the race is. (More accurately, his model tells you where the race was, as data is usually about 2-7 days old.)  But it doesn’t account for where the race is going.  In 2008 that wasn’t a problem as late-breakers broke proportionately.  However, two weeks before the 2004 election, Silver’s model would have underestimated the challenger’s gains, just as his model underestimated the gains of Republican House challengers two years ago.  His is not a dynamic model that takes into account historical patterns and thus, it is unable to project future results.  That doesn’t necessarily make it a bad model, so long as you keep in mind the limitation that he produces a snapshot of the recent past and not a vision of the future.  Based on the most recent historical precedent for the 2012 election, Barack Obama is not in good shape when he is already behind a challenger who hasn’t yet seen his late-breaking surge.

But there is another problem with Silver’s model.  By relying so heavily on the spread between candidates to predict results, it misses the point that not all spreads are the same.  An incumbent with a two-point lead who is sitting at 49 points is in much better shape than an incumbent with the same two-point lead but who is at 45.  The incumbent’s level of support, not the spread, is the most important metric in a re-election race.  That is because it tells you how safe the incumbent is from the effects of a last-minute surge.  On that metric, Barack Obama is not safe at all.  Since at least 2010, when the creation of Obamacare led the news, Barack Obama has struggled with his support, only briefly breaking the 50% barrier. 

Even more alarmingly for the President, dissatisfaction is both strong and stable while his support has been lukewarm.  Rasmussen has polled the level of those who strongly aprove of the President and those who strongly disapprove.  It’s not a pretty picture for the President. 

Rasmussen shows that for a long time well over 40% of the electorate has strongly disapproved of the President, but only about 30% now strongly approve.  Neither the support for the incumbent nor the strength of it appears to be a metric in Nate Silver’s model.  Therefore, the over-reliance on the spread as a predictor of success makes for a model that doesn’t account for hard ceilings and soft support.  If you were a well-funded challenger, where would you rather be two weeks before an election:  4 points behind an incumbent stuck at 47%, or 2 points behind an incumbent at 49?  Even worse for Obama, is that he is stuck at 47 and already behind.

The final problem with Nate Silver’s model is its over-reliance on open-source state polling.  Good polling is very expensive.  That is why so few organizations do it well.  Furthermore, even those who do polling well, don’t do it the same way for all clients.  As an Army analyst, I have hired one of the major national polling companies to conduct recurring nationwide polling of a sufficiently large sample size to get statistically meaningful state level results.  The cost:  $11 million.  Sure, that was in Iraq and not the United States, but labor costs there are much less than here.  Nonetheless, I bring this up to point out that a good poll of the scope and scale necessary to derive national level results from state level input costs more money than even the largest media organizations can afford.  Only the campaigns themselves have the kind of money to do polling right.  All the publicly-released polls cut corners in order to do it on a budget.  They use small sample sizes, and loose voter screens.  Nor do they go back and check sample responses against the answers from a sample of non-responders.  That’s why at the state level you get wide variability and large swings in most polls.

The optimism that Obama’s supporters project is unfounded in hard data and historical precedent.  Against the numbers, his fans point to GOTV, state polls that buck the national norm, and magical statisticians who assure them that all is well.  The only thing missing is the inevitable last minute appeal to the ghost of Harry S. Truman.

Barring an unprecedented shift, Barack Obama is unlikely to win the popular vote.  That alone is enough to place him in an electoral disadvantage, worthy of no higher than a 50-50 chance.  But still, there they are: Democrats and Intraders and their irrational exuberance. 

Tomorrow, Part II will explore the reasons why the reality-based community is having trouble coming to grips with their reality.