The Numbers
A Run at the Latest Data from ABC's Poobah of Polling, Gary Langer
Gary Langer is director of polling at ABC News, where he's covered the beat of public opinion for nearly 20 years - conducting and analyzing ABC News polls, evaluating data from other sources and setting the news division's standards for poll reporting. Langer has won two Emmy awards for ABC's reporting of public opinion polls in Iraq, and The Numbers blog was honored this year as winner of the 2008 Iowa Gallup Award for Excellent Journalism Using Polls.
ARCHIVES
SUBJECT INDEX
RECENT POSTS
- Views on the Terror Trials
- Sarah Palin: Rogue for President?
- Chasing Feathers
- Tomorrow's Elections: An Obama Referendum?
- Schwarzenegger's Nastygram: One in 10 Billion?
- Executive Pay? Cut Away
- Pols, Polls and Pushback
- On Nuclear Iran, Diplomacy and Sanctions are Preferred to Hostilities
- Polling, Politics and Nobels
- The War in Afghanistan: Reassessment, Eight Years On
MONTHLY ARCHIVES
« Previous | Main | Next »
Work, Widgets and Perfect Polls
November 06, 2008 2:30 PM
The lists of “best pre-election polls” and the news releases trumpeting polling perfection are starting to roll in. They’re an exercise in vacuity of the highest order. And computed foolishly, as well.
The problem is not just in the silliness of these lists, but in the damage they do to the real task before us. Elevating the inconsequential through meaningless distinctions is not just a waste of time; it breeds misconception of the purpose and value of survey research.
The lists that I’m describing evaluate (to stretch the meaning of that word) pre-election polls on the basis of which came closest to the actual vote. They live in the fictional world of pinpoint accuracy in pre-election polling. They mistake estimates for something akin to laser surgery. They encourage the horse race lottery as a pollster’s path to fame and fortune. And they ignore the apparently minor matter of substance.
Don’t get me wrong: Given the intense if misguided focus on final pre-election poll estimates, we take ours extremely seriously, and I'm pleased we did very well again this year. Our estimate of Barack Obama’s vote was 53 percent; in the latest raw vote totals that’s what he got. Our estimate for John McCain was 44 percent; he got 46.
This demonstrates what ABC News pre-election polls have been demonstrating since 1984: We can interview a couple of thousand people and reliably report within a couple of points what 130 million are going to do. That is a rewarding affirmation of the principles of inferential statistics. It’s also as good as it gets.
At ABC News we don't stake our claim on the horse race; never have, never will. Yes, a well-founded estimate must be close; if it isn’t, something’s wrong. (See New Hampshire.) But even junk polls can have a good day, as some did this year. Three guys at the gas station can guess the horse race, too; that doesn’t lift them over the hurdle of reliability we demand. Polling is not a game of darts.
Turning the horse-race estimate into the golden ring of survey research values the result above the rigor of the methods used to reach that result. That can have the dangerous effect of encouraging even good pollsters to drop their devotion to empirical data and indulge in a last-minute round of spitball for the numbers that seem right, empiricism be damned. I saw an unseemly scramble in some data this Sunday and Monday that made me… wonder. Especially given the stability of our own daily results.
Even worse, making horse race estimates the sine qua non of pre-election polling does damage to what should be our true aim. Some on these “best poll” lists – even those that bothered with the niceties of rigorous methodology – asked essentially nothing but the horse race. This kind of commodity polling reduces the product to a numerical widget, with a shelf-life of 15 minutes and a value of 15 cents.
We poll at ABC to advance our understanding of the election and add meaningful narrative to our news coverage – not to dwell monotonously on the score of the game. We interviewed a total of 10,213 randomly selected adults in the 19 days before the election, and many thousands in the months before. As I blogged when reporting our final data early Tuesday morning, our purpose was to explore how the public came to its choices. We tried to identify the issues and candidate attributes that mattered, to measure voters’ responses to the thrust and parry of the contest, to get a look inside the campaigns’ own playbooks as they formed and made their appeals and to have in hand a reliable alternative to the spin and speculation that rush in when empirical data are absent.
We reported on the overwhelming role of the economy in this election and the way it cut through demographic groups in influencing vote preferences. We looked at the questions of Obama’s experience and race, McCain’s age and temperament, the tone of their campaigns, the long shadow of George W. Bush and the short one of Sarah Palin. We saw and reported the unusual gap in Democratic vs. Republican allegiance, the surge in early voting, the dramatic tilt among young voters and its influence on the outcome. Yes, our final estimate was substantive, too. But to mistake it for our purpose is to mistake the package for its contents.
Many of our colleagues, thankfully, undertook the same kind of work. Many producers of horse-race commodity polls did not. Nor did the online aggregators bother to distinguish between the two, or even to distinguish between what are and are not reliable data in the first place. Just when we need it least, they, like these best-poll lists, dumb us down.
The lists add insult to injury by going about their own task in a distorting way. Many of them gauge accuracy on the basis of the reported gap between the candidates. Through this absurd procedure a poll that had Obama at 8 percent and McCain at 1 percent would be knighted as dead-on, since it nailed the gap – ignoring the outstanding 91 percent of respondents. If you’re going to engage in this game, the appropriate measure is to see how close the estimate of each candidate's support came to his actual support.
Better yet, give the horse race a rest. Look for good methodology, and when you have it, look for the substance that separates real work from widgets.
November 6, 2008 in 2008 General Election | Permalink | User Comments (8)
You can follow this conversation by subscribing to the comment feed for this post.
I take it all these "best" lists are of popular vote only and not of the electoral college count. I mean, with North Carolina and Missouri still grey on the map, those numbers could change.
Posted by: Mortimer Snerd | Nov 6, 2008 3:05:19 PM
Gary:
Kudos as usual!
But as to getting the rest of the world to understand such a nuanced argument, good luck!
Mike
Posted by: Mike ONeil | Nov 6, 2008 3:55:38 PM
Gary,
Your polls on oil did not exist. We think you should use Alaska's oil.
Posted by: Julie McLean | Nov 6, 2008 10:37:37 PM
["We poll at ABC to advance our understanding of the election and add meaningful narrative to our news coverage – not to dwell monotonously on the score of the game."]
____
Huh ?
Exactly what is there to "understand" about the election (??)
There is a winner, and losers. The election is THE poll. Other pre-polling is completely superfluous to the official election... except as commercial amusement & airtime-filler.
Posted by: ronstor | Nov 7, 2008 8:39:25 PM
"Nor did the online aggregators ...bother even to distinguish between what are and are not reliable data in the first place."
That seems like a blanket statement unsupported by the facts. FiveThirtyEight.com went to great lengths to distinguish between reliable and unreliable data and published their methods. There are thorough discussions on Pollster.com about pollster reliability. Perhaps one can argue by RCP's and CNN's undisclosed methods about how they choose which poll is reliable that they are not doing what the first two are doing, but I'm sure they would argue against your claim too. Now as for the rest, perhaps you are right though I think their viewership is fairly low so why bother commenting on it.
OTOH, I think the overanalysis of the horserace aspect is exceedingly dominant. Although not entirely so, it is perhaps somewhat of a problem on, IMO, the two best aggregators: (FiveThirtyEight.com and Pollster.com).
But on the third hand ;-), I also think the small sample sizes of polls make lead to one-dimensional analysis of underlying factors. People don't vote because of only 1 factor and NO ONE does a statistically valid survey to try to accurately model voter's decisions based on 3 variables (and no, I don't mean the usual dumb demographics as proxies... I mean the real reasons people actually decide in their head to vote).
Personally I'd like to see a LOT less analysis and polling based soley on race, or sex or age.
Posted by: Jeff Winchell | Nov 8, 2008 5:01:24 AM
Despite your somewhat dismissive attitude to towards aggregators, the fact is that 538 picked the horse race at 52.3-46.2, as well as all states correctly, with the exception of Indiana.
This was based on a sophisticated methodology that took account of past pollster accuracy as well as sample sizes.
Even more importantly, during the race 538 (and Pollster) provided what was most certainly the most reliable indicator of the race while most polls were under or over-shooting or swinging around wildly. (I omit RCP since their opaque and subjective methodology reflects the worst habits of pollsters.)
The service provided by these online aggregators is, I believe, changing the game for individual polling outfits. Hopefully, this will make people more open about their methodologies and models, as well as deterring the media from the absurd cherry-picking of polls that is currently endemic.
Posted by: JohnC | Nov 8, 2008 5:06:53 PM
Why don't you do a survey or poll how the American people feel about a $25 billion dollar loan to the Big Three.
Also, who owns Chrysler, if Damler from Germany, what percentage is owner by American firm, and why would we consider a bail out for Chrysler.
Posted by: Evie | Nov 19, 2008 10:51:00 AM
ABC did a RV poll on Oct.20 which gave Obama a 52-42 lead. After allocating 75% of undecided voters to Obama, the spread becomes 55.3-43.2, with 1.5% for third-party voters.
In fact, the final 4 RV polls gave Obama a 13-point spread: 52.75-39.75%
The LV polls only sampled former voters, but there were about 20 million new voters and Obama won 71-27%, an 8 million vote margin.
Posted by: Richard | Jan 15, 2009 1:27:01 PM
Post a comment
