Category: Favorite Posts | Main

New Hampshire's Polling Fiasco

January 09, 2008 12:05 AM

There will be a serious, critical look at the final pre-election polls in the Democratic presidential primary in New Hampshire; that is essential. It is simply unprecedented for so many polls to have been so wrong. We need to know why.

But we need to know it through careful, empirically based analysis. There will be a lot of claims about what happened - about respondents who reputedly lied, about alleged difficulties polling in biracial contests. That may be so. It also may be a smokescreen - a convenient foil for pollsters who'd rather fault their respondents than own up to other possibilities - such as their own failings in sampling and likely voter modeling.

There have been previous races that misstated support for black candidates in biracial races. But most of those were long ago, and there have been plenty of polls in biracial races that were accurate. (For more on past problems with polls in biracial races, see this blog I wrote for Freakonomics last May.) And there was no overstatement of Obama in Iowa polls.

On the other hand, the pre-election polls in the New Hampshire Republican race were accurate. The problem was isolated to the Democratic side - where, it should be noted, we have not just one groundbreaking candidate in Barack Obama, but also another, in Hillary Clinton.

A starting point for this analysis will be to look at every significant Democratic subgroup in the New Hampshire pre-election polls, and see how those polls did in estimating the size of those groups and their vote choices. The polls' estimates of turnout overall will be relevant as well.

In the end there may be no smoking gun. Those polls may have been accurate, but done in by a superior get-out-the-vote effort, or by very late deciders whose motivations may or may not ever be known. They may have been inaccurate because of bad modeling, compromised sampling, or simply an overabundance of enthusiasm for Obama on the heels of his Iowa victory that led his would-be supporters to overstate their propensity to turn out. (A function, perhaps, of youth.)

Prof. Jon Krosnick of Stanford University has another argument: That the order of names on the New Hampshire ballot - in which, by random draw, Clinton was toward the top, Obama at the bottom - netted her about 3 percentage points more than she'd have gotten otherwise. That's not enough to explain the gap in some of the polls, which presumably randomized candidate names, but it might hold part of the answer.

The data may tell us; it may not. What's beyond question is that it is incumbent on us - and particularly on the producers of the New Hampshire pre-election polls - to look at the data, and to look closely, and to do it without prejudging.

...Wednesday afternoon p.s.: Some folks are suggesting that "late deciders" made the difference - a common explanation for poor estimates. But the exit poll doesn't support the notion. Remove voters who decided on Tuesday and the New Hampshire exit poll result is Clinton +2 – exactly her actual margin. (Among those who decided "just today" it was Clinton +3.) Next theory.

January 9, 2008 in Favorite Posts | Permalink | User Comments (886) | TrackBack (0)

IA and NH: A Quizzical Homage to Firstness

January 02, 2008 9:00 AM

The turning of the New Year is a dandy time for us to pay quizzical homage to the upcoming Iowa caucuses and New Hampshire primaries. Herein, a data-based effort to make some small sense of this quadrennial brouhaha.

Fair warning: It might not work.

It once seemed clear why New Hampshire mattered: Starting with Dwight Eisenhower, no one who lost the state’s first-in-the-nation presidential primary went on to get elected president. Granite Staters spent four decades invoking this factoid, proof positive of their power, prescience and uncanny ability to back the right horse.

Then it crumbled, as coincidences do – yet another alleged bellwether done in by the law of averages. The Democrats went first: Bill Clinton lost New Hampshire but won the presidency in 1992. And then George W. Bush did it in 2000. (Plenty, indeed, don't even get nominated: New Hampshire winners have gone on to the eventual nomination in seven of 10 contested Republican races since 1952, and in just six of 12 on the Democratic side. The total success rate, 13 out of 22, is a tepid 59 percent.)

At least there’s Iowa – only its track record is worse. Just once since Iowa fever took hold in 1972 has the winner of a contested caucus been elected president – Bush in 2000, when he went mano-a-mano with… Steve Forbes. The stronger contender, John McCain, sat it out, pretty much presaging the McCain/Giuliani/Thompson approach this time around.

The gotcha squad might double the score to say that Jimmy Carter took an Iowa victory to election in 1976. So make it two. But Carter didn’t really win… he came in second, to “uncommitted,” an option that represents one of the many oddities of Iowa caucusing. (Intramurally, counting Carter in ’76 and Ed Muskie in ’72 - he also actually lost to “uncommitted” - Iowa winners have gone on to take their party’s nomination five of eight times on the Democratic side, and just two of five times in contested Republican races – a 54 percent hit rate.)

The real fuss, it seems, derives simply from being first – the first caucus, the first primary – and not much else, kind of like the teen-ager who camps out for a week outside the Cajundome in Lafayette, La., to be first in line for Hannah Montana tickets.

There is a payoff: She gets a good seat; Iowa and New Hampshire get first crack at the winnowing process, not so much selecting the next president as culling untenables from the field. The compressed calendar, though, may now mitigate some of that effect.

It’s conjecture whether Iowa and New Hampshire winnow better or any differently than any other states would. There’d be some solace in their obstinate firstness, perhaps, if they looked a lot like the rest of the country, ensuring a mix of issues and voter groups that translate well to the national stage. But consider:

-Ninety-four percent of New Hampshire residents and 92 percent of Iowans are white, compared with 67 percent of the nation’s population.

-Just 55 percent of Iowans and 62 percent in New Hampshire live in or around a major population center, compared with 83 percent of all Americans. Four in 10 Iowans and New Hampshirites alike live in outright rural areas, double the national rate.

-Among the country’s 251 cities with more than 100,000 people, Iowa’s biggest rank 107th (Des Moines) and 190th (Cedar Rapids). New Hampshire’s biggest, Manchester, ranks 216th. Out of 251.

-Three percent of Iowans and 5 percent of New Hampshire residents were born in a foreign country, compared with 12 percent of all Americans. Of the 1.1 million immigrants admitted legally to the country in 2005, 4,536 settled in Iowa, fewer than one-half of one percent; 3,298 in New Hampshire, fewer than one-third of one percent.

-Iowa’s median household income, $41,350, is just under the national average, $44,468. New Hampshire’s, by contrast, is far above it – $55,589.

In the Iowa caucuses, moreover, hardly anyone actually participates. Total turnout in the 2004 Democratic caucuses was 122,193; peak reported Republican turnout was just over 108,000, back in 1988. This in a state that’s home to around 2.2 million eligible voters.

New Hampshire does much better in turnout: it peaked at 42 percent of the voting-age population in 2000, far more than is customary in other states. And pre-election polling suggests a lot higher turnout this time, with hot races in both parties. Still, Iowa and New Hampshire combined will have hundreds of thousands of voters in a primary process that ultimately will involve tens of millions.

There’s a particular oddity about the makeup of voters in the New Hampshire Democratic and Republican primaries: A heck of a lot of them aren’t Democrats or Republicans. In the 2004 Democratic primary, for instance, a whopping 45 percent of voters were registered independents. So were 32 percent in the 2000 Republican primary. That’s a lot more independents than participate elsewhere – partly because of voting rules, and partly because, hey, that’s New Hampshire. Bottom line: It can help insurgents (e.g., McCain 2000, Buchanan 1996, Tsongas 1992). But it can’t give them legs.

Another quirk is that Democratic turnout in New Hampshire is sometimes predominantly female – 62 percent in 2000, 57 percent in 1996 – while Republican turnout tends to be broadly (57 percent) male. And New Hampshire Republican voters are much less likely than those elsewhere to be evangelical Christians (listen up, Mike Huckabee); indeed, in our last poll there, 52 percent of likely Republican voters in New Hampshire favored legal abortion. You heard right.

Iowa can have its own oddities as well; the Democratic race in 2000, for instance, saw especially high turnout among union voters – one in three, compared with 15 percent of the population nationally. The commitment a caucus requires (this is not drop-in voting) makes get-out-the-vote efforts essential. And as far as pre-election polls modeling Iowa’s minuscule turnout… let us say: Aaagh.

Both these states look different on issues as well as demographics. The economy, for example, ranks much lower both in Iowa and New Hampshire than it does nationally. Republicans in both states rank immigration much higher, and Democrats rank health care much higher, than do their counterparts in the country as a whole.

In the end, though, to get the rapt attention of every political junkie in America, Iowa and New Hampshire don’t have to look like the rest of the country. Nor do they have to pick winners. They only have to go first. And that they will, just a few days hence.

January 2, 2008 in Favorite Posts | Permalink | User Comments (17) | TrackBack (0)

Teens and Steroids: Keep an Eye on the Stats

December 18, 2007 11:00 AM

Data are handy for making a point – so much so that it’s awfully easy to push them harder than justified. That just might have been the case in a couple of paragraphs in last week’s Mitchell Report on steroids in baseball, focusing on the drug’s use by teenagers.

“Some estimates appear to show a recent decline in steroid use by high school students; they range from 3 to 6 percent,” the report said. “But even the lower figure means that hundreds of thousands of high-school aged young people are still illegally using steroids.”

The prevalence data we see are somewhat different; the report looks to have misconstrued what's out there, and missed some updates from 2005 and 2007 alike. Actual current-use incidence is lower - and the "hundreds of thousands" may not be quite that.

In the most current of the two sources the Mitchell Report footnoted, use of steroids in the past year was reported at 2.7 percent among 12th-grade boys in 2006; that fits the lower range it gave. But the report says “students,” not just boys, and not just 12th graders. Annual use of steroids was much lower among girls (0.7 percent in the 12th grade); the total for all 12th-graders was 1.8 percent. It was lower as well among younger, 10th-grade high-schoolers, 1.2 percent. And recent use - in the past month, rather than in the past year - was lower still.

That’s from the 2006 “Monitoring the Future” survey by the University of Michigan's Institute for Social Research, for the National Institute on Drug Abuse. As it happens, just last week (apparently after the Mitchell Report was completed, though before its release), MTF released its 2007 data. Continuing a trend over the past several years, it found 2007 annual steroid use down to 1.4 percent among 12th graders, and 1.1 percent among 10th graders.

Since its peak years (from 1999 to 2002, depending on age), “the annual prevalence rate has dropped by more than half among the 8th- and 10th-grade males… and by 40 percent among the 12th-grader males,” Monitoring the Future reported. It also reported an increase in the number of 12th graders who see “great risk” in trying anabolic steroids and “a sharp drop in 2005 in the perceived availability of these drugs, very likely due to the Anabolic Control Act of 2004.”

Putting a sharper point on these findings, the MTF summary quoted Lloyd Johnston, its principal investigator: “While a number of states are considering implementing expensive programs to test student athletes for anabolic steroid use, the problem has been diminishing sharply,” he said. “It appears that supply control efforts, in combination with educational efforts, are having the intended effects.”

That’s a somewhat different tone from the Mitchell Report’s, which repeated that “hundreds of thousands of our children are using” steroids, adding, “every American, not just baseball fans, ought to be shocked into action by that disturbing truth.” If action means what Johnston calls “implementing expensive testing programs,” there seems some room for debate.

Indeed, comparing his steroid-use data with the characterizations in Mitchell’s report, “These are pretty good declines, and I’m not sure he’s taking them into account,” Johnston told me. “The problem has gotten substantially better.”

The Mitchell Report cited another source for steroid use data among teens, the Centers for Disease Control’s “National Youth Risk Behavior Survey.” Mitchell’s report footnotes the CDC data from 2003, which put lifetime prevalence at 6.1 percent. But, inexplicably, that seems to miss more recent CDC data, from 2005, which put it at a lower 4 percent. And the CDC measures lifetime use – ever in your lifetime – as opposed to MTF's measurements of annual use and 30-day-use (which, as noted, is lowest of all). The Mitchell Report employed the active verb “using.” That best describes 30-day use; at a slight reach it’s OK for annual use, but it’s not an accurate depiction of lifetime use.

As an aside, there also are differences in how the CDC and MTF studies ask about steroids that could contribute to different results. The MTF question looks preferable to us, since it defines steroids much more clearly.

So how about those “hundreds of thousands” of high-school-aged steroid users? The Census Bureau estimates there are 16,564,000 high school students in the United States (a generous estimate, since it includes all 9th graders). If we average the MTF 2007 “annual use” figures for 10th- and 12th-graders, we get an annual prevalence estimate of 1.25 percent, or 207,050 – barely there. Using 30-day use, it’s 0.75 percent, or 124,230 kids - plenty too many, but not hundreds of thousands.

Clearly any illegal, unprescribed use of steroids is wrong, and – particularly in the case of teenagers – cause for alarm. But the subject surely is worthy of sticking with a careful reading of the best and most recent data. In this, the Mitchell Report might keep in mind the truism of baseball itself: There’s always someone watching your stats.

December 18, 2007 in Favorite Posts | Permalink | User Comments (0) | TrackBack (0)

MOE and Mojo

December 03, 2007 10:58 AM

The Des Moines Register’s new poll, released Sunday, has Barack Obama 3 points ahead of Hillary Clinton in Iowa, which it characterizes as a “lead” for Obama. Our own ABC/Post poll two weeks ago had Obama 4 points ahead – and we called it “close,” not a lead.

What gives?

The answer is that it’s all about how far you’re willing to push the envelope. To the Register, we’re probably being too conservative. To us, the Register is going overboard. But there’ll be more of this to come in the weeks ahead, so it’s worth understanding how we get here.

A poll is not laser surgery; it’s an estimate. The reliability of the estimate is (in part) a function of its sample size. This is expressed as the margin of sampling error, and it’s customarily given at the 95 percent confidence level. For a poll of 500 likely voters, which was the sample size in both the ABC/Post and the DMR polls, that’s plus or minus 4.5 points. This means a candidate would need a lead of 9 points or more for us to say with 95 percent confidence that it’s statistically significant. Neither poll comes close.

However, the estimate a poll gets is in fact the likeliest true value, and the likelihood decreases as we move toward the extreme ends of sampling error. In fact we can calculate the level of confidence we can have that Obama really leads in either of these polls.

The answer: With Obama +4 vs. Clinton in the ABC/Post poll, we could have said with 77 percent confidence that – all else equal – he really had a lead. In the DMR poll, with Obama +3, the confidence level is 64 percent. Apparently 64 percent confidence is good enough for the Register to call it a “lead.” It’s not for us; nor was the 77 percent probability in our last poll.

Why not? One reason is that the customary confidence level in survey research is 95 percent, not 77, or 64. Another is that these probabilities only hold if all else is equal – and it isn’t. These estimates also are subject to non-sampling error, the likeliest cause of which is their estimate of who in fact qualifies as a likely voter. In our tighter likely voter model, more closely approximating caucus turnout in 2004, we didn’t have Obama +4, we had him +2, with 28 percent to Clinton’s 26 percent. The probability of that being a statistically significant lead was only 37 percent. Thus the prudent course was to call it close, which is what we did, and what we think it is.

Now on the Republican side, with 400 interviews, we had Mitt Romney with 28 percent support, Mike Huckabee with 24 percent. Huckabee had all the mojo – he was the guy making the move, as our analysis two weeks ago made clear. The probabilities still had us call the race close.

That Huckabee mojo looks to be continuing; the Register now has him at 29 percent support, to Romney’s 24 percent. The Register, again, calls Huckabee the leader. The confidence level 88 percent. Is that enough to call it a “lead”? It’s tempting. But before going there we’d want to see what turnout their likely voter model anticipates and what their other models (if any) show. Meanwhile, in our book, this one, too, is close.

As you can see, there's a bit of judgment in all this. The shorthand approach used by the AP is to say that when a candidate's numerical advantage doesn’t exceed sampling error, but is at least half of what sampling error demands, it can be called a "slight" lead. That's of course not what it is; it's really a possible lead in which we cannot be wholly confident.

All this underscores one of the fundamental points about pre-election polls: They are estimates. Even with good-quality methodology, the notion of pinpoint accuracy is a myth. And the reason we do them is not simply to try to puzzle out who's ahead – but to understand how and why the voters are coming to their choices.

December 3, 2007 in Favorite Posts | Permalink | User Comments (3) | TrackBack (0)

The Use and Misuse of Polls

September 24, 2007 10:02 AM

I made the following comments last week (Sept. 20) at the annual conference of the World Association for Public Opinion Research in Berlin – in part commenting on a keynote address by the sociologist Kurt Lang, and also reflecting on my own approach to the work we do. The session was entitled "The Use and Misuse of Polls in Politics and the Media."

-

The day before yesterday I got the following message on my BlackBerry: "Two simultaneous car bombs outside the health ministry in central Baghdad killed seven people and wounded 23 on Tuesday. There was also another car bomb attack near Sadr City in Baghdad; eight people were killed and 22 wounded. Three earlier car bombs killed a total of nine people and wounded 28."

A day previous I received a rundown of available video shots from that day's violence: "#052: Shows Baghdad's Jamila district – U.S. troops and Iraqi police at the site. Close up of wreckage of car. Grieving men hugging each other and crying.”

"#040: Shows wide shot of the bombing site. Remains of the car bomb. Man is shouting at his friends, 'Here comes the Americans, be aware 'cause they shoot randomly.'"

Receiving these messages over the last two days did not put me in an ideal frame of mind to hear Kurt Lang's keynote address at this conference on what polls cannot tell us about public opinion. Having just completed our fourth national public opinion poll in Iraq, I am far more focused at the moment on the essential nature of what polls can tell us about public opinion, often in a unique and irreplaceable way.

Let's take the case of Iraq. If we want to know what's occurring there – the living conditions, the extent of the violence punctuating daily life, the attitudes that spring up from these experiences, the implications for our place and policy in Iraq – there is nothing more fundamental than for us to go to a random, representative national sample of Iraqis and ask them. And then to independently and accurately report their answers.

I've had many a grizzled newsman waggle a finger at me and intone, "Polls are no substitute for good reporting." They're right, but for the wrong reason. Good polling is good reporting. You hear talk of "polls and pundits" when in reality good polling is anti-pundit. A good poll, honestly done, chases away the spin and speculation that exist and even thrive unchallenged in its absence.

Kurt's concern as I heard it is that public opinion as expressed in polls is substantively inadequate to direct public policy. That seems in his view to represent a fundamental flaw since he describes the effect of polls on policy as "the ultimate payoff from polling."

I don't see that as our purpose. Indeed to portray our efforts to know and understand public opinion as an enterprise whose ultimate aim is to impose majoritarian rule on our elected officials is to draw an exaggerated caricature. It's a good straw man to help support the point that we need to be mindful as well of other, anecdotal, expressions of public sentiment. I'm OK with that. Public opinion polls don’t end the debate on any subject. Salience matters. Expertise matters. Available options, political realities, economic constraints – all these and many more frame policy choices. But I'd hope we'd never argue that public opinion deserves anything less than a full setting at the table.

Our real aim, like any reporter's, is to cover our beat, fully and well. Covering the beat of public opinon enables us to give voice to those who lack it, to assess conditions and attitudes independently, and to inform our judgment across a range of issues with valid, reliable data. This is what we bring to the table with our polling in places like Iraq and Afghanistan – and every bit as much with our polling at home in the United States.

But here's where life gets tricky. I've been talking about good polling – good data. It lends authority. Substance. It elevates our reporting above mere anecdote into the realm of empiricism. Numbers carry a weight – a heft – that I personally suspect is no less than biblical in origin. You know the phrase the "writing on the wall?" Anyone remember what it said?

"Mene, mene, tekel, uparsin" – "numbered, numbered, weighed and divided" – "Numbered, the Lord has numbered your kingdom and come to its end; weighed, the Lord has weighed your kingdom and found it wanting; divided, the Lord has divided your kingdom and parceled it out." So ended the days of Belshazzar, grandson of Nebuchadnezzar, in Babylon – as it happens, in today's Iraq.

If such is the clout of numbers – we have to be damned careful with them.

News organizations, in particular, for far too long have indulged themselves in the lazy luxury of being both data hungry and math phobic. It is unacceptable – more than ever these days, when we swim in a sea of what I call manufactured data, produced not to portray public opinion independently and accurately but to promote a product, a policy or a point of view. Data cooked up on the cheap – often via the internet – to grab a quick headline, to please a paying client, perhaps innocently to misinform, perhaps actively to disinform.

At ABC News we try like hell to produce good research. But we burn just as many calories trying to kill bad research.

Years ago, with the support of management, we set up a polling standards and vetting operation. First we developed fair-minded but rigorous standards for what survey research we will and will not report. And then a procedure by which any survey being considered for air at ABC News goes through my unit first – or is supposed to – for a review in which we check it out, and either clear it for air or kick it out the back door.

It's the same as what reporters are supposed to with any other alleged news that comes in over the transom – we verify it before we report it, because that's our job. And doing our job protects our credibility. And our credibility is our stock in trade.

I've been on this little crusade for a decade now and it's gradually getting traction, because if there's nothing a reporter loves more than a good story, there's nothing a reporter hates more than being wrong. Except perhaps getting used. Purveyors of manufactured or just shoddy data use us in ways we must be aware of and protect against.

And so we are. My friends at The Associated Press – where I started my career 27 short years ago – have put in place polling standards and vetting modeled on our own. Last year my counterpart at The New York Times took me to lunch, picked my brain, reviewed our operation – and put in place polling standards and vetting in her newsroom. The new polling director at The Washington Post is my old assistant director at ABC News; I'm thinking he picked up a trick or two along the way.

In the end, Kurt's right – polls are imperfect. He's right – we need a range of intelligent, meaningful inputs to inform our judgment and to help our leaders shape policy. Journalists need – without question – to see that it's not all just numbers, to recognize the difference between good data vs. bad, reliable vs. unreliable, independent vs. manufactured – to make these distinctions and to report accordingly.

But for all that – after a week of military assessments, Congressional testimony, a presidential address, rebuttals from the other side – if you really want to know what's going on in Iraq – you could do worse than going over to ABCnews.com and taking a look at our latest poll there.

September 24, 2007 in Favorite Posts | Permalink | User Comments (3) | TrackBack (0)

Online Ballots: Let the Clicker Beware

August 28, 2007 2:05 PM

Feathers have been flying lately about a couple of online ballots posted on ABCNews.com after the recent ABC-sponsored presidential debates in Iowa. The alleged “winners,” if you choose to buy into these compilations of clicks, were Republican Ron Paul and Democrat Dennis Kucinich. Cue howls of outrage from their respective partisans, protesting that ABC hasn’t adequately reported the so-called results.

As my 12-year-old would say, chillax.

The reality is that these things are not polls or surveys, nor does ABC News identify them as such. They are called “online ballots.” They’re posted to encourage a sense of community and participation. But while the clicks they receive are tallied, they never are percentaged. And they’re supposed to carry this disclaimer: “Not a scientific survey. For entertainment only.”

Hold that thought – “for entertainment only” – as we dig a little deeper. Because ABC policy goes further, to require that any subsequent use of these ballots, beyond simply posting them, must note their vulnerability to outside manipulation.

That’s the real issue with online click-ins. Pointing out that they’re produced by self-selection and thus not reliably representative of any broader population is true, and fatal; but it sounds like a technicality – a talking point for the Sominex-sponsored Third International Colloquium on Inferential Statistics. More fatal (so to speak), because it’s at once easier to grasp and more immediately threatening, is the fact that these things can be, and often are, intentionally manipulated by groups or individuals with an interest in the outcome.

It happens all the time. People who want to stuff the ballot box just forward around the click-in’s URL, burying it in an orchestrated cascade of votes for the favored person, position or point of view. Others go a step further, building automated voting bots that jack up the tally for the pre-selected winner. Did this or that debate "winner," say, get clicks from 15,343 people – or from one person clicking 15,343 times? It can be impossible to tell.

Often we don’t know for sure when campaigns to manipulate online ballots occur; there aren’t always smoking guns. We have, though, found a posting on a meetup.com page, urging readers to vote for Paul in the Iowa debate ballot ("Ron Paul Winning ABC Debate Poll! Vote Now!") and to distribute the link elsewhere. The call to arms: “Lets keep RON PAUL ON TOP!” (sic). And Kucinich links to the ballot from his own campaign website; the headline reads, “Kucinich's Lead Keeps Increasing - ABC Debate Poll.”

None of this is remotely new. With the help of a hyperactive online community, Alan Keyes smashed the opposition as winner of a Republican debate in New Hampshire in December 1999, with 49 percent (against five opponents) in a Fox News/Vote.com online ballot. Sadly for the clickers, Quinnipiac University conducted a real poll (that is, a representative, random-sample telephone survey) on the same debate; Keyes got 13 percent, far behind John McCain (who, as it happens, went on to win the primary).

Nearly a year later, on Oct. 4, 2000, the day of the first presidential debate in the general election, Republican National Committee Chairman Jim Nicholson sent an e-mail to his entire membership, urging the nation’s Republicans to click in on ABCNews.com and CNN.com ballots to show their support for George W. Bush. That night, 58 percent in the ABCNews.com ballot picked Bush as the debate’s winner; by contrast, in actual polls by ABC News, Gallup and NBC News, he got 39, 41 and 36 percent, respectively. (Our prohibition on percentaging click-ins came later. Its aim simply is to make them less attractive to would-be manipulators.)

Some members of the FreeRepublic.com online community even have a name for this - they call it "freeping" - and when their side is losing they go so far as to "reverse freep," boosting the other side's vote to make the result so lopsided it's unbelievable. "REVERSE FREEP...the fix is in...they are cheating so vote for Kerry...99-1 invalidates online polls since they are invalid anyway," read one posting in the heat of the 2004 presidential campaign.

The gamesmanship goes far beyond election politics. For some it's a business: then-New York Attorney General Eliot Spitzer's settlement in a payola case against Universal Music Group in 2006 disclosed that UMG had paid a "request company" to "jack TRL (MTV's "Total Request Live") for Lindsay (Lohan)." I’ve got a thick sheaf of manipulated online ballots on everything from Tom Cruise to Don Imus to Drum Corps International. (Some joker jacked the results of the lineup ballot for their 2005 Masters of the Summer Music Games in Murfreesboro, Tenn. How low can you get?)

Rich Morin, former polling director at The Washington Post, wrote a priceless description of the issue nearly a decade ago, featuring, among others, Hank the Angry, Drunken Dwarf as People magazine’s Most Beautiful Person of 1998. (The piece still lives here, in what Morin presciently called “the world’s newest and most cluttered attic.”)

Today it’s online click-ins; in an earlier time it was 1-800 call-ins. Back in 1990, for instance, USA Today invited its readers to call in and say whether they liked or disliked Donald Trump. We love the Donald, came back the headline – a smashing 81-19 percent rout in Trump’s favor. At least until a correction appeared in the paper a month later, reporting that an audit had found that 5,640 of the 7,802 pro-Trump calls had come from precisely two phone numbers at an insurance company in Cincinnati owned by an admirer of Trump’s. Thank heaven for redial.

For all its internet fizz, then, this really is just old snake oil in new bottles. Sometimes it's fun and games. Other times it rises to a more serious level – misinformation, even downright disinformation. If it's your thing, click away. Just remember: When it comes to online ballots, if there’s a buck to be made or a point to be scored, chances are very good that someone, somewhere, has a finger on the scale. It's not remotely a scientific survey. And it is, decidedly, for entertainment only.

August 28, 2007 in Favorite Posts | Permalink | User Comments (29) | TrackBack (0)

Election Polls: What We're After

July 24, 2007 11:48 AM

A colleague here sent me a nice pointed challenge to our latest election poll yesterday: National surveys by themselves are "close to meaningless," he said, because they measure national preferences in what'll really be a series of state caucuses and primaries.

It's a fair complaint, and a serious one – because it cuts to the heart of just what our new survey, and its multifarious brethren, are all about.

It’s true, of course, that a poll of current preferences nationally does not tell us about current preferences in Iowa, New Hampshire or anywhere else. Without knowing who’s thriving in Iowa and New Hampshire, it's hard to predict who survives to South Carolina, much less who wins where on Mega Tuesday and wakes up with the crown on Feb. 6.

But wait: That – I hope – is not what our polls are after.

I like to think there are two things we cover in an election campaign. One is the election; the other is the campaign.

The campaign is about who wins. It's about tactics and strategy, fundraising and ad buys, endorsements and get-out-the-vote drives. It's about the score of the game – the horse race, contest-by-contest, and nothing else. We cover it, as we should.

The election is the bigger picture: It's about Americans coming together in their quadrennial exercise of democracy – sizing up where we're at as a country, where we want to be and what kind of person we'd like to lead us there. It's a different story than the horse race, with more texture to it, and plenty of meaning. We cover it, too.

We ask the horse race question in our national polls for context – not to predict the winner of a made-up national primary, but to see how views on issues, candidate attributes and the public’s personal characteristics inform their preferences.

The horse race is one question of 51 in our last poll, on a wide range of subjects. Rarely does it make the lead of one of the analyses my unit produces; we’d rather write about what’s behind it, as in our assessment, yesterday, of the role of “strength and experience” vs. “new direction and new ideas” as a campaign dynamic – a dynamic I expect to be central not only in the primaries, but even more so in the general election beyond.

The early caucuses and primaries do play a winnowing role in the campaign, although it’s an open question whether they do much more than eject subpar candidates who wouldn’t make the cut anywhere else, either (or, conversely, give short-term legs to the occasional sure loser). Nor do they necessarily anoint nominees (not even New Hampshire, since 1992 and 2000), much less presidents. And their impact this year, with Mega Tuesday a few weeks down the road, may conceivably be attenuated.

Whatever the horse race in the early-state campaigns, I’d argue that national polls add something unique and useful to our understanding of the election, by widening the field of our scrutiny. A few hundred thousand caucus-goers in Iowa and primary attendees in New Hampshire – largely homogenous states with few big cities, few minorities and few immigrants – can tell us what they’re thinking. But they can’t tell us what the nation’s thinking, and in covering the election – not just the campaign – that matters even more.

July 24, 2007 in Favorite Posts | Permalink | User Comments (5) | TrackBack (0)

If the Dunce Cap Fits, Wear it

June 25, 2007 11:40 AM

The latest Newsweek poll, purporting to show how dim most Americans are, puts the dunce cap on the wrong head.

The magazine is following a well-worn path – sneaking up on a bunch of well-intentioned survey respondents, springing a bunch of unrelated factual questions on them and then gleefully reporting that they don’t know jack. As a cheap polling trick, it works every time. As an exercise in meaningful measurement, it makes Paris Hilton look sage.

The first problem here is the all-too-common confusion of knowledge with recall. At best, recall is all that Newsweek measures – the ability to recite disassociated facts at the drop of a hat. It asks, in order, Who wrote "Pride and Prejudice?" Who won American Idol this year? What's the world's most popular spectator sport? And so on – 29 questions in all, jumping around from topic to topic like Ken Jennings on too much caffeine.

This has decidedly nothing to do with knowledge. Knowledge reflects the ability to think – not merely to recite information, but to use it to draw connections and build concepts. You can have deep knowledge, but weak on-command recall. You can have terrific recall, but little knowledge. Life is not a game show.

Imagine, for example, that I ask you to recite the 10 Commandments. Maybe you can. But if not, does it mean, de facto, that you’re not a religious person, that you don’t comprehend the basic tenets of the Judeo-Christian ethic, and that therefore you'll fry in hell with nothing to read but tattered dentist-office copies of a certain national newsweekly magazine? Maybe not.

The measurement, moreover, is as faulty as the concept. Opinion polls are called opinion polls for good reason. They're excellent at measuring opinion. They also happen to be pretty good at measuring behavior. But they're poor tools for measuring recall, much less knowledge.

The reasons are simple. Think of memory as a stack of file cards. Attitudes (and most behaviors) are right there toward the top – frequently used, thus easily accessed. Facts, by contrast, are buried deeper. (When's the last time you pondered Jane Austen?) A person participating in an opinion poll will gaily give you their opinions. Digging out facts requires cognitive work of a completely different order – and with the spaghetti on the stove and two kids squalling in the other room, it's far less likely to fly.

Facts and knowledge carry two other qualities that bedevil their easy measurement. One is that they're memory-activated; engage someone in a thoughtful conversation about the Middle East and they'll be able to remember and conceptualize more about it. Another is that facts are freighted with the weight of error. Opinions are risk-free; they can’t be right or wrong. Ask people a factual question and they’re more likely to beg off to avoid the embarrassment of a bad answer. Hence Newsweek tells us that 81 percent don’t know the name of the Supreme Court's chief justice. For some that's surely so. (And what of it?) For others, it more likely means that they just didn’t want to play this silly game.

Nor is it any wonder that more people can name Jordin Sparks than John Roberts. Sparks is a current recipient of the glare of popular culture; that she's simply better-known does not remotely suggest that her presence is more profound than anyone else's – Roberts most certainly included.

Yet another problem with Newsweek's approach is the unreality of some of its questions, such as one asking if the United States has captured Osama bin Laden. Some of the 11 percent who answered yes may well have been thinking, "OK, if you’re goofing with me, I'll goof with you." Not a helpful dynamic to create with 21 more questions yet to go. (Price of oil? Chairman of the Fed? Number of nuclear-armed nations?)

Imagine another approach: Say we ask our respondents to come to a testing center where we engage them in discussion of an issue of the day. Then we hand them sharpened No. 2 pencils and ask them to fill out answer sheets testing both facts and knowledge relating to that issue – with the promise, for good measure, of $10 for every right answer, and $20 for every well-argued concept. I'll bet that simple change in measurement would make our "dunce-cap nation," as Newsweek would have it, suddenly look a lot sharper.

June 25, 2007 in Favorite Posts | Permalink | User Comments (3) | TrackBack (0)