« Previous | Main | Next »

The Use and Misuse of Polls

September 24, 2007 10:02 AM

I made the following comments last week (Sept. 20) at the annual conference of the World Association for Public Opinion Research in Berlin – in part commenting on a keynote address by the sociologist Kurt Lang, and also reflecting on my own approach to the work we do. The session was entitled "The Use and Misuse of Polls in Politics and the Media."

-

The day before yesterday I got the following message on my BlackBerry: "Two simultaneous car bombs outside the health ministry in central Baghdad killed seven people and wounded 23 on Tuesday. There was also another car bomb attack near Sadr City in Baghdad; eight people were killed and 22 wounded. Three earlier car bombs killed a total of nine people and wounded 28."

A day previous I received a rundown of available video shots from that day's violence: "#052: Shows Baghdad's Jamila district – U.S. troops and Iraqi police at the site. Close up of wreckage of car. Grieving men hugging each other and crying.”

"#040: Shows wide shot of the bombing site. Remains of the car bomb. Man is shouting at his friends, 'Here comes the Americans, be aware 'cause they shoot randomly.'"

Receiving these messages over the last two days did not put me in an ideal frame of mind to hear Kurt Lang's keynote address at this conference on what polls cannot tell us about public opinion. Having just completed our fourth national public opinion poll in Iraq, I am far more focused at the moment on the essential nature of what polls can tell us about public opinion, often in a unique and irreplaceable way.

Let's take the case of Iraq. If we want to know what's occurring there – the living conditions, the extent of the violence punctuating daily life, the attitudes that spring up from these experiences, the implications for our place and policy in Iraq – there is nothing more fundamental than for us to go to a random, representative national sample of Iraqis and ask them. And then to independently and accurately report their answers.

I've had many a grizzled newsman waggle a finger at me and intone, "Polls are no substitute for good reporting." They're right, but for the wrong reason. Good polling is good reporting. You hear talk of "polls and pundits" when in reality good polling is anti-pundit. A good poll, honestly done, chases away the spin and speculation that exist and even thrive unchallenged in its absence.

Kurt's concern as I heard it is that public opinion as expressed in polls is substantively inadequate to direct public policy. That seems in his view to represent a fundamental flaw since he describes the effect of polls on policy as "the ultimate payoff from polling."

I don't see that as our purpose. Indeed to portray our efforts to know and understand public opinion as an enterprise whose ultimate aim is to impose majoritarian rule on our elected officials is to draw an exaggerated caricature. It's a good straw man to help support the point that we need to be mindful as well of other, anecdotal, expressions of public sentiment. I'm OK with that. Public opinion polls don’t end the debate on any subject. Salience matters. Expertise matters. Available options, political realities, economic constraints – all these and many more frame policy choices. But I'd hope we'd never argue that public opinion deserves anything less than a full setting at the table.

Our real aim, like any reporter's, is to cover our beat, fully and well. Covering the beat of public opinon enables us to give voice to those who lack it, to assess conditions and attitudes independently, and to inform our judgment across a range of issues with valid, reliable data. This is what we bring to the table with our polling in places like Iraq and Afghanistan – and every bit as much with our polling at home in the United States.

But here's where life gets tricky. I've been talking about good polling – good data. It lends authority. Substance. It elevates our reporting above mere anecdote into the realm of empiricism. Numbers carry a weight – a heft – that I personally suspect is no less than biblical in origin. You know the phrase the "writing on the wall?" Anyone remember what it said?

"Mene, mene, tekel, uparsin" – "numbered, numbered, weighed and divided" – "Numbered, the Lord has numbered your kingdom and come to its end; weighed, the Lord has weighed your kingdom and found it wanting; divided, the Lord has divided your kingdom and parceled it out." So ended the days of Belshazzar, grandson of Nebuchadnezzar, in Babylon – as it happens, in today's Iraq.

If such is the clout of numbers – we have to be damned careful with them.

News organizations, in particular, for far too long have indulged themselves in the lazy luxury of being both data hungry and math phobic. It is unacceptable – more than ever these days, when we swim in a sea of what I call manufactured data, produced not to portray public opinion independently and accurately but to promote a product, a policy or a point of view. Data cooked up on the cheap – often via the internet – to grab a quick headline, to please a paying client, perhaps innocently to misinform, perhaps actively to disinform.

At ABC News we try like hell to produce good research. But we burn just as many calories trying to kill bad research.

Years ago, with the support of management, we set up a polling standards and vetting operation. First we developed fair-minded but rigorous standards for what survey research we will and will not report. And then a procedure by which any survey being considered for air at ABC News goes through my unit first – or is supposed to – for a review in which we check it out, and either clear it for air or kick it out the back door.

It's the same as what reporters are supposed to with any other alleged news that comes in over the transom – we verify it before we report it, because that's our job. And doing our job protects our credibility. And our credibility is our stock in trade.

I've been on this little crusade for a decade now and it's gradually getting traction, because if there's nothing a reporter loves more than a good story, there's nothing a reporter hates more than being wrong. Except perhaps getting used. Purveyors of manufactured or just shoddy data use us in ways we must be aware of and protect against.

And so we are. My friends at The Associated Press – where I started my career 27 short years ago – have put in place polling standards and vetting modeled on our own. Last year my counterpart at The New York Times took me to lunch, picked my brain, reviewed our operation – and put in place polling standards and vetting in her newsroom. The new polling director at The Washington Post is my old assistant director at ABC News; I'm thinking he picked up a trick or two along the way.

In the end, Kurt's right – polls are imperfect. He's right – we need a range of intelligent, meaningful inputs to inform our judgment and to help our leaders shape policy. Journalists need – without question – to see that it's not all just numbers, to recognize the difference between good data vs. bad, reliable vs. unreliable, independent vs. manufactured – to make these distinctions and to report accordingly.

But for all that – after a week of military assessments, Congressional testimony, a presidential address, rebuttals from the other side – if you really want to know what's going on in Iraq – you could do worse than going over to ABCnews.com and taking a look at our latest poll there.

September 24, 2007 in Favorite Posts | Permalink | User Comments (3)

User Comments

Posted by: Michael O | Sep 24, 2007 12:13:39 PM

Posted by: John Nienstedt | Sep 25, 2007 8:18:10 PM

Posted by: stephen earley | Jan 7, 2008 12:30:10 PM

Post a comment

Name:

Email Address:

URL:

Remember personal info?

Comments: