I spend a major chunk of my days locked in mortal combat with data that don't meet our standards for validity and reliability. One of today’s entries cuts pretty close to home – so close that a response may look like I’m carrying water for my employer. Stick with the evidence, though, and see what you think.
This item comes from an outfit called Public Policy Polling, which produces robopolls, in which calls are made by an automated dialer and a recorded voice asks people who go along to press buttons on the telephone keypad to indicate their views. (I’ve heard that some regard the phrase “robopoll” as pejorative. Sorry.)
In our ABC News polling standards we don’t regard autodialed, pre-recorded polls as valid and reliable survey research. Our concerns include noncoverage, nonresponse, lack of respondent selection, nonvalidation, opaque weights and the absence of a persuasive, independent literature evaluating the approach. Some other news shops – the few that, in my view, take coverage of public opinion seriously – share these judgments.
The PPP poll at hand asks registered voters to rate news organizations: “Do you trust ABC News? If yes, press 1. If no, press 2…” It finds greater trust for Fox News (49 percent, or 58 percent when you winnow down to those who have an opinion) vs. 31 percent for my employer, ABC News (40 percent among those with an opinion).
OK, ouch. But honestly my problem’s not with the results, but with the methodology. And I see support for my concerns in other data included in PPP’s report – on items such as partisanship, ideology, 2008 vote, the age and even the sex of participants.
Thirty-five percent of respondents in this survey pressed “2” to say they’re Republicans. Good for Fox, since as PPP notes, trust in Fox spikes among Republicans. But in our own polling, the number of registered voters who describe themselves as Republican is far lower – 25 percent in our latest poll.
Don’t like ABC/Post polls? Look at the Republican component of any of a long list of other independent good-quality surveys. The range among the general public (not registered voters, but there’s very little difference) is from 19 percent to 28 percent in recent polls by NBC/Wall Street Journal, CBS, AP-GfK, the Kaiser Family Foundation, the Pew Research Center, USA Today/Gallup and others. Just one came close to 35; another robopoll.
Try ideology. Fourteen percent in the PPP data pressed “1” to identify themselves as liberals. In our ABC/Post data, again among registered voters for the strictest comparison, it’s 25 percent. And our data on liberal self-identification have been steady for years.
There’s also 2008 vote. In the PPP poll, 47 percent pressed “2” for having voted for Barack Obama in 2008. In our last poll, among registered voters, it was 54 percent. Among actual voters, Obama, you might recall, got 53 percent of the vote.
Fifty-seven percent of the PPP participants pressed “2” to indicate they were women. Among registered voters in our data, 53 percent are women. And just 9 percent pressed “1” to indicate they were age 18 to 29. Sixteen percent of registered voters in our results are that age. A problem here may be the apparent absence of cell-phone-only respondents. I see no reference as to whether they’re included in the PPP sample, but it is illegal to autodial cell phones.
Some defenders of robopolls will point in another direction, at those that have accurately predicted election outcomes. From my perspective, accurate modeling is not an adequate stand-in for good polling. For reliability and validity alike, what matters are sound methods. And what matters next are substantive measurements, not horse-race bingo.
I could pick further. The questions PPP asked didn’t include the alternative proposition – even just minimal balancing, such as “or not?” That could have biased all the results in favor of trust, something my employer may not love hearing, but the case nonetheless. What it does indicate to me is a lack of awareness of the basic niceties of question wording.
We also have other data on hand, not asking the same question but measuring basic views of the TV news media. In a Pew poll last October, among people who had an opinion of ABC News, 39 percent called it “mostly liberal,” 17 percent “mostly conservative” and 44 percent “neither in particular.” For Fox the numbers were 16-55-28 percent. Fewer perceived Fox as avoiding an ideological slant than any of the other five networks tested.
Pew, back in May 2008, had another approach, a 1-4 scale of believability. Again among those with an opinion, local TV news and CNN did the best, both rated as believable (a 3 or 4) by 70 percent. ABC News came in at 64 percent; Fox, 59 percent.
There are plenty of ways to measure trust in the media, and in any research, the questions matter. But by my lights, methodology always comes first.