« Previous | Main | Next »

Shooting Blanks

January 05, 2009 2:31 PM

When it comes to survey research, the Military Times is shooting blanks.

The publisher released an alleged poll last week purporting to show that most U.S. service members are uneasy with the prospect of an Obama presidency: “Six out of 10 active-duty service members say they are uncertain or pessimistic" about their new commander-in-chief.

Sad to say, the methodology behind this piece of work is about as reliable as a rusted carbine; FUBAR, you might say. It’s not even a survey at all, but a woefully incomplete census of Military Times readers. And in terms of their political and ideological leanings, the participants look nothing at all like what good data have found.

Per the Military Times summary: "In keeping with previous surveys, nearly half of the respondents described their political views as conservative or very conservative. Slightly more than half said they consider themselves Republicans, 22 percent independents and 13 percent Democrats."

That is radically out of line with a high-quality, representative sample of active-duty members of the U.S. Army conducted in 2004, which I discussed this past July. Thirty-eight percent in that survey identified themselves as conservatives, vs. "nearly half" here. Twenty-nine percent were Republicans, vs. "more than half." The Military Times itself allows that its respondents are older, more experienced, more likely to be officers and more career-oriented than the military population at large. It did not attempt to adjust its results to account for these differences.

To be fair, the newspaper's own report notes, “The responses are not representative of the opinions of the military as a whole. The survey group overall under-represents minorities, women and junior enlisted service members, and over-represents soldiers.” Nonetheless, it goes on to say that the results represent “a snapshot of the professional corps” that highlights skepticism Obama faces among military careerists. And its caveat – “not representative” – shows up in the 13th paragraph of its article on the results.

The question is whether these kinds of numbers reliably support any conclusion about anything. Look at how it was done: Military Times publications have about 250,000 subscribers. About 15 percent of them, when subscribing, provided e-mail addresses. The publisher sent survey invitations to that non-random subset. Those roughly 36,000 come-ons produced about 5,200 completed surveys. And about 1,900 of those participants said they were on active duty – just over three-quarters of one percent of all subscribers.

Still it’s not the number of respondents, but the lack of representative, random sampling, that knocks the treads off this tank.

We do know from that reliable Army survey in 2004 that officers are disproportionately conservative and Republican – but that enlisted service members, who account for the bulk of the population, are not. Relatedly, as I reported in my last posting, we know from the exit poll that veteran voters in 2008 were less Republican than is widely assumed.

The Military Times, though, tells a different story – one unsupported by good-quality empirical data. It's a shame, because there is much of value to know about the attitudes of the men and women wearing the uniform of the United States Armed Forces. So much so that they – and we – deserve real data in our attempts to know it.

1/9 follow-up:
1) The Military Times called to correct a number it gave us earlier; participants were drawn from its 125,000 subscribers, not 250,000.
2) A colleague notes that Army personnel accounted for 1,062 of the 1,947 respondents, and that 48 percent were officers and warrant officers, far above their actual proportion in the Army, 16 percent.
3) By e-mail I invited the editors of Military Times to reply to this post. I’ve not heard back.

January 5, 2009 | Permalink | User Comments (8)

User Comments

Posted by: Peace Train | Jan 5, 2009 2:55:51 PM

Posted by: scabby do | Jan 5, 2009 2:58:05 PM

Posted by: jhw539 | Jan 5, 2009 3:07:06 PM

Posted by: MSN exposed | Jan 5, 2009 3:10:29 PM

Posted by: jhw539 | Jan 5, 2009 3:11:46 PM

Posted by: jhw539 | Jan 5, 2009 3:18:15 PM

Posted by: hang | Jan 5, 2009 3:20:35 PM

Posted by: leftyintexas | Jan 5, 2009 3:20:40 PM

Post a comment

Name:

Email Address:

URL:

Remember personal info?

Comments: