Edison/Mitofsky released their massively-detailed evaluation of the 2004 NEP exit poll last week. As Mark Blumental of Mystery Pollster observes, there is much useful data in the report, even if much of it is presented in fairly arcane fashion.
Among other things we learn the following:
1. There was, in fact, more error in this exit poll than in previous exit polls and data are presented on WPE (within-precinct error) that pretty much clinch the case. The increased level of error meant that, after the exit poll data were initially processed and corrected, they had to be further weighted, to a greater degree than in previous exit polls, to correspond to the actual election outcome. Hence the Kerry 51-48 popular vote win in the initial exit poll data that became a 51-48 popular vote defeat upon final weighting.
2. There seem to have been serious problems with inexperienced and poorly-trained interviewers, who were not able to administer the questionnaires in the fashion intended by Edison/Mitofsky.
3. There seem to have been serious sampling problems (“clustering effects”) with the national exit poll that, for example, contributed to the 44 percent support figure for Bush among Hispanic respondents, which Edison/Mitofsky appears to be backing away from.
The problem with all these admissions and the copious data in the Edison/Mitofsky report is that nowhere to they really make much of a stab at answering the underlying questions that are raised by the report’s findings.
1. If there was higher error than normal, why was that? What makes this election different and will the next election be even more different? If Kerry voters were more willing to be interviewed (or Bush voters less willing), which appears to be Edison/Mitofksy’s primary rationale for the high error level, why was this and why would this kind of behavior be particularly common in this election? And how does this square with the lack of variation in response rates across precincts that voted heavily for Kerry, heavily for Bush and inbetween? (See Mystery Pollster’s good discussion of this issue.)
2. If so many interviewers did so poorly, why was that? Did Edison/Mitofsky do an exceptionally poor job of selecting and training interviewers and, if so, why was that? Or is exit polling just getting harder to do, so interviewers that might have been adequate 4-8 years ago are not adequate today?
3. If the sampling problems were so serious, why was that? Were they worse than in previous years and, if so, why? Did Edison/Mitofsky do a poor job of dealing with these problems or was there nothing they could have done? Should we have been using aggregated state data instead of the national data for relatively small demographic groups all along like they imply we should with this year’s data? Or, again, is there something about the way the country’s changing that’s making always-exising sampling problems of the exit polls worse?
I don’t know the answers to these questions. But I think they deserve answers, hopefully sooner rather than later, while there’s still time to restore faith in an exit polling system that is now faced with widespread skepticism.
Thanks, Steady and cmac, for saying what’s been on my mind for some time.
There are days when I think I must be insane:
How can I look at the available information and conclude that it indicates Bush did not win this election, when so few others look at the same information and come to the same conclusion?
Given the fact that he didn’t win the 2000 election, either, and that questions have been raised about the 2002 midterm elections that, as far as I can tell, have not been resolved, what does this say about our country?
Is democracy finished?
Has Jeb Bush already been selected as our 44th president?
If my worst suspicions are true, then what is the endgame? How far does it have to go before we can talk openly about whether or not it’s happened?
Maybe I’ve missed it but noone seems to be talking about the increasing number of absentee voters. I was at a conference with Warren Miskofsky in July where he acknowledged that as absentee voting increases, at some point exit polling will become meaningless. You can’t interview someone who isn’t there.
I suspect they’ve tried to adjust the numbers by creating another poll of early voters but the margins of error in that type of poll have got to be off the wall.
It’s not hard to explain why Bush supporters were more reluctant to talk to the exit pollers—–a significant proportion of them were aware that support for the president was viewed as intellectually indefensible by the elites of the society and these Bush supporters cared about the opinion of those elites. So they stayed in the closet. I certainly did all I could in Minneapolis to make clear my contempt for any brain that could believe George Bush was a viable choice for president.
As you have been showing all week, a significant number of Bush supporters already supposedly diapprove of him and his policies in opinion polls. Would that they could have thought clearly on Nov. 2!
Occam’s Razor.
It’s so obvious. The exit polls were correct. It’s interesting to watch so many otherwise-sensible people twist themselves into pretzel shapes to avoid acknowledging this. If we continue to ignore it, though, how much hope is there for the future of our democracy?
Thank you for asking exactly the right questions about the E/M exit poll report.
As someone who remembers from my teen years of early political awareness the first uses of exit polling to call races (from about 1970 on) with virtually no tabulated votes in, I have been extremely dubious about the E/M excuses that for some reason air-dropped from nowhere into the 2004 race, exit polling has suddenly become unable to execute its most fundamental logistics competently.
I’ve always been a pretty sober and moderate person politically (thus my screen name), e.g., pissed at McCarthy for not supporting Humphrey in the 1968 general, working for Ed Muskie over McGovern, Hart over Mondale, etc., and was a politically appointee in the Clinton Administration, but in this election I’ve had to go with Emerson that “Some circumstantial evidence is very strong, as when you find a trout in the milk.” When we can see repeated overt manipulation by avowedly partisan Secretaries of State and Boards of Elections, I don’t see anything implausible about covert manipulation of vote aggregators by avowedly partisan private corporations with the motive, means, and opportunity to do so. I know you, Mystery Pollster, and many other strong Democrats disagree, and I respect that, but the trout are starting to overflow the pitcher.
Think about your prior post, “How Did We Wind Up With Bush?” and the several earlier others here about the lack of any significant bump up in Bush’s approval rating or his major policies despite his election. I also found Bush’s national margin odd in that while Kerry’s margin in the states he won seemed to mirror pretty closely the average of the last state-by-state pre-election polls, Bush’s margins in many states (e.g. MO, TN, AR, NC) seemed to go from 2-5% in the last pre-election polls to 12-15% in the actual voting. Of course these were not “battleground” states so no one was going to be looking very closely at why., and I’ve never heard a plausible explanation or even serious examination.
I fully appreciate and agree with the critiques of Kerry’s campaign from the standpoint of his unwillingness (or persuasion by loser consultants not) to do an effective and forceful Clinton-type rapid response effort against seven months of relentless Bush lies about him, and the failure to learn from Gore’s campaign how much harder but more necessary the corporate media’s stenographic flacking for Bush would make such a response. So it’s clearly not implausible that Kerry lost.
But the polls continue to make it awfully hard to explain November 2 as a genuine and meaningful statement of national opinion. So again, I especially appreciate your asking these questions and urge you to pursue answers. Without more transparency and greater public access to the mechanics of the voting process proper, exit polling is the only check we have on the integrity of the vote. If never-radical 50-somethings like me are starting to question the mechanics of the consent of the governed, it deserves at least this level of continuing scrutiny.