By Alan Abramowitz
Jeff Jones of Gallup recently posted this reply to critics on their editors’ blog:
Some consumers of the polls (including the Gallup Poll) have questioned poll results because party identification and other characteristics do not match the 2000 exit poll data. There are very good reasons why they may not match the exit poll data.
First, some treat the exit poll as a census. It is not a census, it is a survey based on sampling of voting precincts. There is a reason it is called an “exit poll” and not an “exit census.” That’s because it is a poll, and as such is subject to sampling variation and other polling errors just as any other poll would be. In fact, because exit polling relies on quota sampling (hand selection of survey spots according to population size and other factors), it has a higher degree of potential error than do the random samples on which telephone surveys are based. Thus, there is no basis on which to believe the exit poll numbers are in any way more accurate than any other number you get from a poll. They are all estimates. The one advantage of the exit poll is that they know everyone they interview is a voter, while pre-election polls rely on models to determine who is likely to vote and who is not. However, that does not mean their estimates are necessarily better, and they are definitely not error-free estimates of the electorate as many treat them. In fact, when multiple exit polls existed in the past, they very routinely differed in their estimates of the vote as well as their estimates of the demographic characteristics of the electorate. Even today the Los Angeles Times exit poll differs from the larger exit poll used by the networks.
Second, the exit poll measure of political party ID is fundamentally different from ours. We know that survey results can differ depending on how the data are collected. Our questions are read and responses obtained verbally over the phone. Their responses are obtained in self-administered questionnaires that present the questions in a visual format. Most survey research experts would be extremely cautious in comparing data obtained by a telephone interview versus that obtained in a self-administered paper-and-pencil questionnaire. That is in addition to question wording differences in the party ID question that can also have an effect on the results.
Third, a lot has changed since 2000. In the post-9/11 environment, terrorism has become one of the chief problems for government to deal with. The Republican Party has a large perceptual advantage on the terrorism issue. To assume that everything is as it was four years ago is a very risky assumption. While it is possible that in the end things could change once again so that partisanship looks much like it did in 2000, that is by no means certain or even likely.
My reply:
Mr. Jones’s comments are inaccurate and misleading. First of all, yes, the national exit poll is based on a sample. But it’s a huge sample–over 13,000 respondents in 2000. Because of its size, and because, as Jones acknowledges, the exit poll includes only individuals who have definitely voted, the margin of error should be much smaller than with pre-election telephone polls. Second, there is no reason to believe that a self-administered questionnaire would produce significantly different results from a telephone survey for an attitude as basic as party identification as long as the question asked is worded identically. The problem with the Gallup party ID question, as I have indicated elsewhere, is that it is a poorly worded question. Because of its lead-in, “in politics today,” the Gallup question, in contrast to the question used by the National Election Studies, the CBS/New York Times Poll, and the national exit poll, measures a combination of current political preferences along with long-term partisan commitment. Finally, and this is most important, there is simply no reason to believe that the distribution of party identification in the American electorate has changed significantly since the 2000 presidential election.
Indeed, an examination of national exit polls and CBS/New York Times polls conducted since 1992 shows that there has been no significant change in party identification for the past 12 years despite wars, recessions, and quite variable election results. Even in the immediate aftermath of the September 11th attacks, the CBS/New York Times polls found no noticeable change in the distribution of party identification. The distribution of party identification in CBS/New York Times polls this year, approximately a four point Democratic advantage, is almost identical to the average for the past 12 years. It is therefore highly unlikely that the distribution of party identification in this year’s national exit poll will differ substantially from that in the previous four national exit polls, all of which showed a Democratic advantage of between
3 and 5 points.
Unfortunately, Mr. Jones’s comments are typical of the head-in-the-sand attitude that the Gallup Poll has displayed in recent years in response to any criticism of their work. During the 2000 campaign, we would remind readers, the Gallup tracking poll was the laughingstock of the polling community as its likely voter results gyrated wildly from week to week and sometimes from day to day, producing a ludicrous estimate of a 13 point lead for George Bush on October 26th. Although Gallup quietly abandoned their tracking poll this year, they continue to display the same arrogance and insensitivity to criticism that we saw then. Evidently party identification is not the only attitude that is impervious to change.
~From North Carolina~
We are dedicated, we are truly pissed off, and despite many many problems at the polls due to the local Republican regime in Charlotte—like a 60-cent postage needed for the absentee ballots– and not telling anyone, listing the political affiliation on the ENVELOPE of the absentee ballot, and even –I kid you not– sending the ballot cards and omitting the Presidential choice card on my Mom’s ballot (90 years, old, voting for the first time, not sure if it’s Bush I or II who’s hijacked the country but damn sure she wants him OUT lol).
Lots of obstacles placed in our paths by locals, but that only makes us angrier and even MORE determined to oust Bush the Thief.
LOTS of newly registered voters!
We’re here, we’re mad as hell, and we’re GOING to push NC into the Blue territory!
“Third, a lot has changed since 2000. In the post-9/11 environment, terrorism has become one of the chief problems for government to deal with. The Republican Party has a large perceptual advantage on the terrorism issue. To assume that everything is as it was four years ago is a very risky assumption. While it is possible that in the end things could change once again so that partisanship looks much like it did in 2000, that is by no means certain or even likely.”
Prove it.
If Gallup had polled for current party identification, they would have presented their statistical support for this hypothesis by now. They apparently haven’t.
Thus, Gallup’s party ID premise, made by an organization whose business is statistical testing, (1) biases Gallup’s sample selection for political polling, (2) runs contrary to recent experience, and (3) is unsupported by statistical testing.
Something smells.
“Third, a lot has changed since 2000. In the post-9/11 environment, terrorism has become one of the chief problems for government to deal with. The Republican Party has a large perceptual advantage on the terrorism issue.”
Well, that might explain Gallup’s overstating the percentage of self-identified Republicans, but it doesn’t begin to account for Gallup’s understating the percentage of minority voters.
I think the comments about the relative inflexibility of party ID are right on course. Remember that the Nixon and Reagan landslides did not produce a large immediate shift of party ID to the Republicans. A lot of Democrats simply voted for Nixon and Reagan. They even coined the term “Reagan Democrats” to explain the phenomena of a landslide Reagan victory despite the Democratic party ID advantage. More Democrats showed up on election day, but they voted for Reagan.
My father is a perfect example of this. By all objective measures he’s a very conservative Republican. He listens to Rush Limbaugh and I doubt he’s voted for a Democrat since 1972. Yet if he were polled and asked his party affiliation he would say that he’s an independent.
Why? Because he’s viewed himself as an independent for decades and isn’t about to change now. It’s part of his self-image.
This is another important point. I strongly believe that the older the voter, the MORE RESISTANT they are to changing their party ID — just as other habits become more ingrained as one ages. But of course, this would suggest that younger voters whose party ID has not been fixed for years and self-identity is generally more flexible anyway would be the most likely to shift towards the Republicans. Yet we know that the opposite is the case. Younger voters are trending strongly towards Kerry with 18-24 year olds leading the way.
So, the Gallup data seems doubly suspicious, since the voters one would tend to think would be MOST open to changing their party ID show the STRONGEST tendency towards increasing rather than decreasing Democratic party ID!
For Mark,
As a follow up to your comments. I belive like you there will be a landslide for Kerry.
My reason being is the complete diregard of balance in the cable media news at this time. I was especially taken aback by CNN’s abrupt turn to the right in the past month. This tells me they are concerned by the possibility of a Kerry win and thus would not get further de-rugulatory favors as easily as with a Bush Government.
Secondly, if the polls are (atrificially) kept close (privately knowing that Bush support is actually down) they will in effect encourage the Rebublican base to come out to vote and maybe surpress thoes in the “Red” states to vote for Kerry.
I was at the Marlton, NJ rally for the president on Tuesday. I live locally. This is a very rebublican area, southern Jersey. You might be interested to know that there were more local supporters for Kerry than for Bush. For those with tickets, they came from Delaware, Maryland, New York and of course Pensylvania. Four bus loads where also brought in from Pensylvania for the event. Hardly overwhelming New Jersey support for Bush.
Finally, although this has nothing to do with polls, I thought it might be intersting and give an insite as to why Bush would even know to come to south Jersey. The GOP Chairman, Ed Gilespie is from Browns Mills (approx. 10 miles from Marlton). He graduated from Pemberton High School. Understand, Pemberton H.S. school serves Fort Dix and McQuire A/F base. Brown Mills is 99% rural apart from the military bases It is a combination of farmland and preserved pinelands (Wharton Forest is a National Forrest). It has one of the worst levels of education in the state, probably due to the constant turnover of the military kids. It is not an area that would produce someone with intelect, cruriosity or worldly sophistication let me tell you! Brown Mills is pure working class, whereas Marlton is wealthy and more importantly has two perochial schools, St. Joan’s and St Mary’s. This Catholic market is the Mmarket that I believe Bush was putting himself in the midddle of. I believe by coming here he was actually trying to shore up his base rather tht be competative here. New polls indicate New Jersey is a sold blue state for Kerry.
In addition to all the excellent points made so far, let me mention that alleging the weakness of the 2000 exit polls cannot explain or defend those current polls where the answer to “How did you vote in 2000?” fails to match the _known_ results.
Right on, Ruyster! Your diagnosis of the persistent Gallup error has been on target from the beginning. They do have their heads in the sand–at best. The deeper problem is that they have committed to a flawed policy but are unable to recognize (or acknowledge) their mistake and so persist in error. Isn’t this always a disastrous course?
I applaud your policy on posting. Politics is too serious a vocation to be handled at the level of insults and name-calling. We need an elevated level of discourse–thanks for providing it. I will continue to flog your blog at Politics1.com and myDD.com.
One love.
But, you guys are missing the point. You need to ask, how come Gallup has Kerry ahead in Ohio but trailing Bush by 8 nationwide?
It seems Gallup is putting forward a circular argument. They say that Gallup polling shows more people are Republicans (allegedly because of terrorism fears) therefore there are more Republicans in Gallup polls, which thereby prove that there are more Republicans, which then shows up in Gallup polling.
Hi there… this is a great site… quick question:
What gives with the WP / ABC polls? … they seem to constantly give Bush a 5-7 point lead, even among registered voters…
Also, does anybody have a feel for how many new voters the minimum wage initiatives will bring out in Fla and NV? … potential Kerry votes there, it seems to me…
Thanks!
eg
Ruy (or anyone else),
I apologize if this topic was broached already. But I wondered about the significance of the fact that two separate pollsters — Zogby and Survey USA — each have Kerry within 3 points in Virginia and within 3 and 4, respectively, in North Carolina. If these figures are accurate (and I tend to think they may be since they come from more than one source), I am wondering if this portends something really big on Election Day — ie., an electoral landslide for Kerry.
After all, Bush won N.C. by 13 points and Virginia by 8 points in 2000. If those large margins have been all but eliminated this time, it seems to me that there is a Kerry “surge” under way nationwide that would logically flip a lot of “red” states from 2000 to “blue” on Nov. 2.
Am I reaching here? Wishful thinking? Or does this seem to be a real possibility given this data?
— Mark
“Third, a lot has changed since 2000. In the post-9/11 environment, terrorism has become one of the chief problems for government to deal with. The Republican Party has a large perceptual advantage on the terrorism issue.”
Is it just me, or is this a downright bizarre thing for a pollster to say in defending his methodology?
The assertion may be true, but it doesn’t prove that voters are more likely to self-identify as Republicans for that reason. Other factors might push them the other way, the net result being a wash (as seems to be the case).
If Gallup has actual polling data to support the assertion that more people call themselves Republicans – for whatever reason – they ought to just stick with providing the relevant polling data.
As it is, though, this sounds a bit like faith-based polling – “there MUST be more Republicans, so we’ll weight our sampling on the assumption that there are.”
— Rick Robinson
The exit poll has an enormous advantage over telephone polls — more important, in my opinion, than that you know the voter has voted — that Jeff Jones chooses not to discuss. It is free from many, maybe most, of the systematic biases that arise from telephone polling. Sure, you can’t logically reject the possibility that differences in party ID measured by the Gallup poll reflect real changes in public opinion, but you have to explain why all of the many other possible causes aren’t at work.
I’ve made a lot of postings here recently — I thought this would be a good place to summarize reasons why I think Kerry will get more votes than polls indicate. For completeness, I’ll start with some generally admitted, on which I have nothing new to say:
1) Undecideds generally break against incumbent.
2) Heavy registration of new voters; non-voters are mostly Democrats.
3) Greater commitment of Democrats in this election. (Some deny this, but I don’t.)
Now to further points, which nearly all boil down to the fact that low-income voters, who tend to be more Democratic, have lower poll response rates, and all methods of correcting this bias (other than deciding in advance what percentage of Democrats and Republicans will vote, and guessing right) only partially fix the problem.
1) Gore did better than polls in 2000. This was arguably due to poll bias, not to last-minute changes in preference.
2) Voting preferences vary this year in opposite directions with education and income. In past years, demographic weighting for education was an indirect (and partial) method of correcting for income biases in response rates. This year, education-weighting makes the income biases worse.
3) In recent past elections, undercounting of active military in polls imparted a Democratic bias to the results which partially canceled the Republican bias created by undercounting of low-income voters. This year, the active military is voting Republican by a much smaller margin.
4) Increasing numbers of low-education Hispanic and Asian voters in the electorate, who have very low participation rates due to language barriers and long work hours. When poll demograpics are corrected using parameter weighting, the high-education and high-income Hispanic and Asian respondents who (unlike high-education blacks) vote Republican get heavily weighted. (The ABC poll uses a better weighting method that avoids this mistake — but it then does even worse by not correcting for Hispanic status at all.)
5. My impression is that polls are using more weighting parameters to adjust demographics. This is the mistake of overparameterization, which increases random error in the results and may possibly also lead to undercorrection of the variables that matter. (This is speculative.)
6. Many polls correct age demographics using an 18-29 category. My impression is that 18-24 year olds are heavily pro-Kerry and heavily underrepresented, while 25-29s are fairly evenly split in preference and less underrepresented in samples. Some missing 18-24 year old Kerry voters are replaced by 25-29 year old Bush voters in the weighting process.
7) Cell phones. Note that the cell-phone-reliant are more urban, and therefore more pro-Kerry than the average of their age cohort, because cell phones don’t work in some rural areas.
The way Gallup poses its party ID question does indeed seem bizarre. It seems effectively designed to encourage as whimsical a response as possible to the question of a voter’s party ID.
For the life of me, I can’t see why posing the question in that way would be thought more useful than to do so in a way that gets at much firmer, more permanent commitments on the part of the voter.
I’ve got to seriously wonder whether their question isn’t so posed precisely to give them a defense against the charge that they are overrepresenting one or another party, and for no other basic reason.
He didn’t bother with the other piece of related criticism, either. Not only do Gallup’s reported mix of party affiliations not match previous exit polls, they fluctuate wildly on a week to week basis. Gallup has been extremely incosistant in this matter. It is not just that they have to posit a large shift in affiliation since 2000, they have to stand behind large shifts every couple of weeks and have offered little or no reason to think this happens.
Four years ago today, Gallup had it
Bush 50%
Gore 41%
Actual vote
Gore 48.0%
Bush 47.9%
Four years ago, Gallup did the same thing, showing a wide Bush lead until the last two days, when they closed it to two points or so.
Gallup has joined in something like 200 “partnerships” with business since it morphed from polling group to something that sells its name as a marketing gimmick.
Check their site and you will see how they form these so called partnerships with business. They rent their name out for marketing is what it comes down to.
(1) I have just been banned for some reason by the webmaster at eschaton — people might check out my lengthy posting under the section on the ‘white catholic’ vote and see if there’s anything offensive.
(2) On Gallup — what is his explanation for the divergence of the Gallup polls from the ones cited here both on party ID and on overall results? It was very telling that the Repubs said that Gallup was the only poll they trusted!
Let’s see: AP-Ipsos; Marist; Economist; Pew; Zogby; Harris; and NBC-WSJ; Dem Corps — that’s EIGHT polls showing the gap within the MoE, with the average almost exactly a dead heat (less than 1% difference and no variation) and they don’t see ANYTHING wrong with their results?
My suspicion is that these are, if need be, to make any Diebold business, should that be necessary, seem more plausible.
(3) I am of the strong opinion, following the flipflop spin (Dem Corps says 60% identify Kerry with that label), then the Bai spin (still unconfronted as it needs to be, just as the flipflop spin never was), that all the extra protest of the Sinclair stuff of their “news” infomercial about “Stolen Honor” will only make more people watch it!
(You can get a transcript of stolen honor — WHAT A HATCHET JOB!!!! — at:
http://www.dailykos.com/story/2004/10/20/20312/528
I say that dems and progressives should demand that some network — since Sinclair won’t do it — air a one hour piece using the sections from “Fahrenheit 9/11” about the Bush Administration on TV in the same areas. When the TV networks balk, then they should really be roasted for it. I believe in MORE speech rather than restraining speech both in principle and strategically. 9/11 would be SO MUCH more powerful a program than the other crap, it would give the Dems back CONGRESS!
Like we really lived in a nonmachine society, not based upon tramoya!
Let’s assume for the sake of argument, that what Gallup says is true, it still does not explain a) Why their assumptions vary wildly from the turnout in terms of minorities and other groups in 2000 (are we to believe that blacks were not black in 2000?) b) it does not answer in my mind how they were so far off the mark in 2000 if their methogology is so accurate.
I just had to put my two pennies worth in about Gallop, so please forgive my rather crude attempt at adding to the discussion.
I was always skeptical of Gallop (and Strategic Vision which I am gratified to see that MOST now add the “Rebublican sponsored organization” after quoting their polls) so I investigated them a little and I came up with these facts:-
Polling is done from only three centers in the country, Nabraska, California and….TEXAS!
Owner of Gallp is a Texan, and a registered Rebublican. He bought the company in 1990.
Mr. Newhart, the public face of Gallop, hails from Houston Texas as well, was a partner in a PR firm and was also a radio show host there. Has admitted in his new book “Polling Matters” that the information contained in their collected data is somewhat subjective and open to interpretation and that most companies don’t have necessary “media editors’ to disseminate the information corectly. He did say that CNN & USAToday do have “media editors” (dare I suggest the obvious, that they might be editing too selectively)!
Finally, one has to buy the data from Gallop, this would preclude most people from seeing the method by which the information has been obtained. What the actual slant of the questioning was, the actual demographics etc. And quite simple is what they purport to be true, true!