Having experienced some vertigo in sorting through polling data this year, I looked into some of the reasons for all the disparate findings, and wrote about it at New York:
There’s been a lot of talk about polling accuracy this election year, as there has been in the last five election cycles. Four of those election years (2012, 2014, 2016, and 2020) produced results significantly different from the expectations created by the best-known and (previously) most reliable outfits conducting national and state-level public-opinion research. In 2012, Democrats overachieved their standing in the polls, as did Republicans in 2016 and 2020. In 2018, the polls pretty much nailed the results nationally, though there were some misses in Senate races won by Republicans.
There are several reasons fears about polling accuracy are strong right now. First, 2022 is a midterm election where very small changes in the results could yield big consequences, thanks to the dead-even Senate and the tiny Democratic margin of control in the House (there are also many potential 2024 presidential battlegrounds where partisan control of the election machinery is up for grabs this year).
Second, there is a bit of residual trauma in the political commentariat about pro-Democratic polling errors before the astonishing victory of Donald Trump in 2016 and before his near-reelection (echoed by strong House gains by Republicans) in 2020. Pro-Republican errors in 2012, and the mostly accurate 2018 surveys, have been all but forgotten. Pro-Republican polling errors in 2022 special elections have all but been ignored or minimized.
Third, there are some pretty significant differences in what the pollsters are showing nationally and in individual contests this year. Consider the most-cited (and typically most reliable) indicator of the House national popular vote, the polling question known as the generic congressional ballot. The polling averages (per RealClearPolitics) on this indicator have been nearly even since the beginning of August. But one pollster, Trafalgar Group, has been showing Republicans with a five-to-eight point advantage in monthly soundings since July. (Another pollster Republicans love to love, Rasmussen, has consistently shown the GOP leading in the general ballot as well, though not by as large a margin.)
Similarly, Trafalgar has Republican Senate candidate Tiffany Smiley statistically tied with Washington’s Democratic incumbent Patty Murray, while all but one of the other polls of this race have Murray up by double digits. The Cook Political Report rates the contest as “Solid Democratic,” which means not remotely competitive. Then there’s the Pennsylvania governor’s race, which is turning into a Democratic rout, thanks to the incompetent campaigning and extremist antics of Republican nominee Doug Mastriano. Six of the last seven public polls have shown Democrat Josh Shapiro up by double digits. The exception? Trafalgar Group, which showed a statistical tie in mid-September.
There are some races where Tragalgar isn’t so much an outlier as one end of a pretty broad spectrum of findings. In the Ohio Senate race, for example, the RCP averages have Republican J.D. Vance leading Democratic Tim Ryan by 1.2 percent. Trafalgar Group shows Vance up by 5 percent, while Siena has Ryan up by 3 percent.
Now as it happens, Trafalgar got a lot of positive attention after the 2016 presidential election for accurately showing Donald Trump ahead in Florida, Michigan, and Pennsylvania, when nearly every other national polling outfit had Hillary Clinton winning all three states. And the same pollster wound up with a relatively low average error in 2020, particularly as compared to some of the big established firms like Monmouth, Quinnipiac, and SSRS (though Trafalgar Group founder and former Republican operative Robert Cahaly incorrectly predicted a Trump reelection, and erroneously showed him ahead in Arizona, Michigan, and Pennsylvania). While Cahaly stubbornly keeps his full methodology private (he uses live calling, robocalling, and online sampling), he is famous for claiming he adjusts his findings to reflect alleged “social desirability bias,” which mostly means putting a thumb on the scales of red voters who allegedly assume pollsters want them to support blue candidates. So Trafalgar assumes a general pro-Democratic polling bias that he aims to correct. You can see how that might or might not work out well.
Another common source of polling differences involves the basic sample. Often Republicans look better in polls of “likely voters” rather than “registered voters” or “all adults,” particularly in a midterm election with Democrats controlling the White House, a scenario that usually (but not universally) gives the GOP a turnout advantage. But at this stage of the election cycle, virtually all pollsters have already “switched over” to likely voter models, eliminating one artificial reason for differences in findings.
The thing about a recent record for polling accuracy is that it earns pollsters more business, so Trafalgar Group (and to some extent Emerson College, which did pretty well in 2020) is expanding its footprint this year, and its arguably affecting the polling averages more than in the past. But in some of the more heavily polled contests, averages probably still smooth out the differences between pollsters and their methodologies. In the red-hot Georgia U.S. Senate contest between Democratic incumbent Raphael Warnock and Republican challenger Herschel Walker, there have been eight published polls in the RealClearPolitics database since the beginning of August. Four (including Trafalgar Group, Emerson, Insider Advantage, and the University of Georgia) show Walker ahead, and four (Quinnipiac, Marist, YouGov, and Fox News) show Warnock ahead. The average puts Warnock ahead by 0.7 percent; in other words, the race is tied. For variety’s sake, you can consult the polling averages at FiveThirtyEight, which weights poll findings according to pollster accuracy and partisan bias. It’s still a tie race, with the projected vote share being 49.8 percent for Warnock and 48.6 for Walker. Indeed, if you are placing a bet on the contest the best wager is that neither candidate will win a majority and Georgia will again hold one of its notorious general election runoffs.
We won’t know until after the elections how to assess pollsters, or how to retroactively adjudge the impact on expectations of the very real differences in their findings. But at this point we can say that if Trafalgar Group’s polling is correct, there is a broader range of competitive statewide elections in play (if Patty Murray is truly in trouble, which Democrats are really safe?), and Kevin McCarthy can go ahead and put in an order for a Speaker’s gavel. But like partisan activists, a lot of people in the political prediction business will be white-knuckling it and composing their spins on and after November 8.
Read this, my fellow Dems and Kerry supporters and take heart all ye of faint heart (or hort, as Bush might say in his ersatz TX, affected, pork rind eatin’ accent):
http://zogby.com/news/ReadNews.dbm?ID=873
Shrub is in big trouble.
Well if one starts to believe it, then when the election is… umm what’s the word?…. yes, stolen with electronic voting from Diebold, a Bush inner-circle member who has sworn to “do everything in his power to elect G W bush” … folks won’t be so surprised… after all, they’ll say, he was ‘up by 13%’.
How do we know that the VNS exit poll results are the true partisan distribution numbers by which we should be weighting?
You wrote: “The only way you can produce a 13 point Bush lead with these internals is if you have quite a few more Republicans than Democrats in the sample–my guess is 7-8 points more. If you re-weight their sample to the 2000 exit poll party ID distribution (and I kind of have to do this, just to drive certain pollsters and their acolytes into a frenzy), you wind up with a modest Bush lead of 2 points.”
Is this what happened, or are you just guessing? Did Gallup poll REGISTERED voters or LIKELY voters? What exactly were the samples used, and if they were not representative, how so? Without this information, any substantive criticism of the Gallup results is meaningless; i.e., your numbers are less reliable than you claim Gallup’s are, and you’re blowing smoke.
There is a reason Gallup has more Republicans than Democrats in their poll. The reason is that Gallup, unlike other polls, doesn’t consider political affiliation to be a demographic. That is to say, if they – in their completely random dialing – get more Republicans than Democrats, then they assume that this simply means that there are more Republicans than Democrats in the population. There are obviously a lot of cons to this, but it is not totally devoid of pros, either. What is most important to remember with this is that Gallup has consistently done this for the entire time they’ve been taking polls, ie. they have not chaned their methodology for this election. So, the Gallup poll should be comparable to previous Gallup polls, but not so with other polls taken currently. And one more thing to ponder: Gallup consistently had Bush ahead in the popular vote in 2000, including in all of their polls in the final days of the election. If I were a Kerry fan, I wouldn’t pay any attention to ANY of the national polls. They really mean nothing, especially when you have states like Utah and Vermont, states where each candidate is up by ridiculously huge margins. The more important polls to look at are ones that look at the electoral map. There, sadly, Kerry’s chances are fading by the day.
the gallup poll will come in line with other polls the last week before the election. they did the same thing in 2000, they had bush up 13 the week before
election day and the poll right before the election he suddenly was down to a 2 point lead. they spin for the republicans until the very end and then switch to an accurate poll so they don`t foolish.
I Mourn the death of the Gallup Poll
I strongly applaud the actions here, but I also want to be sure that everyone here realizes the seriousness of what we are witnessing.
We are dancing on the grave of what was until recently a respected institution. As we point out these obvious and deliberat flaws, we drive nails into the coffin of this once venerable organization.
I am overjoyed to see people loudly taunting the right wing effort to “manufacture consent” for Dubya, our commander in thief, but I am unsettled by what it means for our future trust in the supposed ethics and impartiality of nationwide polls of any sort.
What Gallup has done here is akin to what Enron did to the energy trading business. I used to work for UtiliCorp in a Kansas City (now called Aquila if it’s still around). This was a company that I believe was destroyed by the criminals at Enron. UtiliCorp had flaws, but it was a good company. Guilt by association, however, is a horrible thing and virtually impossible to escape.
I am sure Colin Powell would agree with me right about now.
Some of the latest state polls
CA: K 53; B:40 (Rasmussen)
NY: K 53; B:41 (Rasmussen)
FL: K 47; B; 49 (Gallup)
PA: K: 46; B:42 (Quinnipac)
NJ: K:49; B: 46 (Rasmussen)
WA:K: 50; B: 43 (Rasmussen)
OR: K: 50; B: 43 (Rasmussen)
IL: K: 51; B: 42(Rasmussen)
MI: K: 48; B:45 (Rasmussen)
MA: K: 64; B: 27 (ARG)
Gallup: B: 54; K:41.
Quite a Bush landslide indeed! I would not believe these state polls if they were outliers. Actually most other state polls except the partisan ones confirm these results!
The constant bombardment by major media of “Bush with big lead” is erosive to Kerry’s position, and is done for that reason. Big media is backing Bush in a big way. This should end the myth of “liberal media” but it won’t.
I agree that Gallup has to know what it is doing, and is doing so for business reasons. They sell the name, and they do it to people who can afford to pay top dollar. It’s all about the name ID and the ability to sell it to big business.
Has anyone here posted the internals on the Gallup number? I read elsewhere that the spread between Republicans and Dems was even greater than the last poll — 43% GOP, 31% Dem.
It’s as if Gallup’s been reading all the debunking of its LV model, and decided, Okay, I’m going to screw around with the RV model and REALLY play with their heads.
The ABC poll, as many have said, is more disturbing, because it’s not quite as GOP tilted (though it probably cedes a few points) and because the approval rating puts Bush over the magic 50. The question then becomes, who’s got the right formula for approval rating this year? Such things didn’t matter in ’92 — when Bush’s rating was so low, it didn’t matter whether the high or low estimates were correct — or in ’96, when Clinton was so far over 50% a few points could be subtracted with no significant change. With this Bush, however, hovering around the 50% mark, it matters alot whether the low estimates (45-48) or the higher (51-52) have it right. I don’t see how we’re going to know before Election Day — unless the market crashes on oil/job/consumer confidence numbers, or Iraq truly explodes, sending Bush’s numbers decisively south in even his best polls.
Ruy: Great site – lot’s of good info. Question: What explanation has Gallup given for weighting their samples heavily with R’s? And WHY don’t they provide that information openly?
Clearly, Gallup must be aware of the contradiction in their numbers based on the state by state data. Why don’t they just say, “If turnout is 7-8 points in favor of R’s, we expect Bush to win by 13. If it turnout resembles 2000, we expect Bush to win by a nose, based on current polling data.” Wouldn’t that be better than misleading everyone?
Good Post, perhaps you should consider creative writing.
I agree.
The painfully obvious conclusion is that they are doing it to sell a position, not to get an accurate poll.
Ruy’s arguments are persuasive as far as they go but only look at one side of the equation. The ABC poll has Bush up by 7, the CBS poll by 8, the TIME poll by 4, the AP poll by 9. A couple of polls show a tighter race, but in the range of things, the Gallup poll is no more out of the pack than the Marist poll showing a close race.
I am willing to give Ruy the benefit of the doubt and suggest that there is MORE bias in the Gallup poll perhaps than some of the others, but why should he interpret it so that it supposedly comes out at the other end of the bell curve. The idea of party ID as a fixed rather than idea issue has been debunked on a number of blogs quite cogently, and while the arguments for consistency are not without merit, I don’t see why when a slew of polls come up with random samples (the four polls cited above together with Gallup average an 8 point lead — NOT a neck and neck race) that show a wide lead a pollster would have so much resistance in recognizing that the average is probably fairly close to accurate. The notion is that the polls showing a close race are right and all the others are biased and THAT argument, as a Democratic voter who has donated to and will vote for Kerry (in Mass where it won’t do much good) I don’t buy it.
Strategically, as I have said before, the repeated flipflop mantra, a mere fatuous spin, unconfronted can do what happened when Dukakis failed to fight back properly in 1988. The issue of running against the incumbent or the VP as a difference is not the determinative factor in this problem. UNLESS KERRY CONFRONTS THE FLIPFLOP ISSUE, DISMISSING IT AS THE “SPIN” THAT IT IS, AND POINTS TO THOSE ARTICLES THAT HAVE NOTED HIS CONSISTENCY SUCH AS THE SEPT 23 SF CHRONICLE PIECE AND THE SEPT 7 SPINSANITY PIECE AND EXPLAINS HIMSELF HE IS A GONER. He can use the ‘voted for it before I voted against it’ blooper to his advantage, by pointing out NOT HALF-HEARTEDLY THE WAY HE COUNTERED O’NEILL’S SLANDER IN 1971 OR RATHER INTRODUCING THE “STUDY” THAT HE WAS THE MOST LIBERAL SENATOR EITHER forcefully, like the NYU speech, that there were two versions of the bill. He must establish this point in the debate, probably the first one, or forget it. He’s losing even states Dukakis carried. Of course he can help himself in other ways, comparing the linear and one-sided thinking of Bush on Iraq with those who, without even thinking about strategy, call for “complete, unconditional, immediate” withdrawal regardless of circumstances. (My own views are close to but not the same as the latter). He can suggest that both views are simple answers but wrong. He can hammer at Bush for his father and Dole both in similar parroted ways trying to promote the SwiftBoat smear, while the candidates stay above the fray. He can lambaste Bush for trying to suppress dissent. He is NOT Dean and is NOT likely to lose it, although Bush might a little, if handled skillfully. BUT IF HE DOESN’T CONFRONT THE FLIPFLOP ISSUE IN THE DEBATES HE IS SUNK.
Only by really turning the issue around can he win, and all this pollyannaism about the polls distracts from that strategic reality.
Also, in the vein of realism, I must admit that it isn’t like what I have seen of Kerry for him to go on the offensive that way, although I am sure he knows how to do it in a firm and gentlemanly way if he goes into the debates with that strategy. But the whole campaign is one of default, of “Dukakissing” and THAT IS WHAT THE POLLS ARE REFLECTING. This would be the third “blown” election by the Democrats out of the last five. And the one who got in was a shill to get an otherwise impossible Republican Congress in 1994. Walter Karp in Indispensable Enemies understood well (despite a few off kilter specific theories) exactly how the two party system works. And any pundit who wants to do more than merely trying to justify the lying needs to recognize that.
I will add that a very big part of the hound that didn’t bark in this election is the virtual silence, especially until very recently, about the fatuousness of the flipflop spin in the media, not only the mainstream but the watchdogs. Factcheck.org had nothing along the lines of Marc Sandalow’s column in the SF Chronicle Sept 23 in over 60 pieces going back to when Bush first raised the spin in a speech. No one called Bush on it at the RNC, tho they were sharp enough to catch McCain on bashing Kerry for referring to the Iraqi “occupation” when so had Bush. But the voted for before voted against that was a centerpiece of Bush’s speech — bupkis. (Sandalow’s article, notable in this campaign for its unusualness and the obviousness of its points is at:
http://www.sfgate.com/cgi-bin/article.cgi?file=/c/a/2004/09/23/MNGQK8TI8O1.DTL) or just sfgate.com on the web. The paucity of press coverage both means that liberals don’t and haven’t picked up on this crucial spin & mantra, and Kerry has a greater lack of material to use to dismiss it as the “fatuous spin” that it is.
Instead of the “selling of the president” they should have books entitled “the arranging of the president”.
We need some analysis of the ABC/Washpost poll. It shows bush over 50% with RV’s. What has been the history of this poll all year? Why are the big name corporate polls all showing large edges for Bush while the independent ones show a tie or a small Bush lead?
I know among the grass roots that these polls are having a demoralizing effect on people bothering to vote. And it is propping up the Kerry is weak so let’s criticize him meme which can only serve to make him seem weaker to the the undecided. That is what they like about Bush and criticizing our own candidate is terribly counterproductive.
I am going to be running a phone bank into Fla. and I have people I may turn away because their carping at Kerry will leak negatively into their persuasion calls.
general poll comment:
a pollster called my mom today and asked her her party affilliation. “independent.” “GREAT. will you vote for Bush?” “no way in hell.” Click ….
They’re getting those numbers the old fashioned way … lyiin’ and cheatin’ and stealin’.
Chin up Rex. Read ya every post.
I wonder why Gallup is stopping at a 13 point spread between Bush and Kerry? Given the absurd assumptions underlying their methodology, Gallup may as well rejig their polling so that Bush can be ahead of Kerry by 20 or even 30 points? That would generate some interesting headlines! Oh heck, why not go for broke and say that Bush is ahead by 40 points. Or that Nader is now running ahead of Kerry. Any of those scenarios would have about as much credibility as Gallup’s current polling showing a 13 point spread in what virtually every other poll is showing as a very close race.
Gallup is a complete joke. I wonder how long it will take for them to come to their senses and change their methodology?
Here’s another way of thinking about the absurdity of a 13 point lead for Bush over Kerry. That would mean that over one-eighth of the people who voted for Gore now favor Bush. One out of eight, absolute minimum, and a hell of a lot more if there are many cancelling Bush voters who now favor Kerry.
So the Gore voter who now would vote for Bush, which has seemed to everyone to be as elusive as a unicorn, is now actually around every corner, and likely at any small gathering.
That really fits in with what people are seeing, doesn’t it?
Sometimes anecdote trumps “scientific” polling.