Today through Saturday, I will be over at Josh Marshall’s Talking Points Memo guest-blogging while Josh is taking a well-earned vacation. So please stop by his site–as most of you do already, I imagine–to catch my latest thoughts on things political.
I will be back at Donkey Rising on Sunday with my usual data-obsessed ruminations. For all those who clicked through from Josh’s site to check out DR, I hope you’ll be visiting regularly when I’m back at my regular post.
Yes, Bill, the 2000 Florida exit poll informed me of a great many things. The fact that news enterprises misused it to jump to erroneous conclusions about who had won the state in the hours after the polls closed does not detract from the wealth of knowledge contained in the exit poll results.
Dennis, as to your question, why pollsters weight for anything, I think the answer lies, at least in part, in the fact that, of course, the techniques they employ do NOT truly create a random, REPRESENTATIVE sample. Asking questions in a phone poll only of those who choose to respond introduces all kinds of biases. I would expect that, simply to adjust for these quite predicable biases, one must start fudging weights in any case (for example, do more women or men answer the phone?)
Even if one had a truly random, truly representative sample, I would expect that weighting the results for known demographic facts — e.g., gender representation — WOULD make for more accurate polls in general. If, for example, in a truly random, truly representative poll 55% happened to be women, and 45% men, wouldn’t it make the poll LESS error prone if those numbers were properly weighted? What the underlying mathematics of this should be left to statisticians, but intuitively it’s pretty obvious that such an adjustment WILL diminish the likelihood of error.
Greg, Frankly: I read the article, but the article didn’t answer my question, which is why weight for anything? I’m basically a math/cs guy, so I take my randomness pretty seriously, and since in a random sample the demographics shouldn’t be too different from what you expect anyway (not to mention that doing things not-entirely-randomly messes up the theorems that make it valid), what’s the reason anyone would weight for anything?
Frankly: I suspect that the idea you had about how weighting must work is true; tossing people out seems to be the best way to preserve the sanctity of the sample, but I’m wondering if it’s more complicated than that.
Ruy and Josh do different things, providing very different types of information from very different sources, both first class and superb. Some people need to put their sharp tongues back in the sheath and leave them there. How about of bit of civility and respect. Save your knives for Karl and George, deserving beyond measure.
While it’s good that Ruy is sticking to what he knows, it’s awfully boring for a lot of people. So what is Ruy’s opinion on non-poll news? That’s more what TPM is for, and I for one would like to see that.
Dennis,
I do think it makes perfect intuitive sense to weight samples based on hard demographic data in cases in which the relevant classifications of the sampled population is indisputably accurate.
Gender would seem to be such a variable. It’s exceedingly unlikely a poll taker could get such a thing wrong, and it most definitely affects probability of voting preferences. Likewise, geographical classifications would seem to be highly reliable and relevant. Income and age seem to me on the other hand pretty unreliable, given their dependence on self report. Party ID has not only the self report problem, but the more significant problem that it can vary from day to day.
I’m not sure how exactly the “weighting” affects the way in which error and other quantities get calculated, but my guess is that the MOE is very little affected, since it is so highly dependent on the sample size in any calculation. Certainly the MOE’s I’ve seen seem to be directly calculated from the reported sample size, as if no weighting had taken place. Of course, the whole point of the exercise is to reduce the likelihood of error, so I’m not sure quite how to understand all of this.
One question I’ve always had is, how, if these numbers get weighted, do the pollsters come up with the nice integer numbers they do for their polls? Do they gather much larger samples than the numbers reported, and then just throw out, say, a certain number of women if there are too many women? Do they do the same for all categories? Are the integer numbers reported just a convenient fiction, because the floating point numbers genuine weights would impose would be embarrassing to explain to the general public?
Dennis – as the LATimes article stated, political operatives typically like to see polls weighed because they believe that party ID is more fixed and static and therefore, polls need to be adjusted to reflect that view. Independent posters on the other hand, believe that party ID is more fluid and therefore, you might very well find a +13 on Democrat party ID because lately the news have been heavily anti-administration.
(No place to comment at TPM, so I’ll do it here)
Re Fox polling: So, either Fox News’ poll has it right, and Gallup AND American Research Group AND Quinnipiac all have it wrong. Or….
Can someone (preferably Ruy, but I’ll keep checking comments) explain to me why you would “weight” for anything at all in polls? The LA times article linked in the last post says that you shouldn’t weight for party affiliation (i.e. manipulate your sample so you have the “right” number of republicans, say), but it also says pollsters commonly weight for various more stable demographic types (income, for instance). Shouldn’t any kind of weighting at all screw up the simple random sample principle? Or is it just generally supposed that people failing to respond has already messed this up? If so, what does this do to my (previously fairly Chernoff-bound-oriented) understanding of what error means in political polls?
frankly0 – Well said! I think that while guys like Josh serves a great purpose, Ruy is also performing a great service. Bill, if you’re not a stat head, you can always skip past Ruy’s posts. And if you’ve spent any time at all on the EDM site or flipped through the book, you’ll know that Ruy _isn’t_ a journalist like Ackerman and Josh.
bt – i’ve been reading Josh since day 1 and have conversed with him a few times on th subject, when he first started up, the blogging software isn’t as complex as it is today, obviously what someone mentioned earlier about Josh wanting to appear more “professional” has something to do with it but as I said, the main reason why he doesn’t have comments is that his version of the software doesn’t support it.
On all the polling stuff, I wonder why Ruy hasn’t mentioned anything about the poll that Zogby is doing for WSJ, his latest poll showed that Bush has picked up a couple swing states like NV which had previously been for Kerry. I know that Zogby polls favour Republicans slightly too, but I would be interested to see what Ruy has to say about it.
Bill,
If the details of polls aren’t your cup of tea, maybe you should drink elsewhere?
I hate to see someone so obviously unhappy, when remedy is so easy to come by.
Regarding the latest Bush ad: isn’t it hypocritical past the point of the bizarre for the Bush campaign to accuse the Dems of being wild eyed, and out of control in their rhetoric, in the very same ad in which THEY juxtapose images of Adolph Hitler with those of their political opponents?
Ruy, can you do a favor to regular readers of TPM and never come back? Please?
Regurgitating poll data is *not* reporting, and hardly worth commenting upon, unless you are fishing for a job with corporate media–then go for it!. Look at those FOX polls you comment on–do they really deserve so much attention?
I hope you are not as boring a person as you are a writer, with all due respect.
To follow up, I like the way Ruy’s software permits his readers to comment, but without cluttering up the site. It’s easy to bypass the comments for those who want to.
I’d be very surprised if Josh hasn’t wrestled with this issue and he may well have written about his reasons for not doing so previously. (I’ve only been a regular visitor there for roughly this year.) If you’ve got a product that’s in great and continually rising demand, why mess with it? My thought was that given the huge amount of traffic that is there anyway, why let a good chance to facilitate a lot of networking escape?
This is my first visit to your site. I stop by TPM a few times a week and I’ve enjoyed your guest spot, sitting in for Josh.
I’ll be a regular here now.
Some weeks ago there was speculation about NJ and whether it was a battleground state. I assured you that NJ was solid for Kerry and would be in November. If any of you doubted me, check that campaigndesk.org web site today and see what it says about the polls and NJ. By their logic, NJ is more soldily Kerry this year than it was solidly Gore in 2000. Be of good cheer. I, personally, will continue to work for a Kerry victory this year.
Well Josh, like Queen Sully is a “professional” so its probably just a decision to make his site look neater.
I always figured the mainstream media took you more seriously if you didn’t allow comments đ
I rather like the fact that Josh doesn’t include posting to his site. Helps the site have a good signal-to-noise ratio. Not that I am opposed to comments at blog sites, but given that wealth of opportunity to say my piece throughout the internet, Josh’s lack of commenting seems fairly minor. Of course, if he adds commenting, I wouldn’t object.
Besides, he is very good about responding to email. Over the last couple of years, I have emailed him about half a dozen times, and he has responded every time. Sometimes briefly, sometimes in depth. For me, that’s great. I don’t miss the commenting.
Ron, I agree. I’m regularly amazed at what Josh is able to get into the media and informed public mix traveling outside the DC area barely at all as he does. He obviously works his ass off and has developed great sources.
But, yes, I’ve also wondered why he doesn’t “allow” his fans to respond to his posts and at least have the opportunity to meet and network among themselves through the use of software with the capabilities of Ruy’s site.
Josh’s site, as data he has shared at times indicates, gets an enormous amount of traffic. (rivaled perhaps only by the amount and/or “quality” of traffic here at Ruy’s site)
I hope that when he gets back from his well-earned vacation that Josh will consider the built-in networking opportunities that a change in his software could provide for his many fans, opportunities which could further advance public information and informed debate about the issues he writes about.
Ruy,
Do you think the administration’s decision to give AIDS relief to Vietnam (instead of India, China, etc.) has to do with courting the Vietnamese American vote? I didn’t hear any discussion of this in the media.
Consider: Vietnamese Americans tend to be conservative Catholics and past victims of Communist repression, hence wooable by the GOP. While it is true that they are most populous in California and Texas, they’re also a non-negligible presence in quasi-swing states like Virginia, Washington, and Louisiana.
It’s a little far out, but what do you think? (I would have written an email, but I couldn’t track down your address.) Thanks.
This year’s big media narrative has been the confirmation saga of Neera Tanden, Biden’s nominee for director of the Office of Management and Budget. At New York I wrote about how over-heated the talk surrounding Tanden has become.
Okay, folks, this is getting ridiculous. When a vote in the Senate Homeland Security and Governmental Affairs Committee on the nomination of Neera Tanden was postponed earlier this week, you would have thought it presented an existential threat to the Biden presidency. âScrutiny over Tandenâs selection has continued to build as the story over her uneven reception on Capitol Hill stretched through the week,â said one Washington Post story. Politico Playbook suggested that if Tanden didnât recover, the brouhaha âhas the potential to be what Biden might call a BFD.â Thereâs been all sorts of unintentionally funny speculation about whether the White House is playing some sort of âthree-dimensional chessâ in its handling of the confirmation, disguising a nefarious plan B or C.
Perhaps it reflects the law of supply and demand, which requires the inflation of any bit of trouble for Biden into a crisis. After all, his Cabinet nominees have been approved by the Senate with a minimum of 56 votes; the second-lowest level of support was 64 votes. One nominee who was the subject of all sorts of initial shrieking, Tom Vilsack, was confirmed with 92 Senate votes. Meanwhile, Congress is on track to approve the largest package of legislation moved by any president since at least the Reagan budget of 1981, with a lot of the work on it being conducted quietly in both chambers. Maybe if the bill hits some sort of roadblock, or if Republican fury at HHS nominee Xavier Becerra (whose confirmation has predictably become the big fundraising and mobilization vehicle for the GOPâs very loud anti-abortion constituency) reaches a certain decibel level, Tanden can get out of the spotlight for a bit.
But whatâs really unfair â and beyond that, surreal â is the extent to which this confirmation is being treated as more important than all the others combined, or indeed, as a make-or-break moment for a presidency that has barely begun. Itâs not. If Tanden cannot get confirmed, the Biden administration wonât miss a beat, and I am reasonably sure she will still have a distinguished future in public affairs (though perhaps one without much of a social-media presence). And if she is confirmed, weâll all forget about the brouhaha and begin focusing on how she does the job, which she is, by all accounts, qualified to perform.
Nice resource. Thanks đ
Yes, Bill, the 2000 Florida exit poll informed me of a great many things. The fact that news enterprises misused it to jump to erroneous conclusions about who had won the state in the hours after the polls closed does not detract from the wealth of knowledge contained in the exit poll results.
Ruy, you really need to respond to Bill’s post, I think it’s a great opening to see a fascinating discuss.
Dennis, I think frankly’s got it pretty much right and certainly don’t have much more to add on.
Dennis, as to your question, why pollsters weight for anything, I think the answer lies, at least in part, in the fact that, of course, the techniques they employ do NOT truly create a random, REPRESENTATIVE sample. Asking questions in a phone poll only of those who choose to respond introduces all kinds of biases. I would expect that, simply to adjust for these quite predicable biases, one must start fudging weights in any case (for example, do more women or men answer the phone?)
Even if one had a truly random, truly representative sample, I would expect that weighting the results for known demographic facts — e.g., gender representation — WOULD make for more accurate polls in general. If, for example, in a truly random, truly representative poll 55% happened to be women, and 45% men, wouldn’t it make the poll LESS error prone if those numbers were properly weighted? What the underlying mathematics of this should be left to statisticians, but intuitively it’s pretty obvious that such an adjustment WILL diminish the likelihood of error.
Greg, Frankly: I read the article, but the article didn’t answer my question, which is why weight for anything? I’m basically a math/cs guy, so I take my randomness pretty seriously, and since in a random sample the demographics shouldn’t be too different from what you expect anyway (not to mention that doing things not-entirely-randomly messes up the theorems that make it valid), what’s the reason anyone would weight for anything?
Frankly: I suspect that the idea you had about how weighting must work is true; tossing people out seems to be the best way to preserve the sanctity of the sample, but I’m wondering if it’s more complicated than that.
Sad, Ruy likes soccer đ
Can’t you love Baseball instead? Baseball is the sports of the gods afterall.
Ruy and Josh do different things, providing very different types of information from very different sources, both first class and superb. Some people need to put their sharp tongues back in the sheath and leave them there. How about of bit of civility and respect. Save your knives for Karl and George, deserving beyond measure.
While it’s good that Ruy is sticking to what he knows, it’s awfully boring for a lot of people. So what is Ruy’s opinion on non-poll news? That’s more what TPM is for, and I for one would like to see that.
Dennis,
I do think it makes perfect intuitive sense to weight samples based on hard demographic data in cases in which the relevant classifications of the sampled population is indisputably accurate.
Gender would seem to be such a variable. It’s exceedingly unlikely a poll taker could get such a thing wrong, and it most definitely affects probability of voting preferences. Likewise, geographical classifications would seem to be highly reliable and relevant. Income and age seem to me on the other hand pretty unreliable, given their dependence on self report. Party ID has not only the self report problem, but the more significant problem that it can vary from day to day.
I’m not sure how exactly the “weighting” affects the way in which error and other quantities get calculated, but my guess is that the MOE is very little affected, since it is so highly dependent on the sample size in any calculation. Certainly the MOE’s I’ve seen seem to be directly calculated from the reported sample size, as if no weighting had taken place. Of course, the whole point of the exercise is to reduce the likelihood of error, so I’m not sure quite how to understand all of this.
One question I’ve always had is, how, if these numbers get weighted, do the pollsters come up with the nice integer numbers they do for their polls? Do they gather much larger samples than the numbers reported, and then just throw out, say, a certain number of women if there are too many women? Do they do the same for all categories? Are the integer numbers reported just a convenient fiction, because the floating point numbers genuine weights would impose would be embarrassing to explain to the general public?
Dennis – as the LATimes article stated, political operatives typically like to see polls weighed because they believe that party ID is more fixed and static and therefore, polls need to be adjusted to reflect that view. Independent posters on the other hand, believe that party ID is more fluid and therefore, you might very well find a +13 on Democrat party ID because lately the news have been heavily anti-administration.
(No place to comment at TPM, so I’ll do it here)
Re Fox polling: So, either Fox News’ poll has it right, and Gallup AND American Research Group AND Quinnipiac all have it wrong. Or….
Can someone (preferably Ruy, but I’ll keep checking comments) explain to me why you would “weight” for anything at all in polls? The LA times article linked in the last post says that you shouldn’t weight for party affiliation (i.e. manipulate your sample so you have the “right” number of republicans, say), but it also says pollsters commonly weight for various more stable demographic types (income, for instance). Shouldn’t any kind of weighting at all screw up the simple random sample principle? Or is it just generally supposed that people failing to respond has already messed this up? If so, what does this do to my (previously fairly Chernoff-bound-oriented) understanding of what error means in political polls?
frankly0 – Well said! I think that while guys like Josh serves a great purpose, Ruy is also performing a great service. Bill, if you’re not a stat head, you can always skip past Ruy’s posts. And if you’ve spent any time at all on the EDM site or flipped through the book, you’ll know that Ruy _isn’t_ a journalist like Ackerman and Josh.
bt – i’ve been reading Josh since day 1 and have conversed with him a few times on th subject, when he first started up, the blogging software isn’t as complex as it is today, obviously what someone mentioned earlier about Josh wanting to appear more “professional” has something to do with it but as I said, the main reason why he doesn’t have comments is that his version of the software doesn’t support it.
On all the polling stuff, I wonder why Ruy hasn’t mentioned anything about the poll that Zogby is doing for WSJ, his latest poll showed that Bush has picked up a couple swing states like NV which had previously been for Kerry. I know that Zogby polls favour Republicans slightly too, but I would be interested to see what Ruy has to say about it.
Bill,
If the details of polls aren’t your cup of tea, maybe you should drink elsewhere?
I hate to see someone so obviously unhappy, when remedy is so easy to come by.
Regarding the latest Bush ad: isn’t it hypocritical past the point of the bizarre for the Bush campaign to accuse the Dems of being wild eyed, and out of control in their rhetoric, in the very same ad in which THEY juxtapose images of Adolph Hitler with those of their political opponents?
Ruy, can you do a favor to regular readers of TPM and never come back? Please?
Regurgitating poll data is *not* reporting, and hardly worth commenting upon, unless you are fishing for a job with corporate media–then go for it!. Look at those FOX polls you comment on–do they really deserve so much attention?
I hope you are not as boring a person as you are a writer, with all due respect.
To follow up, I like the way Ruy’s software permits his readers to comment, but without cluttering up the site. It’s easy to bypass the comments for those who want to.
I’d be very surprised if Josh hasn’t wrestled with this issue and he may well have written about his reasons for not doing so previously. (I’ve only been a regular visitor there for roughly this year.) If you’ve got a product that’s in great and continually rising demand, why mess with it? My thought was that given the huge amount of traffic that is there anyway, why let a good chance to facilitate a lot of networking escape?
Oh, and sorry my remarks are off-topic.
This is my first visit to your site. I stop by TPM a few times a week and I’ve enjoyed your guest spot, sitting in for Josh.
I’ll be a regular here now.
Some weeks ago there was speculation about NJ and whether it was a battleground state. I assured you that NJ was solid for Kerry and would be in November. If any of you doubted me, check that campaigndesk.org web site today and see what it says about the polls and NJ. By their logic, NJ is more soldily Kerry this year than it was solidly Gore in 2000. Be of good cheer. I, personally, will continue to work for a Kerry victory this year.
Well Josh, like Queen Sully is a “professional” so its probably just a decision to make his site look neater.
I always figured the mainstream media took you more seriously if you didn’t allow comments đ
I rather like the fact that Josh doesn’t include posting to his site. Helps the site have a good signal-to-noise ratio. Not that I am opposed to comments at blog sites, but given that wealth of opportunity to say my piece throughout the internet, Josh’s lack of commenting seems fairly minor. Of course, if he adds commenting, I wouldn’t object.
Besides, he is very good about responding to email. Over the last couple of years, I have emailed him about half a dozen times, and he has responded every time. Sometimes briefly, sometimes in depth. For me, that’s great. I don’t miss the commenting.
Ron, I agree. I’m regularly amazed at what Josh is able to get into the media and informed public mix traveling outside the DC area barely at all as he does. He obviously works his ass off and has developed great sources.
But, yes, I’ve also wondered why he doesn’t “allow” his fans to respond to his posts and at least have the opportunity to meet and network among themselves through the use of software with the capabilities of Ruy’s site.
Josh’s site, as data he has shared at times indicates, gets an enormous amount of traffic. (rivaled perhaps only by the amount and/or “quality” of traffic here at Ruy’s site)
I hope that when he gets back from his well-earned vacation that Josh will consider the built-in networking opportunities that a change in his software could provide for his many fans, opportunities which could further advance public information and informed debate about the issues he writes about.
Good to see Ruy addressing some positive news for Bush polls.
The trouble with Josh Marshall’s site is that it’s a one-way conversation. I hope people will post comments here.
Ruy,
Do you think the administration’s decision to give AIDS relief to Vietnam (instead of India, China, etc.) has to do with courting the Vietnamese American vote? I didn’t hear any discussion of this in the media.
Consider: Vietnamese Americans tend to be conservative Catholics and past victims of Communist repression, hence wooable by the GOP. While it is true that they are most populous in California and Texas, they’re also a non-negligible presence in quasi-swing states like Virginia, Washington, and Louisiana.
It’s a little far out, but what do you think? (I would have written an email, but I couldn’t track down your address.) Thanks.