A Pollster Speaks
A recent book by Anthony Salvanto, “Where Did You Get This Number?” has a lot of interesting insights into surveying. Salvanto is a pollster. He explains sampling:
The first step is to forget for a moment anything about the specific size of the poll, be it one thousand people or ten thousand people, and right now simply think in terms of knowledge about the world—knowledge that you can either get or not get.
There are plenty of times people can gauge how well they know something by what portion of all the available information they have. In school, for instance, when tomorrow’s history test covers the whole textbook, but you only read half of it, you can correctly gauge that you’re in trouble. (I found this out the hard way a few times.) Or if you’re buying a new car, and you haven’t read the crash test ratings or found out the gas mileage yet, you could justifiably feel uninformed walking into the dealership. Those are problems of completeness: you haven’t seen all the information that’s out there, and what you do know just will not substitute for what you don’t.
A poll, as traditionally conceived, does not try to fit into those categories of information gathering. There are other occasions, more akin to polling, when we gauge whether we truly know about something by whether or not we’ve sampled it well; that is, when we think what we’ve already seen is a good enough representation of all that we have not seen. It’s the restaurant you visit twice, not a hundred times, before you decide if it’s good.
A classic analogy for the mechanism behind this was mentioned by Gallup in a chapter he wrote in his book The Pulse of Democracy called “Building the Miniature Electorate,” in which he compared sampling the country to tasting a “bowl of soup.”
He adds on sizing:
On a sample of 1,000, a poll will often report a margin of error of 3 points. If a poll reports an estimate of 50 percent with a margin of error of 3, we’re saying we’d get values between 53 and 47 if we kept repeating the poll, and that the truth is in that range. That’s often good enough for us to tell a meaningful story, such as how many movie fans there are. And we sometimes have to, because the margin doesn’t get a lot better as we collect more samples from there. On a sample of 3,000 it’s . . . about 2 points. We just tripled our sample size from 1,000 to 3,000 and barely dropped the margin of error. That’s because there’s always going to be at least some uncertainty arising from the fact that we haven’t talked to everybody. Even if we drew huge samples of one million people, sometime along the way of drawing them pick by pick we’d get some samples that were 59 percent to 41 percent, or even 60-40, instead of being evenly balanced. Not many, but some. That’s randomness at work, too. Samples, it turns out, work mathematically a lot like experience in life. Getting some is necessary, and getting a lot makes you good. But no matter how good you get, no one is perfect.
In India, the focus needs to be on India’s 4000 Assembly Constituencies to ensure the spread that is needed. Prashnam does just that.
Tomorrow: Part 6
