Nate Silver’s FiveThirtyEight website, now owned and hosted by ESPN, features an interesting mix of articles that generally fall into two disparate topics: sports and political polling. An article yesterday, posted by Silver himself, had a stimulating discussion about the disparities in the polls for the Alabama senate election that delved into topics that are seemingly of decreasing consequence anymore among researchers: the impact of modes of interview, response rates, representativeness of samples, and weighting.
Let’s just focus on the last topic, weighting. Even among experienced and savvy researchers, weighting can still be a bit of an arcane dark art. Among the inexperienced – or those who never learned any better – weighting is just a magic black box that supposedly fixes whatever might be wrong with a crappy (to use a highly technical term) sample. Silver’s article included a great, real-life example of the effects of weighting.
To their credit, especially given my usual attitude towards self-service survey companies, SurveyMonkey published 10 different outcomes of their data for the Alabama senate race – the underlying unweighted data was the same, it was only the weights that differed. The various outcomes showed a range from a 9-point Jones lead to a 10-point Moore lead. As Silver points out, these results emphasize “the extent to which polling can be an assumption-driven exercise”.
These results also show the impact of not just demographic weighting but attitudinal or behavioral weighting – in this case, forcing a sample to match things like political affiliation or likelihood to vote from a previous election, which may or may not reflect contemporary conditions.
Getting to the (data) point
The point of this post is not to denigrate weighting but merely to point out this nice example of the potential impact of weighting and its potential influence on data outcomes and insights. Get your weighting scheme right, and it no doubt helps. Get it wrong, and it can lead to incorrect conclusions that have significant implications for an candidate or for the success of a business (depending on the sphere of the survey).
Better yet, if you start out with high quality sample – rather than the least expensive – and demand high cooperation rates, weighting should be of secondary consequence.
David Tice is the principal of TiceVision LLC, a media research consultancy.
Get notifications of new posts – sign up at right or at bottom of this page.
