The Research Grouch got a double whammy Monday morning. First, having seen the headline “We Are Finally Getting Better at Polls” on Bloomberg.com, I clicked right to the piece. Once there, however, I discovered it was a self-promoting opinion piece by a UK polling company.
Already grouchy from feeling somewhat misled, the content of the piece ratcheted up the grouch level. The author discussed what he considered to be new, innovative ways to reach “representative” samples for polling. These new ways included using IM instead of email to reach out to respondents (OK, that makes sense); showing respondents how their responses sit against others taking the poll (debatable, especially if it’s raw instantaneous data); giving people surveys on topics they like in order to maintain interest (again, debatable); emotional testing rather than direct questions (not sure how this solves a sample issue); recruiting from non-political or non-news websites, or from social media (diversification of online recruiting does not create representativeness).
About the only indisputable point made in the piece is politically active people need to be recruited in their correct proportions to get the best data. If this is considered news to political pollsters in the UK, then no wonder they had issues predicting recent elections.
I may be a grouch but I’m not against innovation. These suggestions are certainly ways to potentially increase respondent engagement and diversify online sample sources. But, despite claims from online research firms everywhere, a volunteer opt-in online sample is by definition not representative regardless of panel size, recruitment techniques, weighting, or other manipulations.
Such samples may often give the same answers as a truly representative sample, but that doesn’t make them representative. Neither does the use of the terminology, such as response rates and error margins, lifted from traditional probability-based research and which really don’t apply to volunteer samples of any kind. A serious problem for this industry is that there is a whole generation of researchers that don’t realize this.
There really are probability-based online panels
I do recognize it’s a different world today than 20 years ago. Opt-in online samples are generally “good enough” for many applications, and I’ve used them myself many times. But, at least in the USA, there are sources of online access panels recruited using traditional probability techniques (including GfK’s KnowledgePanel, NORC’s AmeriSpeak, and RAND’s American Life panel) which are available for important research, whether political polls or key business decisions.
Yes, these panels are expensive compared to opt-in sample, but you get what you pay for – and as many pollsters, candidates, and businesses have found over the years, the most expensive research is bad research.
David Tice is the principal of TiceVision LLC, a media research consultancy.
Get notifications of new posts – sign up at right or at bottom of this page.
