Last week’s Big Fail by the press and punditocracy in predicting Donald Trump’s election has been attributed by some to rampant bias. That’s an understandable charge: Trump supporters in media or among the political class were few and far between.
But the real problem was the polls, which misled analysts of all stripes. Their failure raises the question: is public opinion polling, one of the great innovations of the 20th century, dying in the 21st? That has implications for business as well as politics.
My former colleagues at the Pew Research Center have done a good job summarizing the three leading theories behind the poll miss in a blog post here.
The first is the most fundamental – known as “non-response bias.” The science of polling is based on an understanding that you can accurately infer the attitudes of a large population by polling a relatively small random sample. Random selection of phone numbers thus became the heart of modern polling, and the best pollsters – like Pew – extended that to cell phones. But these days, most people won’t take the time to answer a pollster’s questions. Response rates have fallen below 10%. And while statistical methods are used to ensure the polling sample accurately represents the broader population, there’s still the nagging question of whether some inherent bias separates those who will talk to pollsters from those who won’t. It’s a big issue for polling generally. But I’m not convinced it has much to do with the Trump fail.
Second is the “shy Trumpers” theory – the notion that some Trump supporters weren’t willing to admit their support to telephone pollsters. But comparing telephone polls with online polls suggests that difference was pretty small. So again, not persuasive.
That leads to the third, and most important, reason. Pollsters may have done a pretty good job measuring the attitudes of the American public. Where they failed was in accurately predicting who would show up at the polls. Their “likely voter” models were based largely on history; and there was nothing in recent history like Election 2016.
Bottom line: today’s polls, while challenged by the difficulties of obtaining truly random samples, are nevertheless still pretty good at divining public attitudes. But what they can’t do is predict public behavior. “The weakest point of all election polls is likely voter modeling,” says my friend Jon Cohen, chief research officer at SurveyMonkey. “That’s now completely broken.”