The polling industry has produced some epic fails over the years.
Most recently, pollsters predicted that the U.K. election would be neck-and-neck between the Tories and Labour. The Conservatives won a historic majority.
I saw a similar incident back in 1992 during that British election. I was doing some beer-assisted psephological research in the pub at the famously left-wing London School of Economics on election night. The polls had been predicting a hung parliament, with the Tories taking a beating after a decade of Thatcherism. When the BBC announced a Tory majority late in the evening, the pub went silent and all you could hear was people dropping their pint glasses in stunned amazement.
Eric Grenier, who runs a prominent Canadian poll aggregator website, points out that not a single poll in the 2013 B.C. election showed the Liberals in the lead, even though they ended up with a majority government. The 2012 Alberta election was another polling fiasco.
And here in the Yukon, who can forget the 2011 poll by local firm Datapath that showed Larry Bagnell leading Ryan Leef by 20 points just a week before election day? Leef won.
So, are polls just a socially useless waste of space on the Internet, like Justin Bieber fansites and Youtube’s cat video channel?
Like so much else these days, it depends. There are quality polls and shaky polls. There are even polls that aren’t designed to find out anything, but rather exist to plant messages in people’s heads. You know, the kind of poll where they phone everyone in town and ask questions like, “Would you be more or less likely to support Democratic candidate Joe Blogs if you knew he used to smoke crack with Osama Bin Laden?”
It depends what question you are trying to answer, and on the reliability of the poll. If you are interested in national results, perhaps because you are a campaign manager deciding where to spend the advertising budget, or a big oil company trying to decide to make a billion-dollar investment in Alberta, then you should look at the poll aggregators.
These outfits combine multiple polls using fancy statistical techniques to essentially get a bigger sample size and a more reliable estimate. They also layer in non-polling data such as the past reliability of the polling company, the age of the poll, candidates’ voting records, incumbency status, whether the voter is registered or likely to vote, and so on.
The most famous poll aggregator is Nate Silver in the U.S., who became a minor rock star during the 2012 presidential campaign. He appeared on Jon Stewart’s Daily Show, and his book The Signal and the Noise was named No. 1 non-fiction book for 2012 by Amazon and sold in huge numbers (although we might want to do a poll to see how many people actually read it). The accuracy of his presidential predictions is almost spooky. He called all 50 states correctly in 2012.
In Canada, Eric Grenier’s Threehundredeight.com tries to do the same thing. Thanks in part to Nate Silver’s success, Threehundredeight has attracted a lot of attention in Canada during this election. If you read his methodology, it is clear a huge amount of thought and expertise went into crafting the model and the assumptions behind it.
Its current output suggests the three big parties are roughly tied in national support, with this translating regionally into the Conservatives winning the most seats.
However, there is a serious issue with Threehundredeight, and it has nothing to do with Eric Grenier or his methodology. It is instead the relatively small number of polls his model has to aggregate, and their record of accurate predictions. Grenier’s model bombed the 2013 B.C. election. But that’s no surprise, since as mentioned above the polls he was plugging into it were all horribly wrong. Furthermore, we have fewer polls covering fewer elections in Canada for him to fine-tune his model. Nate Silver, on the other hand, has literally thousands of polls covering hundreds of presidential, Senate, governor and other elections to work with.
I watch Threehundredeight’s output closely. It is the best we have in Canada. But I wouldn’t bet my claim on it. In particular, as some pundits have speculated, it is quite possible that Conservative support is systematically under-reported (as occurred in some of the polling disasters mentioned above). This could be because of differences between who is polled and who votes. Or because people are reluctant, for one reason or another, to tell pollsters they plan to vote Conservative. My bet is that Conservative results on election day will be a few points better than the pre-election polls.
Fortunately, I’m not a national campaign manager or oil company executive with a decision to make. I just have to decide how to vote in the Yukon.
This is where an actually useful poll makes its appearance. Leadnow.ca, a campaign that encourages strategic voting to oust Conservative MPs, hired Environics Research to do polls in 31 ridings where the Conservative MP is considered vulnerable.
Their Yukon sample size is a respectable 497 and their poll was taken September 19-21. It was a robo-poll, using what the industry calls Interactive Voice Response technology. It targeted directory-listed phone numbers with postal codes attached to find Yukon voters, but did not include cellphone numbers. The latter is something that Nate Silver thinks is very important to do given the growing number of people without landlines. The poll also, as far as I can tell, didn’t separate likely voters from the general population.
The poll results were that Larry Bagnell of the Liberals led with 39 per cent of decided voters, while the NDP’s Melissa Atkinson had 29 per cent. Conservative Ryan Leef had 27 per cent and the Green Party’s Frank de Jong had four per cent.
The poll had a theoretical margin of error of 4.4 points, within a 95 per cent confidence interval (which is what is often called “19 times out of 20” in the papers). Looking at the math, Atkinson’s lead on Leef is two points, which is within the 4.4 point margin of error. This means it is likely she is ahead of Leef, but the word “likely” means a less than 95 per cent chance. Meanwhile, Bagnell’s lead of 10 points over Atkinson is more than double the margin of error. This means that the chance of Atkinson being ahead of Bagnell is unlikely, with “unlikely” meaning a less than five per cent chance.
These are the polls. Take them for what they are worth with all their caveats and questions.
Nonetheless, the LeadNow poll is the most reliable and most recent poll we have. For those interested in strategic voting to get rid of Ryan Leef, the data suggests Bagnell is the best vote.
Disclosure: I am a friend and supporter of Larry Bagnell.
Keith Halliday is a Yukon economist and author of the MacBride Museum’s Aurore of the Yukon series of historical children’s adventure novels. He won this year’s Ma Murray award for best columnist. You can follow him on Channel 9’s “Yukonomist” show.