We live in a world where anything potentially dangerous usually carries a warning label. So why are public opinion polls published by news organizations without any words of caution?
Readers should be warned, at a bare minimum, that “these poll results should be interpreted with caution. Users should understand how margin of error and random sampling operate prior to drawing any conclusions.”
Since pollsters can’t poll everyone they have to take a sample of the population. That means asking a number of people which candidate they plan to vote for or where they stand on a particular issue and then extrapolating those numbers to the entire population.
Polling actually works surprisingly well. It has been shown that you don’t need that big of a sample size before you start getting results that are pretty close to the actual numbers. Even with just a thousand respondents in a country the size of Canada - a country of 33 million people - you can predict where the public stands to within about three percentage points (19 times out of 20 - I’ll come back to that). That’s actually pretty impressive if you think about it.
Perhaps counterintuitively, polling smaller populations is more difficult than polling large ones. You need to poll a much greater percentage of the population to achieve the same margin of error for a small population than a large one.
For example, that Ekos poll released last week purporting to show a Liberal lead here in the Yukon had only 497 respondents - or about 1.5 per cent of the total population - yet had a pretty wide margin of error of plus or minus 4.4 per cent. The same firm’s national poll of about 2,343 respondents - or about 0.0071 per cent of the Canadian population - carries a margin of error of only two per cent.
And margin of error - particularly of the size we saw in that Ekos poll of the Yukon - can make a huge difference. You have to keep in mind that that margin applies to each candidate. So not only might candidate A be several points lower than advertised, but candidate B might also be several points higher - making it either a much tighter race or a more comfortable lead than it might appear.
With that in mind, Yukoners should be cautious in interpreting the results. It is not that the poll results are meaningless - just that any candidate could be as much as 4.4 per cent higher or lower, which is a lot when the three main parties are separated by only 12 percentage points.
And that assumes that the poll wasn’t an “outlier.” Polling is based on the assumption that the sample was representative of the overall population. But that’s not always the case.
Countless factors can skew your poll, and most mainstream pollsters are pretty good at controlling for those factors. For example, using the phone book to select poll respondents is no longer a useful method because there are so many cellphone-only households that wouldn’t be selected. (Contrary to popular opinion, pollsters do now call cellphones.)
But even a perfectly designed poll can be an outlier. It turns out that pollsters reserve for themselves the right to the occasional mulligan, which is what the whole “19 times out of 20” business is all about. Even the best pollsters will be off by more than advertised about five per cent of the time.
Last week the Toronto Star blared “Conservatives swing into lead, close in on majority, new poll suggests.” The poll the headline was referring to, also conducted by Ekos pegged, support for the Conservatives at 35.4 per cent with the Liberals and NDP far behind at 26.3 per cent and 24.5 per cent respectively.
It was good of the Toronto Star to attach the qualifier “new poll suggests.” But they could have done without the alarmist “closing in on a majority,” and they could have drawn attention to the fact that the result was way off what all the other polls were saying during the same period. That Ekos poll was very likely that one out of 20 - the infamous “outlier.”
This isn’t to be critical of Ekos or its methodology. Sometimes the pollster reaches into that sampling bucket of Canadians and pulls out a handful of respondents whose aren’t your “average Canadians.” It is just the way sampling goes, and it can happen to any polling company.
The same is likely true of the Forum poll conducted September 9-10 showing the NDP at 36 per cent - results significantly at odd with every other poll from that time period and which have not been replicated since.
My point isn’t to ignore polls, only to know how to interpret them. This goes for the media as well.
In a close race the margin of error is the story, not an afterthought. If a poll has a margin of error of 2.5 per cent and one party is a percentage point or two above the other, the first party does not have a “lead” over the other, nor is the second party “trailing.” Also, if a party’s numbers changes from one poll to the next but the change is within the margin of error, their support is not “growing” of “falling.” The data is simply insufficient to draw those conclusions, and we should leave it at that.
And if a poll is too good or depressing to be true, it probably is. So take a deep breath and see where the trend is going. Or, better yet, just wait for election day.
Kyle Carruthers is a born-and-raised Yukoner who lives and practises law in Whitehorse.