Misread political polls at your peril

We live in a world where anything potentially dangerous usually carries a warning label.

We live in a world where anything potentially dangerous usually carries a warning label. So why are public opinion polls published by news organizations without any words of caution?

Readers should be warned, at a bare minimum, that “these poll results should be interpreted with caution. Users should understand how margin of error and random sampling operate prior to drawing any conclusions.”

Since pollsters can’t poll everyone they have to take a sample of the population. That means asking a number of people which candidate they plan to vote for or where they stand on a particular issue and then extrapolating those numbers to the entire population.

Polling actually works surprisingly well. It has been shown that you don’t need that big of a sample size before you start getting results that are pretty close to the actual numbers. Even with just a thousand respondents in a country the size of Canada – a country of 33 million people – you can predict where the public stands to within about three percentage points (19 times out of 20 – I’ll come back to that). That’s actually pretty impressive if you think about it.

Perhaps counterintuitively, polling smaller populations is more difficult than polling large ones. You need to poll a much greater percentage of the population to achieve the same margin of error for a small population than a large one.

For example, that Ekos poll released last week purporting to show a Liberal lead here in the Yukon had only 497 respondents – or about 1.5 per cent of the total population – yet had a pretty wide margin of error of plus or minus 4.4 per cent. The same firm’s national poll of about 2,343 respondents – or about 0.0071 per cent of the Canadian population – carries a margin of error of only two per cent.

And margin of error – particularly of the size we saw in that Ekos poll of the Yukon – can make a huge difference. You have to keep in mind that that margin applies to each candidate. So not only might candidate A be several points lower than advertised, but candidate B might also be several points higher – making it either a much tighter race or a more comfortable lead than it might appear.

With that in mind, Yukoners should be cautious in interpreting the results. It is not that the poll results are meaningless – just that any candidate could be as much as 4.4 per cent higher or lower, which is a lot when the three main parties are separated by only 12 percentage points.

And that assumes that the poll wasn’t an “outlier.” Polling is based on the assumption that the sample was representative of the overall population. But that’s not always the case.

Countless factors can skew your poll, and most mainstream pollsters are pretty good at controlling for those factors. For example, using the phone book to select poll respondents is no longer a useful method because there are so many cellphone-only households that wouldn’t be selected. (Contrary to popular opinion, pollsters do now call cellphones.)

But even a perfectly designed poll can be an outlier. It turns out that pollsters reserve for themselves the right to the occasional mulligan, which is what the whole “19 times out of 20” business is all about. Even the best pollsters will be off by more than advertised about five per cent of the time.

Last week the Toronto Star blared “Conservatives swing into lead, close in on majority, new poll suggests.” The poll the headline was referring to, also conducted by Ekos pegged, support for the Conservatives at 35.4 per cent with the Liberals and NDP far behind at 26.3 per cent and 24.5 per cent respectively.

It was good of the Toronto Star to attach the qualifier “new poll suggests.” But they could have done without the alarmist “closing in on a majority,” and they could have drawn attention to the fact that the result was way off what all the other polls were saying during the same period. That Ekos poll was very likely that one out of 20 – the infamous “outlier.”

This isn’t to be critical of Ekos or its methodology. Sometimes the pollster reaches into that sampling bucket of Canadians and pulls out a handful of respondents whose aren’t your “average Canadians.” It is just the way sampling goes, and it can happen to any polling company.

The same is likely true of the Forum poll conducted September 9-10 showing the NDP at 36 per cent – results significantly at odd with every other poll from that time period and which have not been replicated since.

My point isn’t to ignore polls, only to know how to interpret them. This goes for the media as well.

In a close race the margin of error is the story, not an afterthought. If a poll has a margin of error of 2.5 per cent and one party is a percentage point or two above the other, the first party does not have a “lead” over the other, nor is the second party “trailing.” Also, if a party’s numbers changes from one poll to the next but the change is within the margin of error, their support is not “growing” of “falling.” The data is simply insufficient to draw those conclusions, and we should leave it at that.

And if a poll is too good or depressing to be true, it probably is. So take a deep breath and see where the trend is going. Or, better yet, just wait for election day.

Kyle Carruthers is a born-and-raised Yukoner who lives and practises law in Whitehorse.

Get local stories you won't find anywhere else right to your inbox.
Sign up here

Just Posted

During our recent conversation, John Nicholson showed me snapshots of his time working on the Yukon riverboats 70 years ago. (Michael Gates)
History Hunter: Yukon man relives the riverboat days after seven decades

John Nicholson took summer work on Yukon steamers in the 1950s

NDP candidate Annie Blake, left, and Liberal incumbent Pauline Frost. (Submitted photos)
Official recount confirms tie vote in Vuntut Gwitchin riding

Both candidates Pauline Frost and Annie Blake are still standing with 78 votes each

Artist’s rendering of a Dairy Queen drive-thru. At its April 13 meeting, Whitehorse city council approved a zoning change to allow a drive-thru at 107 Range Road. Developers sought the change to build a Dairy Queen there. (Submitted)
Drive-thru approved by Whitehorse city council at 107 Range Road

Rezoning could pave the way for a Dairy Queen

xx
WYATT’S WORLD

Wyatt’s World for April 14, 2021.… Continue reading

Joel Krahn/joelkran.com Hikers traverse the Chilkoot Trail in September 2015. Alaska side.
The Canadian side of the Chilkoot Trail will open for summer

The Canadian side of the Chilkoot Trail will open for summer Parks… Continue reading

École Whitehorse Elementary Grade 7 students Yumi Traynor and Oscar Wolosewich participated in the Civix Student Vote in Whitehorse on April 12. (Haley Ritchie/Yukon News)
Yukon Student Vote chooses Yukon Party government; NDP take popular vote

The initiative is organized by national non-profit CIVIX

Yvonne Clarke is the newly elected Yukon Party MLA for Porter Creek Centre. (Submitted/Yukon Party)
Yvonne Clarke elected as first Filipina MLA in the Yukon Legislative Assembly

Clarke beat incumbent Liberal Paolo Gallina in Porter Creek Centre

Emily Tredger at NDP election night headquarters after winning the Whitehorse Centre riding. (Stephanie Waddell/Yukon News)
Emily Tredger takes Whitehorse Centre for NDP

MLA-elect ready to get to work in new role

Chief Medical Officer of Health Dr. Brendan Hanley. (Crystal Schick/Yukon News)
Two new cases of COVID-19 variant identified in territory

“If variants were to get out of control in the Yukon, the impact could be serious.”

lwtters
Today’s Mailbox: Rent freezes and the youth vote

Dear Editor, I read the article regarding the recommendations by the Yukon… Continue reading

Point-in-Time homeless count planned this month

Volunteers will count those in shelters, short-term housing and without shelter in a 24-hour period.

The Yukon’s new ATIPP Act came into effect on April 1. Yukoners can submit ATIPP requests online or at the Legislative Assembly building. (Gabrielle Plonka/Yukon News file)
New ATIPP Act in effect as of April 1

The changes promise increased government transparency

Most Read