Statistician, heal thyself

The Fraser Institute has been given a poor grade in math by a BC university. The Fraser Institute’s controversial Report Card on Secondary…

The Fraser Institute has been given a poor grade in math by a BC university.

The Fraser Institute’s controversial Report Card on Secondary Schools in British Columbia and Yukon, was recently reviewed by Simon Fraser University.

The results are surprising.

Based largely on standardized tests, the institute’s report focuses on evaluating students’ skills in math and literacy in Canada’s westernmost province and territory.

Schools with the highest marks in arithmetic and English, rank highest on a scale of zero through 10.

Yukon schools fared poorly, with Watson Lake scoring the lowest, at zero.

But Simon Fraser’s review suggests the institute’s math is a tad wonky.

Its numbers are unreliable, according to the recently released Cautions About Rating BC’s Schools.

“We’re told the institute’s ratings of elementary schools, as well as the report it released in April rating secondary schools, are widely discussed,” the report reads.

“Although it doesn’t happen in BC, in the US some jurisdictions use ratings like these, along with other information, to decide how much funding schools receive.”

And what if the numbers are wrong?

“What we focused on is that they don’t include any indication of how accurate, how much measurement error there is in their number that they provide for each school,” said professor and co-author of the study John Nesbit.

All reputable polling organizations show how accurate a number is by giving a frequency (19 times out of 20) and a range (1.1 +/– for example).

“Our main issue with the Fraser Institute’s report is they did not do that,” said Nesbit.

“They never provide anything like a margin of error.”

This raises serious questions about how accurately schools are ranked.

The three academics did their own calculations.

They found that a school’s grade on the report card was accurate within a range of 1.1 on the institute’s scale.

This means that each school could actually be 1.1 point higher or lower than the grade it was given, 19 times out of 20.

“That’s a pretty wide range,” said Nesbit.

“A lot of the schools, in any one area, are often well within that range.”

This is significant.

“When our children and our collective futures are at stake, we can’t afford to behave like the novice carpenter: measure once, cut twice,” reads the report.

“Once opportunities for students and schools are cut, changing that decision can be very difficult.”

It may seem far-fetched that numbers, on one report, from one study, from one independent research firm pose a threat to our future, especially in the Yukon.

The territory’s schools have only recently been included in the index.

With more than half the categories based on exam marks for courses testable by BC provincial exams, many of the territory’s own courses are excluded.

For programs like Music, Arts, Drama and experiential science at the Wood Street school, there is no measurement on the report card.

Many prominent officials in the territory’s education system outlined numerous shortcomings in the report’s evaluation of Yukon schools.

But in Vancouver’s Lower Mainland, parents are uprooting their kids to live in districts with the top-scoring schools.

“Sometimes parents are actually selling their house and moving so they can get their children into a school that’s higher in the rankings,” said Nesbit.

“But they might be doing that for no good reason because the difference could just be based on measurement error, just a statistical blip.”

It’s also important to define what the report card actually measures, said Nesbit.

It’s based on standardized tests with heavy emphasis on numeracy and literacy.

“If your main educational goals are something different than those, of course it would not be valid,” said Nesbit.

How important teachers believe the test is can also influence the results — some take weeks to prepare their classes while others do no prep at all, depending on the philosophy of the school and the parents, added Nesbit.

With such a small number of students in some of the territory’s classrooms, the numbers may also be less accurate. The smaller the school the more room there is for error.

And the results are geared towards university entrance and favour private schools, Yukon school superintendent Lee Kubica said in an earlier interview with The News.

“A public school allows all students, regardless of ability and work ethic, and all sorts of other things that go into making a good student,” said Kubica.

By overemphasizing “the average,” the report card does not fairly evaluate schools, said Yukon Teachers’ Association president Sandra Henderson in April.

“Much can be done to explain the relative comparisons,” said Henderson.

“Socioeconomics, community conditions, physical or emotional concerns among others.”

While all of these factors question the validity of the report card, there is some value to standardized testing, said Nesbit.

“(Standardized tests) are designed for particular purposes; they are designed to help the ministry monitor schools in a certain way,” he said.

Governments likely have an understanding of what limitations are attached to these tests, he added.

“But there’s a danger in taking them out of that context and using them for something they weren’t designed for. And I think that’s what we’re seeing here.”