Learning from the evidence

A wag once said that economists liked to look at something that worked in practice, and then study it to see if it worked in theory. But a new generation of economists is turning this stereotype on its head.

A wag once said that economists liked to look at something that worked in practice, and then study it to see if it worked in theory.

But a new generation of economists is turning this stereotype on its head. Instead of theorizing about why people make the economic decisions they do, these economists use randomized trials to test how real-world people behave.

Esther Duflo and Abhijit Banerjee have recently published a fascinating book on the topic: Poor Economics. In it, they look at why so many well-meaning development aid projects fail. And they convincingly answer some tricky questions, including a remarkable case study on the best way to teach African teenagers about how to avoid getting HIV/AIDS.

While the book’s focus is development, it raises plenty of questions about why some social and economic problems in rich countries like Canada seem impervious to the government programs we throw at them.

The heart of their approach is the randomized trial. Perhaps best known for testing new drugs, this is where you take a group of people, divide them randomly into groups, and give each group a different treatment. There is the “control group,” where you give them a placebo, and then one or more treatment groups who get the drug being tested.

Since the subjects have been selected randomly, a properly executed randomized trial with a large enough sample removes all the noise from the experiment. The control group and the treatment group subjects share the same characteristics; the only difference is whether they got a placebo or the drug being tested.

So you can be pretty confident that if the treatment group gets better and the control group doesn’t, it’s because of the drug. Not because they have different lifestyles, live in a part of town with better access to health care, or some other reason.

An example shows how effective this can be in public policy. Consider HIV/AIDS, which infects nearly two million Africans every year with horrific social and economic consequences. An economist named Pascaline Dupas from UCLA asked herself what the best way was to educate African teenagers to avoid this scourge.

The control group was teenagers who got the usual national curriculum focused on abstinence. The treatment group instead got “risk reduction” information, including the critical fact for young women that sex with older men is much riskier since they have a much higher infection rate than boys their own age.

Dupas measured teen pregnancy in the two groups, as a good proxy for unsafe sexual behaviour. After the study, the control group following the official curriculum showed no change. But teen pregnancy among young women given the risk reduction information went down 28 per cent.

Read the details in Dupas’s paper. They are hard to argue with. Her study has huge implications for the millions of dollars spent on HIV/AIDS programs in Africa every year, suggesting that much of the money is being spent ineffectively. Governments now have evidence to rethink their focus on politically popular but less effective abstinence training. With millions of people being infected every year, this is enormously important.

Randomized trials are increasingly common in business. Online retailers test different page layouts or even the shape and colour of “Proceed to check out!” buttons to see what works best. Barack Obama’s campaign team famously used randomized trials to see which images on the candidate’s website elicited the most donations, netting millions in extra cash.

Gary Loveman, a Harvard economist turned casino CEO, uses randomized trials to coax more money out of visitors’ pockets at Caesar’s. Since many gamblers use swipe cards at the casino, he can track how they behave. He can test offering free dinners or hotel rooms to first-time visitors having a bad day, and see if it makes them more likely to return a second time (it does, by the way). I once heard Loveman on Planet Money saying that there are three things that will get you fired from the casino: stealing, sexual harassment and running an experiment without a control group.

Governments are slower to adopt the practice. We identify a problem, spend a lot of money designing and implementing programs, and then can’t tell if it worked or not.

Say we are paying for more training for the unemployed. If more get jobs, is that because of the training or because the economy got better? Or we spend a lot of money on police and the crime rate went down. Was that because of the extra police, or the fall in the number of 18-25 year-old males as the population aged?

What is something in the Yukon where we might experiment with these techniques? The high school drop-out rate is one idea. Victoria Gold has recently donated a bunch of money for stay-in-school activities.

In a traditional government program, officials would think of some ideas on how to spend the money effectively. Then a few years later, the program would be evaluated and the results would be inconclusive. Was it the stay-in-school programming, improvements in course offerings at school, a slow economy or changes in socio-economic characteristics in the school population?

If the government was clever, it would have a control group and test two or three different stay-in-school strategies against it. Perhaps one-on-one mentoring works best? Or cash bonuses for attendance? Or better information on job opportunities?

It’s hard to know for sure without hard randomized trial data.

Some might say that it’s wrong to do randomized trials on teenagers like this. But the sad fact is that the government already experiments on them with new policies every few years. It just doesn’t have much evidence whether its experiments work or not.

Keith Halliday is a Yukon economist and author of the MacBride Museum’s Aurore of the Yukon series of historical children’s adventure novels.

Get local stories you won't find anywhere else right to your inbox.
Sign up here

Just Posted

Members of the RCMP’s traffic services team examine police markers on Range Road after a six-year-old boy was struck by a vehicle near the Takhini Arena in Whitehorse on Oct. 25. (Crystal Schick/Yukon News)
Six-year-old hit by vehicle near Takhini Arena

Police were called to the scene around 12:15 p.m. on Oct. 25

Dr. Brendan Hanley, Yukon’s chief medical officer of health, speaks to media at a press conference about COVID-19 in Whitehorse on March 30. Two new cases of COVID-19 were identified in Watson Lake over the weekend. The cases are connected to three others in the community previously announced by officials on Oct. 23. (Crystal Schick/Yukon News file)
Two additional COVID-19 cases in Watson Lake bring total up to five

Individuals with symptoms and connections to the three other cases were tested over the weekend

Dr. Brendan Hanley, Yukon’s chief medical officer of health, speaks to media at a press conference about COVID-19 in Whitehorse on March 30. The Yukon government announced three new cases of COVID-19 in Watson Lake on Oct. 23. (Crystal Schick/Yukon News file)
Three new COVID-19 cases identified in Watson Lake

The Yukon government has identified three locations in town where public exposure may have occurred

Teagan Wiebe, left, and Amie Wiebe pose for a photo with props during The Guild’s haunted house dress rehearsal on Oct. 23. The Heart of Riverdale Community Centre will be hosting its second annual Halloween haunted house on Oct. 30 and 31, with this year’s theme being a plague. (Crystal Schick/Yukon News file)
Plague-themed haunted house to take over Heart of Riverdale for Halloween

A plague will be descending upon the Heart of Riverdale Community Centre… Continue reading

The Yukon government is asking for all claims in a lawsuit over the Takhini elk herd be struck by the court. (Mike Thomas/Yukon News file)
Yukon government asks for Takhini elk lawsuit to be struck

The Yukon government is asking for all claims in a lawsuit over… Continue reading

The Yukon government has filed a reply to an outfitter’s petition challenging the reduction of its caribou quota to zero. (Yukon News file)
YG replies to outfitter’s legal challenge over caribou quota

The Yukon government has filed a reply to an outfitter’s petition challenging… Continue reading

The Yukon government is encouraging people to get the flu vaccine this year, saying that with COVID-19, it’s “more important than ever.” (Black Press file)
Get flu vaccine, Yukon government urges

The Yukon government is encouraging people to get the flu vaccine this… Continue reading

Benjamin Munn, 12, watches the HPV vaccine in 2013. Beginning Jan. 1, 2021, the Human Papillomavirus (HPV) vaccine will be available to all Yukoners up to, and including, age 26. Currently the program is only available to girls ages nine to 18 and boys ages nine to 14. (Dan Bates/Black Press file)
HPV vaccine will be available to Yukoners up to, including, age 26

Beginning Jan. 1, 2021, the Human Papillomavirus (HPV) vaccine will be available… Continue reading

Whitehorse City Hall. (Joel Krahn/Yukon News file)
City hall, briefly

A look at decisions made by Whitehorse city council this week

asdf
COMMENTARY: Me and systemic racism

The view from a place of privilege

asdf
Today’s mailbox: Electricity and air travel

Letters to the editor published Oct. 23, 2020

Yukonomist Keith Halliday
Yukonomist: Irony versus Climate

Lately it seems like Irony has taken over as Editor-in-Chief at media… Continue reading