The year 2013 is finally here and, as you’ve all been waiting for, it’s the International Year of Statistics.
The American Statistics Association has even set up a snazzy website, complete with promotional video by the SAS Institute on how statistics improve human welfare (SAS is a statistics software package used when you have some serious data crunching to do).
This is a well-chosen year because, even if you aren’t attending the World Statistics Congress in August, statistics are likely to have an ever increasing impact on your life. We are at the cusp of the age of “Big Data.”
Big Data is the buzzword for the vast clouds of information that organizations are using to gain ever more granular insights into weather, traffic, econometrics and – perhaps most importantly – your behaviour. Huge amounts of what you do are captured digitally now: webpages, purchases, texts, digital photos tagged by your friends, and so on. And via your smartphone, you and your behaviours are now tracked locationally.
This will let you quickly search nearby for things you want. It will also open vast new opportunities for companies to micro-target you with spam. I already notice that I get different prices on websites depending on which Internet connection or account I am using. Companies also treat customers differently, often linking their phone number or account with past behaviour and adjusting prices and service levels. Expect much more of this to come.
And it’s not just private companies doing this. Governments are trying too, perhaps most successfully in the communications surveillance world where electronic intelligence agencies have been making huge investments. But also political parties. One reason for Barack Obama’s recent presidential win was that his campaign created a giant database with a file on every U.S. voter, layered with additional data on magazine subscriptions, past voting patterns and other personal information from various sources. This allowed them to micro-target very specific messages, and to figure out the best way to get likely Obama voters off the couch and into the voting booth.
McKinsey Global Institute, the research arm of the big consultancy, says that Big Data is breaking down the boundary between the IT department and general managers. Organizations where the IT department is viewed as another world, and general managers make decisions by gut feel or a few easy-to-get datapoints, will become increasingly archaic.
And this is not just in Silicon Valley outfits. McKinsey thinks Big Data will affect every sector, including government, and all functions from sales to manufacturing to customer service. In a recent study, McKinsey estimated U.S. health-care providers could save US$300 billion a year using technologies that exist today to improve quality and efficiency. Government officials in Europe could ring up 100 billion by using Big Data to reduce errors, cut fraud, and improve tax collection.
Big Data is rolling out unevenly. The garage in Whitehorse knows exactly how often I change the oil, and when I got my alternator fixed. During my last visit to the emergency ward, they were using clipboards with no access (as far as I could see) to my personal medical history. They even asked important questions like: “Are you allergic to any medications?” to unreliable people (i.e., me, the patient).
Big companies and governments are in the lead on Big Data, but it is increasingly seen in smaller settings too. The engineers who designed the heating system for the new F.H. Collins, for example, had a model with year-round, minute-by-minute weather data from Whitehorse. They could model how many minus-40 hours the school would face, plus the wind and sun patterns. They could do this at night and during school hours, and estimate the impact of various heating options and window configurations. Impressive stuff.
Those who think the Yukon is too small for such fancy techniques should consider the example of Sean Ryan, Prospector of the Year a couple of years ago. His teams of “groundtruthers” criss-crossed the Yukon methodically taking soil samples and generating reams of data. Ryan then integrated this data with GIS and other geological data to make some impressive discoveries like White Gold.
Say what you will about number crunching, there is no doubt that there are a lot more zeroes in Mr. Ryan’s bank account than in those of the prospectors who trudged across the same valleys for decades using more traditional methods.
The geoscience forum is another place to see 3D imagery produced by the massive databases used by Yukon mining companies, combining datastreams from drilling, seismic, magnetic and other sensing technologies. Without this data and the advanced software to interpret it, some properties would probably never be found or mined.
So where does Big Data go next in the Yukon? For possibilities, look for organizations that interact in a complex way with large numbers of customers.
The health system is a classic. Future managers will be expected to know much more about different patient populations, from the “frequent flyers” in our emergency wards to people on multiple medications from multiple doctors.
Bus-system managers would probably like to know a lot more about who rides the bus, where they get on, and where they are going. Periodic traffic studies just aren’t the same as the reams of real-time data that transit systems with smart cards have.
The education system is another. A few years ago, the auditor general dinged the department of education for its inability just to figure out the drop-out rate. In the future, educators will be able to sift through the data in real time looking for patterns between real student attendance, achievement and dropping out. Somebody will invent an app so the school can text the parents when Junior skips Socials 10 (and they’ll probably be able to see that Junior’s iPhone is at Riverside Grocery rather than F.H. Collins).
If all of this seems a bit creepy, that’s because it is. We will see many benefits from Big Data, but also some major privacy issues.
But Big Data is not going away. And as McKinsey points out, those who know how to use it will have a big competitive advantage over those who don’t.
Keith Halliday is a Yukon economist and author of the MacBride Museum’s Aurore of the Yukon series of historical children’s adventure novels