As the capabilities of digital devices soar and prices plummet, sensors and gadgets are digitising lots of information that was previously unavailable. There are many reasons for the information explosion. Data, he explains, are widely available what is scarce is the ability to extract wisdom from them. Hal Varian, Google's chief economist, predicts that the job of statistician will become the “sexiest” around.
This industry is estimated to be worth more than $100 billion and growing at almost 10% a year, roughly twice as fast as the software business as a whole.Ĭhief information officers (CIOs) have become somewhat more prominent in the executive suite, and a new kind of professional has emerged, the data scientist, who combines the skills of software programmer, statistician and storyteller/artist to extract the nuggets of gold hidden under mountains of data. In recent years Oracle, IBM, Microsoft and SAP between them have spent more than $15 billion on buying software firms specialising in data management and analytics. The business of information management-helping organisations to make sense of their proliferating data-is growing by leaps and bounds. Given enough raw data, today's algorithms and powerful computers can reveal new insights that would previously have remained hidden. But this special report uses “data” and “information” interchangeably because, as it will argue, the two are increasingly difficult to tell apart. Scientists and computer engineers have coined a new term for the phenomenon: “big data”.Įpistemologically speaking, information is made up of a collection of data and knowledge is made up of different strands of information. The effect is being felt everywhere, from business to science, from government to the arts. Joe Hellerstein, a computer scientist at the University of California in Berkeley, calls it “the industrial revolution of data”.
“We are at a different period because of so much information,” says James Cortada of IBM, who has written a couple of dozen books on the history of information in society. “How to make sense of all these data? People should be worried about how we train the next generation, not just of scientists, but people in government and industry,” he says. Alex Szalay, an astrophysicist at Johns Hopkins University, notes that the proliferation of data is making them increasingly inaccessible.