Big Data: water wordscape (Photo credit: Marius B)

Information is the most valuable tool we have. Without reliable information, decisions can only be made on hunches. One of the main benefits of computers was that they were supposed to help us deal with information, but there are some unfortunate side effects. Partly because of our hunger for ever more power and partly because of progress in making bigger and better computers, they have steadily grown ever since they were invented in a phenomenon is widely referred to as Moore’s law.

Not only that, but devices have become cheaper and cheaper. The Sinclair Spectrum was priced at £179 at launch 30 years ago, a sum probably equivalent to double that now. That was a machine with no screen, 48K of memory and no storage. Nowadays you could have a tablet, or a phone or even a couple of netbooks all with power that would completely dwarf the humble spectrum. As computers have become more affordable, they have proliferated.

And not just consumer devices. It now makes perfect sense to use computational devices in almost every setting thanks to the cheapness and the ingenuity of the Chinese. IBM told me at impact that there are 9.5Bn connected devices in the world today growing to 20Bn in 2015. All of these devices are producing data at an alarming rate. Storage companies fall over themselves to tell you how fast this data is growing, but take it from me – it’s exponential.

The other side effect of computers is that the average human being now has the attention span of a hyperactive goldfish. Because the answer is available from Google in 0.2 nanoseconds, why would you want to read a report or a book? Twitter delivers a constant stream of news in 140 character long chunks. This means that we are no longer happy to do lots of analysis to get our answers, we want to know instantly.

So how do we make sense of this muddle?

I have always been a fan of infographics as a mechanism for presenting complex information. Ingeniously through pictures, infographics helps the reader to understand the big picture. Spatial and relative relationships are easily picked out, but detail is not. Newspapers use info graphics to great effect – especially when trying to explain concepts like exactly how much money we all owe thanks to the financial meltdown. This is a great site to see some really good examples; http://www.coolinfographics.com/

Big Data was a much vaunted concept at IBM Impact too. If you want to process the huge amount of data available today, then traditional methods are just not going to work. Reading all the data into a relational database and then running a query will mean you need a huge building to house the storage, a power station to feed the computers and half a century to wait for the result. The only way to process this data is as a stream – a bit like the water coming out of a firehose. Then you need to use something called complex events processing to look for relationships between the data.

One of the most interesting concepts in dealing with the information overload is the semantic web. The idea is that computers need to make sense of all this data themselves and answer any questions we have. This is the technology behind the Siri voice response software in the latest iPhone. There is even a search engine. It’s not perfect by any means – try searching for “Who is the richest man in France” or my personal favourite “Who is the sexiest woman” 🙂 But what is interesting is how often it does get it right.

One day – the characters in Star Trek the next generation talking to their computers will look positively antiquated – and I for one can’t wait.

Related articles

• Help me, help me. I’m lost in information overload!!! (greeneyezwinkin4.wordpress.com)
• Curation contributes to Information overload (FUMSI) (scienceintelligence.wordpress.com)
• Information Overload Can be a Misleading Term (billives.typepad.com)