Regional ship on the amazon river (Photo credit: Wikipedia)

Every day we produce 2.5 Exabytes of data. How big is an exabyte? It’s a quintillion bytes. How many in a quintillion? 1 followed by 18 zeroes. Still struggling to visualise it? A quintillion pennies, if laid edge to edge, would cover the Earth two deep. If arranged in a cube, would measure 5 miles a side.

You can’t go to a conference these days without someone telling you that we produce more data today in the blink of a gnat’s eye than we did from the dawn of creation up to when something happened a long time ago. Every day, more people start using the Internet. More people join social networks. More social networks are created. More connected devices are manufactured. They all produce data.

Despite the soundbites, I doubt anyone really knows how much data we produce now, let alone how much we produced 10 or 20 years ago. Any figures you hear must be based on estimates on top of estimates, but no-one doubts we are producing a lot and over time the amount of data we produce increases at an ever-growing pace.

When it’s growing this fast and when we have this much, coping with it becomes a challenge. It’s a bit like trying to analyse the Amazon river. Every second, the Amazon spits out roughly 55 million gallons. Even the largest tank in the world would fill in less than a heartbeat. We have to use a different strategy. Either we sample the water every so often and extrapolate or we find a way whereby when something we’re interested in passes by – we get a message.

It’s no surprise that the software industry recognises the need to process and analyse all this data and it’s even less of a surprise that buzzwords have come about to describe the process. Big data is big business and it’s easy to see why. By analysing weather patterns, large retail chains can make a good guess about what’s going to sell and stock their shops accordingly. By analysing web searches, astonishingly accurate predictions can be made about election results or the potential success of a film or a music artist.

However clever all this seems, I can’t help thinking that we are like toddlers discovering our first toy. The potential in all this data is enormous. Who knows, we may be able to predict earthquakes, volcano eruptions, traffic jams, epidemics and murders by analysing everything from how many big macs are eaten in Bolton through to the water temperature in Tahiti.

Related articles

• What the Hell is “Big Data”? (onelastpage.wordpress.com)
• Big Data: Moving From an Ocean to Stream Culture Will Hurt Analytics (business2community.com)
• Big Data Infographic | How Big is Big Data? | Domo | Blog (domo.com)
• Infographic: The Physical Size of Big Data (domo.com)
• Do You Have What It Takes To Be a Data Scientist? (semanticreatures.com)
• Just How Far Could Big Data Drive Us? (sys-con.com)
• Big deal? (planetblog.co)
• Is blogging the cats whiskers? (mcgrewbarney.wordpress.com)
• Wired Looks at the NSA’s $2 Billion Datacenter in Utah (insidehpc.com)
• Capturing Value from Data in Motion in the Internet of Everything (blogs.cisco.com)