My data – it’s just so big!

Regional ship on the amazon river

Regional ship on the amazon river (Photo credit: Wikipedia)

Every day we produce 2.5 Exabytes of data. How big is an exabyte? It’s a quintillion bytes. How many in a quintillion? 1 followed by 18 zeroes. Still struggling to visualise it? A quintillion pennies, if laid edge to edge, would cover the Earth two deep. If arranged in a cube, would measure 5 miles a side.

You can’t go to a conference these days without someone telling you that we produce more data today in the blink of a gnat’s eye than we did from the dawn of creation up to when something happened a long time ago. Every day, more people start using the Internet. More people join social networks. More social networks are created. More connected devices are manufactured. They all produce data.

Despite the soundbites, I doubt anyone really knows how much data we produce now, let alone how much we produced 10 or 20 years ago. Any figures you hear must be based on estimates on top of estimates, but no-one doubts we are producing a lot and over time the amount of data we produce increases at an ever-growing pace.

When it’s growing this fast and when we have this much, coping with it becomes a challenge. It’s a bit like trying to analyse the Amazon river. Every second, the Amazon spits out roughly 55 million gallons. Even the largest tank in the world would fill in less than a heartbeat. We have to use a different strategy. Either we sample the water every so often and extrapolate or we find a way whereby when something we’re interested in passes by – we get a message.

It’s no surprise that the software industry recognises the need to process and analyse all this data and it’s even less of a surprise that buzzwords have come about to describe the process. Big data is big business and it’s easy to see why. By analysing weather patterns, large retail chains can make a good guess about what’s going to sell and stock their shops accordingly. By analysing web searches, astonishingly accurate predictions can be made about election results or the potential success of a film or a music artist.

However clever all this seems, I can’t help thinking that we are like toddlers discovering our first toy. The potential in all this data is enormous. Who knows, we may be able to predict earthquakes, volcano eruptions, traffic jams, epidemics and murders by analysing everything from how many big macs are eaten in Bolton through to the water temperature in Tahiti.

The synergy of the network

A segment of a social network

A segment of a social network (Photo credit: Wikipedia)

One of the concepts I found challenging to grasp many moons ago during a course in business and finance was synergy. Maybe it was because I was only 16 at the time and the sum total of my possessions added up to one Casio calculator and a dog-eared biro but when he said that the whole was greater than the sum of its parts – I struggled to see what he meant. Of course I now understand that if you deliver two tons of metal, a few bits of rubber and some plastic to Massey Ferguson, you get a very different product compared to the same shipment arriving at Bentley with a commensurate difference in value.

Almost every mobile phone is pretty much the same form factor and made of similar materials and yet we place very different value on different brands. The whole perfume industry relies on people forgetting the inconvenient fact that the product they are selling is nicely scented alcohol served up in a nobby bottle as they hand over a small fortune.

Those who know me well will know that I enjoy playing games. Not the sort that appear on a computer screen because I spend far too much of my life staring at pixels as it is. I enjoy social games where the participants are vaguely humanoid. Some being more vague than others. The most satisfying games I play are also the most difficult to explain. Games which rely on the collective imaginations of the players concerned can result weaving amazing stories. The reason these games work is synergy once again.

The idea of a social network is not new. In the early 1930s, a self-published psychologist named Dr. Jacob Levi Moreno introduced the sociogram, the first formal attempt to map out the relationships within a group of people. Moreno’s sociogram — a cluster of individual points, or “nodes,” connected by straight lines — became a powerful tool for identifying social leaders, outsiders, and what he called the “sociometric star,” the person to whom all others are connected. [Source: Psybernet]

Social networks are built on synergy.

Facebook made its début on the markets this week amid a flurry of law suits and accusations. If Facebook were a country, it would be the third biggest with only China and India boasting larger populations. Something like 1 in 5 couples meet through social media. Opinions vary as to how much time people spend on social networks, but whichever estimate you believe, it is a very large number. Each individual’s contribution to the network is not worth much, but taken together, the value of all the contributions is much higher. Whether that justifies Facebook’s valuation I don’t know but I certainly didn’t buy any shares.

After one of those conversations one sunny lunchtime at the local watering hole where everyone walks away thinking they are going to change the world, we decided to get Yammer up and running. Yammer is a bit like Facebook for companies. My colleague installed it and sent me an invitation. When I signed up, I was asked to invite some other people. After 6 months or so, half the company was online after no promotion and no backing from the company. It’s fair to say that Yammer has revolutionised the way that we collaborate and communicate within the company.

There is a massive proliferation of different social networks available. Where they overlap, usually a winner emerges and the others die off. Where they are complementary, they are linking together. Already, a post you make on WordPress or a fresh pin on Pinterest can automatically appear on Twitter & Facebook. One day, they will become seamless and it won’t matter which social network(s) you are signed up to. Then we will really see some synergy.