Tag Archives: IBM

Nov 21 2013

How temporary is your temporary solution?

English: A roll of silver, Scotch brand duct tape.

English: A roll of silver, Scotch brand duct tape. (Photo credit: Wikipedia)

To say that ending up with wet feet was a surprise is a bit of an understatement. On the first day in our new place, I did the washing up. When I emptied the bowl down the plughole, I expected the water to disappear, not to suddenly reappear at ankle level. A quick check under the sink revealed the issue. The previous owners had a dishwasher (which they took with them). The drainage from the dishwasher connected to the down pipe.

Now it wasn’t there, a gaping hole stood where the join was. To remedy the situation, we either needed a dishwasher, the right sort of plumbing doohickey to cap the pipe or a plastic bag and a hairband. Guess which of the three we had? The temporary repair did a sterling job of keeping my feet dry and was still there 4 years later when we finally bought a dishwasher.

When I joined BP, there was a carbuncle on the side of the office at ground level. It was one of those temporary portacabin type affairs. I asked someone how long it had been there. 10 years! A whole decade. My school had 2 temporary classrooms when I left 27 years ago. They have 4 now. IBM moved into Hursley in 1958 as a temporary measure. They are still there today.

A temporary solution or workaround can be a Godsend. When you’re about to go live with a big system and horror of horrors, you find a problem. You don’t want to accept the risk of taking a new release where new problems might be lurking. But temporary solutions have a nasty habit of becoming permanent. When you come up with a temporary solution, it’s important to understand the inherent drawbacks of not doing it properly. It’s also good to think about when (or if) it might be replaced. If the permanent solution is not under construction or at least in the planning stages right now, the chances are that workaround could last for a very long time.

If that’s the case, it might well be worth putting in a little extra effort to do a good job. Maybe go the whole hog and fix it properly.

Sometimes permanence is a good thing. The Eiffel Tower was originally built for the 1889 Exposition Universelle but caused such a stir they decided to keep it. Sometimes it’s not. Never believe a politician when they tell you a tax is temporary. As Craig Bruce once said; Temporary solutions often become permanent problems.

Related articles
Mar 19 2013

In the cloud

IBM Cloud Computing

IBM Cloud Computing (Photo credit: IvanWalsh.com)

One of the more interesting areas I have been lucky enough to work with is cloud. We run some banks live in the public cloud. The scale of the cloud offerings is simply staggering. There are three main big players; Microsoft, Google & Amazon. Between them, they bought 31% of the world’s CPUs in the last two years and Microsoft alone is pumping a billion dollars a year into the initiative.

I was surprised to learn that the datacenters that make up the cloud are not stocked with the latest generation machines, because slightly older stock makes much more sense on a dollar per CPU cycle basis. Also, the economics are quite surprising. CPU power accounts for less than 1% of the overall cost of the data centre. Between 50 – 90% of power that feeds into the datacenter is eaten up by lighting, cooling, transforming etc. and a sizable proportion is taken up by the storage arrays.

Siting the datacenters is extremely important. The first consideration is power. You need a lot of it and it needs to be reliable & cheap. You also need good internet links. There’s no point siting your fledgling data centre next to a nice shiny power station if people can’t connect into it. Climate is another consideration. Obviously, the hotter the country, the more your cooling costs will be. The last consideration is security. Obviously, the more the customer base builds up, the more attractive a cloud data centre becomes as a terrorist target. The ideal site for datacenters is Iceland. The climate is cool, there is plenty of cheap geothermal energy and all transatlantic internet links pass through Iceland. The UK is a very expensive place for a datacenter due to high land and labour costs.

The datacentres are populated using pre-fabricated containers containing 5,000 CPUs in readymade racks, preconfigured with storage. When the container is delivered, they plug in power, network & water (for cooling) and the container starts initializing. Not all the servers in the container can start up at the same time as the thing would melt, so they start up in waves equally spaced around the container. A typical datacenter will have many of these containers with typically 2 – 3 being added every week.

With this many machines, hardware failure is a fact of life. Anything up to a 3% failure rate is business as usual. If the failure rate hits 10%, the machine vendor dials in and performs remote diagnostics on the container. If the failure rate hits 30%, the machine vendor sends out an engineer to inspect the container and perform any remedial action necessary. If the failure rate hits 50%, the container is powered down and replaced.

I was surprised by how flexible the Azure cloud is. There are three ways you can use it. The first is what they call the “web role”, which allows any kind of web-based activity (web pages, active server pages, web services etc). The second is what they call the “worker role” which allows you to do pretty much anything providing the software you want to run does not require installation (i.e. it can be copied across) and does not need to access the registry.

There were big challenges to overcome. Because you never really know which machine your application is running on or where your data is, it gives regulators a heart attack. There is also the fact that the database is what they call “eventually consistent”. Because your data is spread around and copied onto different machines, there is no real backup as we would normally consider it. You can back up your data to a local backup, but if your database is big, you may want to take advantage of the service they offer whereby they post you a tape once a day. There is no such thing as a printer connected to the cloud so if you want a hard copy of anything, you have to get creative.

The costs of a cloud based solution are *very* compelling, especially when you consider that you are effectively getting a fully managed, fault tolerant data centre. There are a number of pricing options depending on how much you want to commit to. There is everything from pay as you go, right through to volume licensing. If I had my way, I’d put every server in the cloud.

Related articles

 

Nov 12 2012

Technical Isolation

Hursley House - geograph.org.uk - 967947

Hursley House – geograph.org.uk – 967947 (Photo credit: Wikipedia)

If you ever get a chance to visit IBM‘s Hursley Park campus, it’s well worth going. It’s a lovely mansion-house in beautiful surroundings just outside Winchester. There has been a lodge at Hursley since 1413 and the current house dates from 1724. The Heathcote family donated the house together with £5m (a figure equivalent to £116m today) to provide a hospital for US military personnel in WW1. In WW2, the house was used by Vickers to develop the Spitfire. IBM moved in during 1958 as a temporary measure and they are still there today.

The System 360 and the Winchester hard drive were developed there and IBM were granted the Queen’s award for industry for storage & CICS. Today it houses 2800 IBM employees – roughly half of which are software developers making it the biggest software lab in Europe. Altogether IBM has 59,000 employees of which roughly half are based in the USA. They are heavily into the R part of R&D with research going on into particle physics and nanotechnology among other things.

On the Hursley campus, they have an isolation lab which is a completely shielded building used to test the electromagnetic emissions of devices to ensure compliance with the plethora of regulations governing such things. They really needn’t have bothered. There is a perfectly good isolation zone on the South Coast of the UK just outside Highcliffe.

A popular retirement spot, it’s a tiny village which must be the only place in the UK where the funeral parlours outnumber the charity shops. Nothing on any wavelength passes into or out of the place which renders mobile phones completely useless. A couple of times a year, myself and some friends spend a long weekend there and bearing in mind my attachment to social networks and blogging, I find myself isolated for the whole time.

Initially, I find the experience anxious. I can’t check twitter, Facebook, yammer, wordpress or linked in. I can’t phone anyone (unless I search for some change and use the call box). I can’t look anything up on Google or check the news or how the FTSE is doing. I can’t even download the Times.

After a while though, the anxiety ceases and I feel liberated. I get used to being away from email and social networks. I even start to like the idea that I am completely out of contact with anyone. Maybe we should go there more often! Of course, on return, the first thing I find myself doing is checking my email, blog, social networks etc. but it was nice while it lasted.

Related articles
Nov 7 2012

Predicting the future is a very tricky business

future

future (Photo credit: Sean MacEntee)

Predicting the future is a very tricky business. Probably because the rate of change in our lives means that by the time you have made the statement or written the article, time has moved on. Because of this, only the foolhardy or the charlatans make such predictions. Here are some famous quotes about the future to give you the idea;

“There are many methods for predicting the future. For example, you can read horoscopes, tea leaves, tarot cards or crystal balls. Collectively, these methods are known as nutty methods. Or you can put well-researched facts into sophisticated computer models, more commonly known as a complete waste of time.”
Scott Adams

“There is no need for any individual to have a computer in their home”
Ken Olson, President of Digital Equipment Corp

“640K [of memory] ought to be enough for anybody”
Bill Gates, founder of Microsoft

“Computers of the future may weigh no more than 1.5 tons”
Popular Mechanics Magazine

“One-hundred million dollars is way too much to pay for Microsoft.”
IBM 1982

“It would appear that we have reached the limits of what it is possible to achieve with computer technology — although one should be careful with such statements, as they tend to sound pretty silly in five years.”
John Von Neumann 1949

“I have traveled the length and breadth of this country and talked with the best people, and I can assure you that data processing is a fad that won’t last out the year.”
Prentice Hall business book editor 1957

“This ‘telephone’ has too many shortcomings to be seriously considered as a means of communication.”
Western Union memo 1878

“We will never make a 32 bit operating system.”
Bill Gates, founder of Microsoft

“The horse is here to stay but the automobile is only a novelty – a fad.”
The president of the Michigan Savings Bank advising Henry Ford’s lawyer, Horace Rackham, not to invest in the Ford Motor Co., 1903

After Obama’s victory in the US elections, I thought that now is a good time to make some predictions about what the world might look like at the end of his 4 year term. I’ll leave it up to you to decide whether I’m foolhardy or a charlatan.

My version of the future goes something like this;

After  nearly a decade since the banking crisis, the world economies are booming once more. Growth is positive just about everywhere, but some countries are doing better than others. Africa is the new powerhouse of manufacturing with a constant flood of cheap consumer goods flowing both into the Western economies but also Eastwards into the burgeoning Chinese and Indian middle classes. Living standards are improving in Africa but there are many stories of exploitation, low wages and child labour.

After a long period of inflation in India and China coupled with stagnant wages in the Western world, Western companies are busy either repatriating their offshore operations or moving them to the much cheaper African continent.

The European union now operates two separate currencies; the Northern Euro (or Neuro) and the Southern Euro (or Seuro). The whole world is involved in a massive aid operation for Greece which is now a third world country thanks to the austerity measures and constant recession of the last 10 years.

Screens are rare and have largely been replaced by worn screens in the form of virtual reality spectacles and contact lenses. Keyboards are rarely seen, all input is carried out either via voice or gestures. Wireless networking is now free and universal so cables are a thing of the past.

Scientists are in the advanced stages of testing genetic treatments which render cancer an innocculable condition and treatments have been developed to cope with the effects of obesity and alcoholism.

Someone else will have invaded Afghanistan. The Israelis and Palestinians will still be at each other’s throats although peace talks will be looking promising. There will have been 3 gun massacres in the United States by crazed gunmen. After the last such incident in which a huge number of schoolchildren were gunned down, the USA are seriously considering new gun control laws.

Oil is now running at $20 per gallon and people have abandoned petrochemicals at an ever increasing rate. Renewable energy is the new oil and companies leading the field are booming. Most town and city centres have outlawed petrochemical powered vehicles. Global warming has stabilised and the Northern ice cap is starting to grow back once more.

Do I seriously believe my version of the future ? Well – no, 4 years is far too fast for some of the things on this list, but I live in hope that the more positive items in the list do become true and that we wake up in time to prevent the negative items.

Related articles
Oct 10 2012

Artificial intelligence?

IBM Watson (Jeopardy at Carnegie Mellon) - How...

IBM Watson (Jeopardy at Carnegie Mellon) – How I saved humanity! (Photo credit: Anirudh Koul)

People remember Alan Turing  for many different reasons. He was a British mathematician who worked as a codebreaker at Bletchley Park during World War 2. He also went on to become one of the pioneers of computing along with Max Newman. In 1952, Alan Turing was convicted of homosexuality. He accepted treatment with female hormones (or chemical castration) rather than go to prison and 2 years later committed suicide. In short, he was a genius who became a victim of his time. Were he born today, no-one would bat an eyelid at his homosexuality.

One of the his legacies is the Turing test. A machine could be said to be intelligent if it was indistinguishable from a human in conversation. He suggested that it would be better to come up with a learning machine (like a child’s mind) that could be taught and not something that simulates an adult mind. There is some debate as to whether a machine has ever really passed this test. I can think of a few humans who would struggle too.

Whenever I’ve seen any proffered example of artificial intelligence, I could not help but be disappointed. Today, however, I attended a very interesting session on IBM’s Watson semantic supercomputer. As supercomputers go, its $3m cost and 4TB of storage are pretty modest. Built on standard hardware and software, it resembles Turing’s child-like mind that can be taught. Indeed, before it can answer any sensible questions in a  particular domain, it needs to be fed with information. Lots of it. Even that’s not enough, Watson needs to do further research on everything it reads.

English: IBM's Watson computer, Yorktown Heigh...

English: IBM’s Watson computer, Yorktown Heights, NY (Photo credit: Wikipedia)

Once loaded up with information, Watson is ready for questions. The first step in being able to answer any question is to parse it into the component elements. From this, Watson can determine the type of question being asked and what sort of answer is expected. It then generates many different permutations of the question to give the greatest chance of coming up with the right answer. For each interpretation of the question, Watson searches (in parallel) its knowledge base to see what answers it can find, coming up with as many as possible.

For each of the possible answers, Watson goes on the hunt for evidence, both for and against. Based on this evidence, obviously incorrect answers are discarded and the remainder scored based on the quality and reliability of the evidence. Finally, Watson uses the experience it as gained in the past in answering similar questions to gauge the value of the different types of evidence and comes up with a final confidence rating for each answer and ranks them accordingly. The process is very similar to that used by Doctor Gregory House in the hit TV program. You write-up all the possible answers and cross them out as the evidence goes against them.

The first outing for Watson was to win the TV quiz game Jeopardy against two former champions. IBM admit that this was little more than a bit of fun and a publicity exercise. Watson is now being geared up for much more serious applications such as medical research and diagnosis support. The applications for such technology are legion and the team freely admit that there are far more valid use cases than they have the time to exploit right now.

For the first time in my life, I am genuinely inspired by an example of artificial intelligence and I will follow Watson’s progress with interest.

Related articles
Oct 7 2012

A change of identity

Baileys

Baileys (Photo credit: Ivana Di Carlo)

Changing a name is a risky business. Especially if you have an established brand name that’s known around the industry and yet some of the most famous corporations around us have been through a name change. Ever heard of Backrub? Probably not, and yet, that was the chosen name for the corporation that today is known as Google. What about Computing Tabulating Recording Corporation? It has to be said, IBM is a lot snappier.

When I was growing up, the name Datsun was synonymous with rusty old bangers that were reliable, but unfortunately fell to bits. Daewoos used to be the reliable but dreadful cars from Korea that were basically reincarnated Vauxhalls. Today, these cars are bang up to date and adorned with the name “Chevrolet“, which used to be synonymous with American muscle cars.

Her Majesty’s Royal Mail is a fine name for a postal service, in place since Charles the 1st, but the management thought differently. This fine moniker was superseded by “Consignia” because it was modern, meaningful and entirely relevant. The public didn’t think so, and the name quickly reverted to the one that had been in place for centuries.

It’s not just companies that change their names. Somehow, I doubt Michael Sinclair Vincent would have made it, so it’s probably just as well he changed his name to Vin Diesel. How about Frances Ethel Gumm? Somehow, Judy Garland has more star quality.

During a routine meeting the other day, we established that a third of the people were no longer known by the name they were given and I was one of them. I grew up as Martin Grimes. Mum remarried when we were kids and everyone took the new surname of Bailey, except me. I don’t really remember why, probably because I was in the middle of secondary school.

When we came to get married, we talked about names. I wasn’t attached to Grimes. I remember having to spell it out to a call centre worker three times once before the penny dropped and she said “Oh – Grimes. As in dirt”. I tried on my wife’s name for size; “Emburey”. I wasn’t sure about that either, so we agreed to take the name “Bailey” when we got married.

Every so often, I get a reminder of my past moniker when I find an old exam certificate or some old correspondence. Some of my friends I’ve known for a long time will occasionally call me Grimesy. I can’t say I miss my old name, and yet whenever I sign anything, it is my earlier name I scratch out in barely legible writing. Old habits die hard.

Related articles
Aug 18 2012

Stop polishing nose cones!

Rocket Engine, Liquid Fuel, H-1

Rocket Engine, Liquid Fuel, H-1 (Photo credit: cliff1066™)

Working on a “nice to have” feature when there are more important requirements to fulfil is a crime. It becomes a heinous crime when it happens in a resource constrained environment. And yet – I see it all the time. If you ever find yourself working on a nice to have feature – stop, and ask yourself “is this the most urgent problem that needs solving?”

We have a special term for such work which, according to my trusted sources, originated in the IBM lab’s. We call it “polishing nose cones“.

Imagine if you will, a factory building rockets. The man in charge runs a tight ship and he organises his factory into departments. The engine department takes care of the bottom of the rocket, the propellant and coolant department takes care of the mid-section of the rocket and the nose cone department takes care of the very top of the rocket.

The guys down at the business end of the rocket, the engine department, have their work cut out. They have to develop the rocket engine (or more correctly the rocket motor) which involves some tricky engineering. The engine guys have to come up with a rocket motor that will get the vessel into space without running out of fuel and without blowing the rocket into smithereens. Their work takes a long time.

The coolant and propellant guys also have a mountain to climb. They have their specifications from the engine guys and they are pretty demanding. Some how, they have to provide enough coolant to stop the engine consuming the rocket in a ball of flame, but enough fuel to make sure that the rocket can make it to orbit. Not only that, but they have to operate within strict weight criteria.

The nose cone guys have the easiest job of all. All they need to do is manufacture the pointy end. Sure they have weight constraints, but their only job is to make something aesthetically pleasing. So the nose cone guys finish long before the coolant and propellant guys and the engine guys still have a ton of work to do.

So do they go and help the other guys – no – because they are in the nose cone department. Once they have finished the essentials, they start on the “nice to haves”. They start polishing their nose cone.

If I ran the factory, I would get away from the department idea and create a resource pool. All the engineers would constantly be picking up the most important tasks on whatever part of the rocket. OK – so maybe my nose cone wouldn’t look quite as good – but I bet my rocket would be ready for launch first.

Related articles
May 20 2012

Beam me up Scotty!

 

Abstract icon of Enterprise NX-01 of Star Trek...

Abstract icon of Enterprise NX-01 of Star Trek Enterprise (Photo credit: Wikipedia)

After the big comfy armchair that was BP, my second employer “Pentyre” was more like a roller coaster. After the comfort of a blue chip company, I was brought down to Earth with a bump when I was summoned in to an all staff meeting the week after I joined. The entire company easily fitted into a small room. I was one of 7 software developers. The MD explained that a customer had reneged on a bill and the company was in real trouble. Some of the directors were going to forego salary for a couple of months and hopefully, everything was going to be OK. Had I made the mistake of my life ?

I very nearly ended up missing out on working at Pentyre altogether. I had already accepted a job with another company, but there was a recruitment consultant who just wouldn’t take no for an answer. He insisted I visit Pentyre to see what they had to offer. So after work one evening, I turned up outside their offices for an interview. First appearances weren’t too promising – a converted factory with a tatty sign outside. I was shown up the stairs into a demonstration suite. There were various devices around the place; pagers, phones, cameras, alarms, flashing lights.

I was interviewed by the MD himself. An unassuming middle aged man called Malcolm, impeccably dressed, short with wispy white hair. He gave me the potted history of the company, which didn’t take long. After a long career in IBM, he had taken his terms and used his severance money to start up the company. He then took me through a demonstration of what their software could do. It was a dazzling display as he made the various devices do their stuff with just a click with his mouse. The inner geek in me was hooked.

He then took me on a tour of the building. At the back of the building was an area that looked a bit like Q’s workshop out of the Bond movies. There were machines everywhere in various states of assembly. On the benches there were oscilloscopes and multimeters. Propped up against one wall was something called a protocol analyser. The last time I had heard of anything like it was during an episode of Star Trek. Dominating the middle of the room was a train set. I looked at Malcolm quizzically and he nonchalantly explained that the reason it was there was to test the train radio system they were developing for the London underground.

I was mesmerised. I simply had to work for this company. As we went back up the stairs to talk turkey, it was obvious I was hooked. Malcolm asked what it would take for me to go and work there. A brief exchange later, the deal was done. I started work there a few days later.

It was an amazing place to work. The nice thing about working as part of a small company is that every single person really makes a difference. Every few months or so, there was a new assignment – usually involving some brand new technology. My first job was all about pagers. Back then, if you had a large site, the only way to keep in touch with your workforce was to give them a pager. I also developed a building management system, some modelling software for a steelworks and CCTV systems for prisons and nuclear reactors.

My time at Pentyre taught me the power of small teams working together without constraints. The productivity was amazing. There was no demarcation; you sold, designed, built, tested, installed and supported every bit of software that went out the door. Not all the technology was all it was cracked up to be. We worked on a project to detect faces in crowds. I couldn’t get the face recognition software to recognise me standing perfectly still at less than two paces. Voice recognition was similarly inaccurate being unable to determine the difference between “Chicken Tikka Massala” and “Land Rover” even when spoken slowly and deliberately.

The technology failure that has tickled me to this day was DCOM. Launched with some fanfare, DCOM (or Distributed Component Object Model) was Microsoft’s attempt at a computing model for distributed computing. In order to test it, we set up two machines. On the first, we coded a simple button with the label “Sausage” which would then send a message to the second machine which would respond with the on screen message “sizzle”. It was the simplest test imaginable and we were used to getting such things working.

Even so, after a whole morning of messing about with various different settings, we simply couldn’t get it to work. Frustrated, we went off to lunch. When we returned – there was the “sizzle” message on the second machine – we obviously hadn’t left it long enough!

Related articles

 

May 19 2012

A big comfy armchair

Line art drawing of an armchair

Line art drawing of an armchair (Photo credit: Wikipedia)

I once asked a candidate during an interview why he had chosen to leave a company like IBM after 18 years. He told me that the company was like a big comfy armchair and that he wanted to pull himself out of it into the wide world to see what’s out there. Having worked for BP, I understood the analogy perfectly. Like most schoolchildren, I underwent a week of work experience. The teacher who was organising who was going where asked me what I would like to do. I didn’t have a clue. My aspirations varied wildly according to my whims and I had no firm idea what I would like to try.

He came up with a few options which didn’t sound particularly exciting before offering “How about office work?” I gave in more out of boredom than conviction and before I knew it, as a tender 15 year old, I found myself in the reception of one of the world’s biggest oil companies. I quickly learned I was going to work in the mailing room. Because it was all new to me, I was fascinated. I enjoyed working with people and felt mightily important when the telex alarm went off and I had to run up to the third floor and deliver an urgent message.

When it came to the choice between staying on at school to study A levels or to go out into the big wide world, I was again stumped as to the right thing to do. So almost by default I found myself staying on. After a few weeks, it didn’t really feel like I had made the right choice and I found it harder and harder to find motivation. My mum could sense what I was thinking and suggested to me that I look for a job. I knew exactly where I wanted to work. I had enjoyed BP so much, I wrote an unsolicited letter to them, citing my recent work experience and expressing a strong desire to go back and work there.

When I was invited in for an interview, I was cock a hoop. When the letter came through offering me the job, I was at school. Someone passed on the message to ring my mum at home. When she told me about the job offer, I literally danced down the corridor. A passing teacher admonished me but it did nothing to dampen my fervour.

I have a lot to thank BP for. During the 8 years I was there, I progressed from the mail room through a brief stretch in a clerical role before moving into Information Technology. Not many employers would have sent someone to college for 6 years, let alone given them a day off each week to do so. I was well trained in every respect having attended residential courses on everything from creativity to interaction skills. The meals at BP were legendary. Every lunchtime, we were treated to a 3 course meal of restaurant quality food for the princely sum of 5p. There was even a bar on premises. A heavily subsidised sports and social club offered regular trips to the theatre as well as excursions overseas.

So if it was all so good – why did I leave? I can certainly relate to the big comfy sofa analogy. To a certain extent, when you work for a company like BP or IBM, you become a little bit institutionalised. But it was more than that. When I joined BP, there were about 300 people in the systems division. When I left, there were just over a tenth of that figure after all the outsourcing, not great if you wanted a career in information technology. But if I’m really honest – the main reason for my departure was boredom. If you are writing software, oil is not the most exciting subject you could choose.

Some aspects of it interested me, like the exploration side and the huge structures at refineries and oil terminals. If you are a sci-fi fan like me, it’s very hard to stop yourself imagining them covered in stormtroopers. But when it comes down to it, it’s all about pumping around petrochemicals, storing them and changing them into other petrochemicals. Accountants joke about actuaries being boring, but I would imagine that even actuaries are quite exciting compared to the project that finally finished me.

Statuary reports.

If those two words sound boring, I really haven’t managed to convey the sheer mind numbing, soul sucking mundanity of them. When my boos told me my next project was producing statuary reports, I asked him what they were. “No idea” he said, but the government says we have to have them. And there were a lot of them. I was going to spend the next year of my life writing them. After a couple of months, I decided I just couldn’t take any more. I began to look for another job.

Those were the heady days when you put your CV out one Friday morning and by the afternoon, you would have 5 interviews lined up. By the following Friday, you would have at least two job offers. When I handed in my letter of resignation, my boss told me not to be so hasty and that he would see what he could sort out. A few days later, he called me in and offered me a big pile of cash. But only if I finished the statuary reports.

My head hit the desk with a thump.

Related articles
May 15 2012

Information overload

Big Data: water wordscape

Big Data: water wordscape (Photo credit: Marius B)

Information is the most valuable tool we have. Without reliable information, decisions can only be made on hunches. One of the main benefits of computers was that they were supposed to help us deal with information, but there are some unfortunate side effects. Partly because of our hunger for ever more power and partly because of progress in making bigger and better computers, they have steadily grown ever since they were invented in a phenomenon is widely referred to as Moore’s law.

Not only that, but devices have become cheaper and cheaper. The Sinclair Spectrum was priced at £179 at launch 30 years ago, a sum probably equivalent to double that now. That was a machine with no screen, 48K of memory and no storage. Nowadays you could have a tablet, or a phone or even a couple of netbooks all with power that would completely dwarf the humble spectrum. As computers have become more affordable, they have proliferated.

And not just consumer devices. It now makes perfect sense to use computational devices in almost every setting thanks to the cheapness and the ingenuity of the Chinese. IBM told me at impact that there are 9.5Bn connected devices in the world today growing to 20Bn in 2015. All of these devices are producing data at an alarming rate. Storage companies fall over themselves to tell you how fast this data is growing, but take it from me – it’s exponential.

The other side effect of computers is that the average human being now has the attention span of a hyperactive goldfish. Because the answer is available from Google in 0.2 nanoseconds, why would you want to read a report or a book? Twitter delivers a constant stream of news in 140 character long chunks. This means that we are no longer happy to do lots of analysis to get our answers, we want to know instantly.

So how do we make sense of this muddle?

I have always been a fan of infographics as a mechanism for presenting complex information. Ingeniously through pictures, infographics helps the reader to understand the big picture. Spatial and relative relationships are easily picked out, but detail is not. Newspapers use info graphics to great effect – especially when trying to explain concepts like exactly how much money we all owe thanks to the financial meltdown. This is a great site to see some really good examples; http://www.coolinfographics.com/

Big Data was a much vaunted concept at IBM Impact too. If you want to process the huge amount of data available today, then traditional methods are just not going to work. Reading all the data into a relational database and then running a query will mean you need a huge building to house the storage, a power station to feed the computers and half a century to wait for the result. The only way to process this data is as a stream – a bit like the water coming out of a firehose. Then you need to use something called complex events processing to look for relationships between the data.

One of the most interesting concepts in dealing with the information overload is the semantic web. The idea is that computers need to make sense of all this data themselves and answer any questions we have. This is the technology behind the Siri voice response software in the latest iPhone. There is even a search engine. It’s not perfect by any means – try searching for “Who is the richest man in France” or my personal favourite “Who is the sexiest woman” 🙂 But what is interesting is how often it does get it right.

One day – the characters in Star Trek the next generation talking to their computers will look positively antiquated – and I for one can’t wait.

Related articles