The job of a CIO is to make sure that everything works “just right”. If we have a nightmare it’s that the systems that we are responsible for suddenly just stop working just when everyone realizes the importance of information technology. As they sit around waiting for us to fix things, we frantically run around trying to figure out what went wrong. That nightmare become a reality for CIOs recently when the pricy Bloomberg terminals that traders use to conduct business experienced an extended outage.
What Happened?
In the world of financial trading, information is money. The way that traders get access to the information that they need here in the 21st Century is by subscribing to an information service that is offered by Bloomberg LP. For only US$20,000 per year, you can have access to what is called a “Bloomberg Terminal”. These terminals are used by traders to identify current market conditions and then to execute trades using that information.
It turns out that this outage was an even bigger deal than most CIOs would have initially realized. The reason is because over time the traders have become even more dependent on their Bloomberg terminals. In order to execute trades, the traders first have to talk with other traders. They accomplish this by using an electronic chat system that is built into the Bloomberg terminal. When the blackout occurred, this tool also became unavailable.
Just to make things even worse, when the Bloomberg terminals were not available, the traders struggled to use phones and other means to contact other traders. The reason that this was so hard to do was because they didn’t know how to get in touch with them. The reason was because each trader’s electronic “contact list” of who they interacted with along with their phone number and email were stored on their Bloomberg terminal which was no longer available.
Why Did It Happen?
What happened was that the Bloomberg network was hit by a massive computer-network outage. The result of this outage was that many of its terminals were placed out of action for hours. Clearly this caused a huge disruption for the traders who rely heavily on these terminals.
Bloomberg reports that the cause of the outage was a complex set of circumstances. They say that a combination of hardware and software failures within their network occurred. The result of this was that it allowed an excessive increase in the amount of network traffic that the network had to carry. The result of this increase in volume was that customers’ terminals were disconnected because their terminals became overwhelmed with network traffic. Bloomberg claims that they were able to quickly isolate the faulty hardware, they restarted their software, and they are now taking steps in order to determine why their multiple redundant systems failed to prevent this situation from occurring.
From a CIO’s point-of-view, we realize that outages like this can both happen and will happen in the future. What we need to do is to take steps in order to deal with the situation that we will find ourselves in when it does occur. In the case of firms that use Bloomberg terminals this would mean keeping backup copies of trader contact information, establishing non-Bloomberg terminal methods of contacting other traders and perhaps even having an occasional “Bloomberg terminal down” training day.
What All Of This Means For You
As most people who have the CIO job know, you never know how much you rely on a system until it becomes unavailable. This was the case recently when the Bloomberg terminals that financial traders use became unavailable. This is exactly the type of situation that the person in the CIO position tries desperately to avoid.
The Bloomberg network experienced a hardware failure that impacted its software systems. Despite having multiple layers of redundancy, they experienced traffic increases that caused many of their customers to not to be able to log onto their terminals. The Bloomberg system also permits traders to chat with each other and stores contact information, all of which become unavailable during this outage.
As CIOs we fully understand that we can’t control everything. However, we also know that bad things do happen and this means that we need to take steps to prepare for them when they do occur. What this means for CIOs whose firms use Bloomberg terminals is that they need to understand how important these devices are and they need to take steps to allow the company to continue to operate even if the terminals go down in the future.
– Dr. Jim Anderson
Blue Elephant Consulting –
Your Source For Real World IT Department Leadership Skills™
Question For You: How quickly do you think that an IT department would have to react to having the company’s Bloomberg terminals become unavailable?
Click here to get automatic updates when The Accidental Successful CIO Blog is updated.
P.S.: Free subscriptions to The Accidental Successful CIO Newsletter are now available. Learn what you need to know to do the job. Subscribe now: Click Here!
What We’ll Be Talking About Next Time
In the U.S., the government is investing a great deal of money into transforming the country’s medical system. A key part of this transformation is the conversion of medical records from paper to electronic records. On paper, this sounds like a great idea. Once CIOs have overseen the digitization of medical records, they can be easily exchanged between care providers. However, the reality of this transformation is turning out to be a bit harder than anyone thought it would be…