Image Credit
Not sure if you’ve heard about this one, but on Thursday, August 22, 2013, the American Nasdaq stock exchange was shut down for three hours because of a communication problem. Investigators are still trying to understand exactly what happened; however, we should probably asking ourselves if this problem was caused by the Nasdaq’s CIO?
The Scene Of The Crime
So just exactly what happened over at the Nasdaq stock exchange? The Nasdaq is an American stock exchange. Its name stands for “National Association of Securities Dealers Automated Quotations”. On Thursday, August 22, 2013 at 10:57am they started to have problems with their automated systems that share stock trading information with other stock exchanges.
A “momentary interruptions in quote dissemination” problem hit a system that pools data from exchanges that trade Nasdaq securities. Those problems lasted for six minutes before the system returned to normal operation, according to a market status message issued by the exchange. Ok, we can all understand that right? Every IT system can have a momentary gitch.
However, roughly an hour later, Nasdaq said there were problems with submission of quotes to the same system. As a result, Nasdaq halted trading at 12:14 p.m. EDT and the options market followed suit at 12:20 p.m. EDT. Now this is a big deal – a complete stock market shut down. All of those high frequency traders and algorithms that are constantly buying and selling stocks to make lots of slim profits were suddenly left holding the bag.
Who Is Responsible?
Nasdaq trading was halted at 12:15 p.m. because of an apparent problem with Nasdaq’s Securities Industry Processor, or SIP, which aggregates pre-and- post trading information on Nasdaq-listed securities from a variety of different exchanges. Ultimately, the SIP server was responsible for the outage. However, who’s responsible for SIP?
Brad Peterson is the Executive Vice President and Chief Information Officer (CIO) for Nasdaq. Ultimately he’s responsible for everything that happens in his IT department – including the correct operation of the SIP servers. At the end of the day, this outage is Brad’s fault – he understands the importance of information technology and this kind of thing should never happen on his watch.
What went wrong? The current thinking is that the Nasdaq IT department has been spending so much time worrying about how to speed trading information up to meet the needs of algorithms and high speed traders that they have not been doing a proper IT system risk analysis. What they’ve failed to do is to take a look at all of the interfaces that the Nasdaq IT shop has with the outside world and prioritize them based on how expensive an outage on any one of them would be. If they had done this, then they would have realized that the SIP system needed more of a backup system to support it – just in case!
What Does This Mean For You?
As the person in the CIO position you are responsible for everything that goes on in your IT department. If you are the CIO of the Nasdaq stock exchange this means that you are responsible for making sure that your IT systems can talk with everyone else. If they can’t, then that’s your fault.
The Nasdaq stock exchange experienced an outage and more and more it’s starting to look like the Nasdaq CIO is at least partially responsible for creating the IT environment that allowed this event to occur. Keeping a good inventory of your IT systems and their interfaces and then taking the time to rank them by impact of an outage allows a CIO to determine where they should be spending their time. Clearly this didn’t happen at the Nasdaq.
As the person with the CIO job, you need to always be on the lookout for the weakest link in your IT department. You need to be aware where an outage could cause the biggest impact to all of the systems that interface with your IT department. Nasdaq’s CIO, Brad Peterson, clearly was spending his time solving other problems and allowed this outage to sneak up on him. Hopefully, the problem has now been revealed and he’ll spend more time making sure that it does not happen again.
– Dr. Jim Anderson
Blue Elephant Consulting –
Your Source For Real World IT Department Leadership Skills™
Question For You: Do you think that there were warning signs that should have alerted Nasdaq to their eventual problem?
Click here to get automatic updates when The Accidental Successful CIO Blog is updated.
P.S.: Free subscriptions to The Accidental Successful CIO Newsletter are now available. Learn what you need to know to do the job. Subscribe now: Click Here!