It’s been a rough week for some IT guys. United Airlines, the New York Stock Exchange, and TD Ameritrade all had very public network failures. So far, no one has claimed that anything nefarious was going on. That is, these shortcomings were not the result of any cyber attacks or hacking attempts. Rather, they appear to have been the result of failure in standard operating procedures.
How can such a thing happen?
To some degree, such failures are not so much a question of if, but of when. No system is perfect and problems of one sort or another are inevitable. The question is what is an acceptable rate of failure. That is, if the NYSE cannot be guaranteed to be always working perfectly, what is an acceptable rate at which to have disruptions?
The Wall Street Journal had an article that somewhat relates to this point (What We Learned From the NYSE, United Airlines Tech Outages, Jul 9). The article notes that old-fashion land line telephones had an uptime of 99.999% — that is, Ma Bell would leave you without a working phone only about five minutes out of every year. Of course, old school phone systems were regulated and nudge to that level of reliability by their overseers. Unregulated, private networks aren’t held quite to the same standard. The article claims that firms are generally unwilling to invest to the level that would raise their reliability to the level of a land line. It also notes that there are other things going on.
Today’s problems with reliability are more fundamental, a reflection of the complexity of contemporary networks, the volume of data, the pace of change, insufficient organizational and cultural practices, and a legacy of arcane and poorly written business software that traditionally put little emphasis on usability or customer experience.
Outages persist because of the interdependency of computer systems, fueled by the rise of digital services across all industries, particularly those with customer-facing software such as mobile apps, according to former NYSE Euronext CIO Paul Cassell, now CIO of Pico Quantitative Trading LLC.