March 30, 2001 Cmpt System Error System Error: The ValuJet Disaster On May 11, 1996, Flight 592, a DC-9 owned by ValuJet, took off from Miami destined for Atlanta with 110 on board 10 minutes after take-off it crashed into a swamp in the Everglades, killing everybody this is a classic example of a crash caused by system error we will analyze this and try to draw lessons for information systems in safety critical applications great Atlantic Monthly article: William Langewiesche, “The Lessons of ValuJet 592”, 281, 3, March 1998, pp ; also available at also read Charles Perrow, “Normal Accidents: Living With High-Risk Technologies”, 1984 Scott Sagan, “The Limits of Safety: Organizations, Accidents, and Nuclear Weapons”, 1993

March 30, 2001 Cmpt System Error Chronology of Events ValuJet contracted out a maintenance job on 3 of its MD-80’s to SabreTech, a maintenance firm based at Miami airport one of the jobs was to replace the chemical oxygen generators on board, at the end of their licensed lifetime (when lanyard is pulled two chemicals combine, producing heat and oxygen channeled to a mask so passengers can breathe during depressurization) they were stacked in unmarked cardboard boxes with their lanyards cutoff, but without required safety caps over their firing pins (the caps were not available) nevertheless, SabreTech mechanics signed paperwork certifying that they had capped the generators (among many other things) supervisors also signed off on the work many weeks passed, and eventually in early May a SabreTech manager ordered a shipping clerk to clean up the area in preparation for inspection by Continental Airlines, a potential customer for SabreTech

March 30, 2001 Cmpt System Error Chronology (continued) the shipping clerk re-distributed the cannisters, added bubble wrap, sealed the boxes, addressed them to ValuJet headquarters in Atlanta, and labelled them “aircraft parts”; and then added 3 tires to the cargo to be shipped next day he asked a co-worker to add the notation “Oxy Canisters” and “Empty” to the boxes on May 11 the SabreTech driver delivered the cargo to the ValuJet area where the ramp agent accepted it for shipment on Flight 592 even though shipping such material was forbidden the ramp agent and 592’s co-pilot discussed weight distribution and where to put the cargo, and eventually put it in the forward cargo hold during or shortly after take-off one of the canisters ignited a few minutes later fire engulfed the cargo hold and the cabin, smoke filled the cockpit, and the plane spiralled into the swamp, killing everybody

March 30, 2001 Cmpt System Error Factors in the Accident de-regulation in the industry generated intense price competition, rapid growth, lots of new employees, much contracting out, low salaries FAA inspection regime couldn’t keep up - only 3 inspectors assigned to ValuJet, who nevertheless did identify serious problems caused by too rapid growth: a more serious inspection was underway at the time of the accident, but too late SabreTech mechanics were ignorant of the functioning of the oxygen generators, and didn’t realize how serious the shipping caps were to safety: the manuals were written in “engineerspeak” and anyway, the mechanics were highly stressed with other duties and not focussed on what happened after they removed the generators the caps were not available, and the mechanics signed off anyway, both examples of a collective relaxation called by sociologist Diane Vaughan “the normalization of deviance”: the real Murphy’s Law is “what can go wrong usually goes right”

March 30, 2001 Cmpt System Error Factors (continued) there were disconnects among the various people: mechanics never thought the canisters would be shipped, co-pilot and ramp agent did not realize they were not capped the shipping clerk was chosen to “clean up” the area, and he naturally thought in terms of shipping the cargo rather than other alternatives communication problems: confusion by shipping clerk who thought that being out of service meant that the canisters were “Empty”; mechanics who did not realize that “expired” canisters were not “expended”, so warnings in the manuals about unexpended canisters were meaningless multitudinous procedure manuals, rules, and regulations for all concerned: hard to read, impossible to remember, often don’t match reality of day to day job pressures and necessities, breed resentment and rebellion by employees

March 30, 2001 Cmpt System Error Analysis of Errors three kinds of errors –procedural error (eg. pilot error) –engineered error (eg. mechanical failure) –system error (as in ValuJet) system errors “control and operation of some of the riskiest technologies require organizations so complex that serious failures are virtually guaranteed to occur” (paraphrased from Perrow, 1984) “safety ultimately involves a blizzard of small judgments” –interactive complexity “many elements … linked in multiple and often unpredictable ways … cascading failures can accelerate out of control” –tight coupling “lack of slack” many examples: Chernobyl, Three Mile Island, close calls on SAC false positives, Cuban missile crisis, many disasters

March 30, 2001 Cmpt System Error Lessons for Information Technology big software is highly complex, a definitional example of interactive complexity and tight coupling IT is embedded in bigger social or technical systems which are themselves complex there are even more complex interactions between the IT and the system in which it is embedded fallible humans are “in the loop” everywhere: design, implementation, testing, and after deployment IT is brittle, but so fast that humans cannot easily intervene when it goes wrong IT-based systems are constantly evolving and changing, adding still further to the complexity and diminishing the ability of humans to understand and monitor it perhaps error is inevitable, regardless of controls and regimes attempting to eliminate it: sometimes the control regimes themselves cause the errors