vii I started this book, without knowing I was starting a book, in August of 1979.

p.9 as Robert Jervis and Karl Weick have noted, seeing is not necessarily believing; sometimes, we must believe before we can see.

p.21 one of the lessons of complex systems and TMI is that any part of the system might be interacting with other parts in unanticipated ways.

p.22 Unknown to them, there was an intimate connection because of the interactive complexity of the system.

p.70 System accidents involve the unanticipated interaction of multiple failures.

p.71 It is not the source of the accident that distinguishes the two types [JLJ - component failure accidents and system accidents], since both start with component failures; it is the presence or not of multiple failures that interact in unanticipated ways.

p.72-73 Interactiveness per se... is not a useful concept. Almost any organization of any size... will have many parts that interact once we look closely at them... What if parts, or units, or subsystems (that is, components) serve multiple functions? ...The heater has what engineers call a "common-mode" function - it services two other components, and if it fails, both of those "modes" (heating the tank, cooling the reactor) fail. This begins to get more complex.

p.73 The main problem is complexity itself

[JLJ - When you choose the behavior you choose the consequences of the behavior. Designers chose a complex design, they need to accept the consequences, including the unexpected.]

p.75 complex interaction... there are branching paths, feedback loops, jumps from one linear sequence to another because of proximity and certain other features... The connections are not only adjacent, serial ones, but can multiply as other parts or units or subsystems are reached.
 ...linear interactions... even the most complex systems of any size will be primarily made up of linear, planned, visible interactions.

p.78 Complex interactions are those of unfamiliar sequences, or unplanned and unexpected sequences, and either not visible or not immediately comprehensible.

p.79 In systems with a fair amount of complex interactions, however, well-defined and segregated segments do not necessarily exist. Instead, jiggling unit D may well affect not only the next unit, E, but A and H also.

p.83 Much more is simply invisible to the controller... Complex systems tend to have elaborate control centers... because components must interact in more than linear, sequential ways, and therefore may interact in unexpected ways.
 In addition to there being many interactions to control, the information about the state of components or processes is more indirect and inferential in complex systems.

p.83 The reactor core at Three Mile Island had no direct measure of coolant level... operators had to estimate coolant level from indirect indicators.

p.85 Transformation processes... These are processes that can be described, but not really understood. They were often discovered through trial and error, and what passes for understanding is really only a description of something that works.

p.87 In complex systems, not only are unanticipated interdependencies more likely to emerge because of a failure of a part or a unit, but those operating the system (or managing it) are less likely, because of specialized roles and knowledge, to predict, note, or be able to diagnose the interdependency before the incident escalates into an accident.

p.92 Loosely coupled systems, whether for good or ill, can incorporate shocks and failures and pressures for change without destabilization. Tightly coupled systems will respond more quickly to these perturbations, but the response may be disastrous.

p.93 In loosely coupled systems, delays are possible; processes can remain in a standby mode; partially finished products... will not change much while waiting.

p.94-95 In tightly coupled systems the buffers and redundancies and substitutions must be designed in; they must be thought of in advance. In loosely coupled systems there is a better chance that expedient, spur-of-the-moment buffers and redundancies and substitutions can be found, even though they were not planned ahead of time... In loosely coupled systems... Failures can be patched more easily... Tightly coupled systems offer few such opportunities.

p.95 In two of the most famous nuclear plant accidents, Browns Ferry and TMI, imaginative jury-rigging was possible and operators were able to save the systems through fortuitous means.

p.95-96 At TMI, two pumps were put into service to keep the coolant circulating, even though neither was designed for core cooling... What is true for buffers and redundancies is also true for substitutions of equipment, processes, and personnel. Tightly coupled systems offer few occasions for such fortuitous substitutions; loosely coupled ones offer many.

p.98 universities and post offices are quite loosely coupled. If something goes wrong in either of these, there is plenty of time for recovery, nor do things have to be in a precise order.

p.99 This is an example of complex interactions; in contrast to the linear post office, the university is high on complexity.
  But note that we are not likely to have a "system accident." Because universities are loosely coupled there is ample slack to limit the impact of this one personnel decision on other areas.

p.160 Tight coupling reduces the ability to recover from small failures before they expand into large ones. Loose coupling allows recovery.

p.161 Some other aspects of tight coupling probably cannot be reduced with either technological fixes or organizational changes. There are few opportunities for nondeliberate, fortuitous buffers that foster recovery from a dangerous situation, nor significant substitutions of supplies, equipment, or personnel.

p.175 Some technological fixes on individual ships had the unanticipated consequence of changing a loosely coupled set of ship interactions to a tightly coupled one, making recovery more difficult when failures occurred.

p.245 The argument for loose coupling is that when failures occur there is generally room for recovery because affected areas can be segregated; alternative sequences can be used for a time; some slack resources exist, and indigenous substitutions can be made in many cases.

p.316 Hunches and rules of thumb and rough estimates and guesses appear to be patterned and widespread. Cognitive psychologists call these guesses "heuristics," from the word for discovery

p.317 It is universally granted that heuristics are useful, time-saving devices, even if they sometimes or even often get us into trouble... First, heuristics prevent a paralysis of decision making; they prevent agonizing over every possible contingency that might occur... Second, they drastically cut down on... the time and effort to examine all possible choices and then try to rank them precisely in terms of their costs and benefits... Third, they undergo revision, perhaps slowly, as repeated trials led to corrections of hunches and rules of thumb, and do so without expensive conscious effort. Finally, I think, they facilitate social life by giving others a good estimate of what we are likely to do, since we appear to share these heuristics widely.

p.317 Heuristics appear to work because our world is really quite loosely coupled, and has a lot of slack and buffers in it that allow for approximations rather than complete accuracy.

p.319 heuristics are akin to intuitions. Indeed, they might be considered to be regularized, checked-out intuitions. An intuition is a reason, hidden from our consciousness, for certain apparently unrelated things to be connected in a causal way.

p.322 A working definition of an expert is a person who can solve a problem faster or better than others, but who runs a higher risk than others of posing the wrong problem. By virtue of his or her expert methods, the problem is redefined to suit the methods.

p.330 If the complex interactions defeat designed-in safety devices or go around them, there will be failures that are unexpected and incomprehensible.

p.331 Systems with interactive complexity... will produce unexpected interactions among multiple failures.

p.332 Accidents will be avoided if the system is also loosely coupled... because loose coupling gives time, resources, and alternate paths to cope with the disturbance and limits its impact... Unexpected and incomprehensible interactions will not allow immediate analysis of the cause of the accident, but given the slack in loosely coupled systems, this is not essential. It is enough that personnel perceive an unwanted system state... and do so before it interacts with other units and subsystems.

p.372 the configuration of the system... can discourage, even in an error-inducing system, the small errors that make the unanticipated interaction of errors possible