|
CAS - complex adaptive system
p.1 Evaluation is a central issue in all organizations... New strategies are required to evaluate complex adaptive human systems. New tools, techniques and methods must integrate assumptions about the dynamical and complex nature of human systems.
p.2 A CAS is defined in terms of its parts, the behavior of those parts and the emergent behavior of the whole.
p.3 A CAS exists in a state of flux... These complex interactions generate a system that is roiling with change. At no point does the system come to a natural equilibrium or stopping point... You can imagine such action to be permanent whitewater, a sand pile, shifting sands, unshackled action... All of these images connote the ever-changing nature of a CAS.
This change does not always follow a smooth, predictable pattern. Change... may bring surprising outcomes... For this reason, the evaluator cannot expect a smooth, linear path between project start and project end... Such unpredictable patterns cannot be assessed by means of periodic sampling or end-point evaluation only... The evaluator cannot realistically consider an organization or a program to be moving in a predictable way toward a pre-determined end point... the whole concept of projected and predictable outcomes is an artificial construct when evaluating performance in a CAS. An evaluator may be able to frame expectations, but the self-organizing nature of the system may result in completely different outcomes than those expected... Evaluators and evaluation plans must adjust to the perpetual but unpredictable dynamic behavior of a CAS. The changing patterns within the system must be captured and described, without depending on natural end points of behavior or extrapolation or interpolation from timed samples.
p.3-4 Because a CAS is dynamic, evaluation systems should incorporate flexible and dynamic features. Specifically, they should:
- Capture an emerging model of causal relationships... Because the patterns of causation are one of the things that change in a CAS, it is critical to capture baseline representation of those causal relationships, but it is necessary to revise that image frequently. Evolution of the causal model over time can provide a powerful and simple description of the systemic aspects of change.
- Evaluate and revise the evaluation often. Because the CAS baseline is constantly shifting, the evaluation plan should include options for frequent and iterative reconsideration and redesign. Data about the redesign of the evaluation program can also become a rich source of information about the developing patterns of the system.
- Capture, preserve and learn from the "noise" in the system... In a CAS, much of the meaningful information about system future, patterns and dynamics come... from the unexpected system behaviors. For this reason, evaluation should capture the unexpected as well as the expected, the long- and short-term outcomes and the close and distant points of view. Only from this diverse data can an evaluation emerge that is sophisticated enough to reflect the complexity of the system being evaluated.
p.5 A causal diagram provides many different benefits to an evaluation program.
p.5 Relationships in CASs are complicated and enmeshed... look for those "differences that make a difference" in the system
p.5 In addition to being intractable (if not infinite) in number, the dimensions that drive behavior of a CAS have nonlinear relationships with each other. A small change in one variable may generate exponential change in another. This pattern is exacerbated because CASs depend on iterative processes... The output of a previous process becomes the input for the next one. Iteration magnifies the effects of nonlinearity, so that simple causal relationships are virtually impossible to detect, to measure, control or evaluate.
p.8 Iterative redesign generates an evaluation program that reflects the massive entanglements of the system... it is time consuming
p.9 The complex outcome behaviors of a CAS may be the result of the iterative application of a "short list of simple rules."
p.10 Micro-design and evaluation acknowledge that the system is changing too quickly to support large-scale planning or assessment.
p.10 Evaluate to inform action... Focus on "differences that make a difference."
p.12 Make evaluation a part of the intervention.
p.13 CASs exhibit emergent, of self-organizing, behavior. New patterns are generated by the interaction of the agents. New structures are established, and old ones disappear. These structural changes are not designed and imposed by some force outside of the system. They self-organize as the internal dynamics of the system play out over time.
p.13 It is feasible to define an outcome in a CAS when it is recognized as a possible scenario result rather than a predicted outcome.
p.14 Sensitive dependence is one aspect of emergence in CASs... A second aspect of emergence deals with patterns that appear over time in the behavior of CASs. These patterns, called "attractors," provide some insight into the emerging relationships in a CAS.
p.14 an attractor is the pattern that forms as the individuals in the system interact. The individual behaviors form the pattern, and then other individuals are constrained to perform within the pattern. In this way, an attractor is emergent and self-reinforcing... the activities of the agents in a complex system are patterned, though they are not predictable... The attractor is the primary method of "seeing" system-wide changes in behavior over time... Because the emerging attractor is the most trustworthy picture of system-wide behavior, evaluation methods should be designed to ensure that they capture and analyze the data that would reveal such patterns.
p.15 The emergent nature of a CAS unfolds over time, so the only way to observe this emergence is through the use of time series analysis.
p.15 In spite of its drawbacks, time series analysis and reconstruction of attractors promises to give real insight into the complex, entangled dynamics of evaluation of human systems. All of these tools and techniques (causal diagrams, iterative redesign, shorts and simples, feedback analysis and time series analysis) provide ways for the evaluator to capture and interpret information about the performance of a human CAS.
p.16 As CASs, human systems are dynamic, entangled, scale independent, transformative and emergent. These characteristics... necessitate new evaluation approaches that are as rich and varied as the human systems they are designed to assess... By stating... the dynamic patterns in the environment, the group can begin to build mechanisms to cope in the future... By framing an evaluation method as experimentation and learning, the evaluator can encourage individuals and groups to value their mistakes and to learn from them.
p.16 Effective adaptation is the best indicator of success in a CAS... the evaluator is less an instrument of assessment in the organization and more an instrument of transformative change.
p.17 As human systems move toward complex adaptive behavior... the assumptions that are the foundation for evaluation are no longer valid... A CAS perspective on evaluation opens the door to approaches that truly reflect the complexity and adaptation of the human systems they represent.
p.18 A CAS model of evaluation is most useful when complexity renders other methods of evaluation ineffective; when evaluation will be used to challenge existing assumptions of linear causality; or when the interventions to be evaluated are designed to reflect the complex adaptive nature of the system.
|
