|
p.17 Situational responsiveness is the challenge to which creative, yet practical, evaluation thinking is the response.
p.17 In Creative Evaluation I reviewed the research findings from studies of human heuristics that support this claim that most of the time evaluators, and the decision makers with whom they work, are running - and thinking - according to preprogrammed tapes. It turns out that most of the time, in order to make even trivial decisions, we rely on routine heuristics, rules of thumb, standard operating procedures, standard tricks of the trade, and scientific paradigms.
p.17-18 A major difficulty posed by the human reliance on paradigms and heuristics for problem solving and decision making is not just their existence, but our general lack of awareness of their existence... while it is neither easy nor usual, it is possible to become aware of our paradigms and heuristics, and in that awareness take control of our decision processes, thereby releasing our creative potential and enhancing our ability to be truly situationally responsive and adaptive.
p.19
- situational responsiveness is imperative for effective... evaluation practice;
- realizing the potential for creative evaluation requires, first, awareness of routine response patterns and blinders that limit situational insight, then a willingness to consciously move beyond limitations to risk new ways of doing things;
p.20 the prerequisite for situational responsiveness is a firm grounding in fundamentals. The primary purpose of this book is to provide grounding in evaluation fundamentals
p.21
People evaluate. Many times they do it without a fund of knowledge about the systematic evaluation process... [Gephart, 1981: 257].
p.27 evaluators must be well schooled in fundamentals and able to adapt those fundamentals to specific situations. The new standards of evaluation - utility, feasibility, propriety, and accuracy - require situational responsiveness rather than implementation of standardized procedures. Knowledge and skill in applying evaluation fundamentals under these conditions requires ongoing analysis of specific situations and calculated judgments about what is possible, what is probable, and what just might be useful, given what is known...
p.37 The mandate to be situationally responsive includes having sufficient flexibility to understand which definition(s) of evaluation are appropriate and meaningful in a particular context
p.51 Knowledge of the many different types of evaluation increases the options available for situational responsiveness.
p.71 A group larger than ten begins to become quite cumbersome for decision making.
p.76 Experimental evidence suggests that the greater the need for a creative solution to a particular evaluation problem, the less able a group is to generate acceptable solutions.
p.77
The term "brain storm" was coined in 1906 by Harry Thaw's lawyer. Clouds of witnesses had seen his client pumping bullets into Sanford White, so it seemed that the defendant would need something more than an alibi... Accordingly, the cunning advocate convinced the jury that Thaw suffered, that very evening, a violent fit of passing madness, a brain storm, which momentarily excused him from certain moral imperatives. Thus, Thaw escaped the electric chair
p.87 Situational responsiveness cannot be optimized if one approaches an evaluation problem with a single, content-laden model in mind.
p.88 If local decision makers and information users want to monitor changes in attitudes or behaviors... in a system, it is not necessary to presuppose that the collection of such information necessitates a causal framework. What decision makers and information users are looking for may well be information to guide them in making immediate and concrete decisions. Under such conditions they are concerned more with the meaningfulness of particular information about what is happening than they are with comprehensive, causal questions.
p.101 Goal-free evaluation (Scriven, 1972, 1979) and now "goal-free planning" (Clark, 1979) are strategies for situational responsiveness that avoid the routine heuristic of assuming that every evaluation and/or planning situation automatically requires the delineation of clear, specific, and measurable goals. Nevertheless, for... management reasons, goals specification remains the most common way to focus evaluation and planning processes... Knowledge of alternatives to goals-based evaluation is also an important part of an evaluator's repertoire of fundamentals.
p.104 For may part, I prefer to have soft or rough measures of important goals rather than have highly precise and quantitative measures of goals that no one really cares about... when I work with groups on goal clarification I have them write their statement of goals and objectives without regard to measurement. Once they have stated as carefully and as explicitly as they can what they want to accomplish, then it is time to figure out what indicators and data can be collected to monitor relative attainment of goals... by separating the process of goals clarification from the process of goals measurement, it is possible for program staff to focus first on what they are really trying to accomplish and to state their goals and objectives as explicitly as possible without regard to measurement, and then worry about how one would measure actual attainment of those goals and objectives.
p.112 New conceptualizations can be helpful in opening our minds to potentially new ways of perceiving and experiencing the world. The eolithic alternative... is meant to serve this thought-provoking, awareness-enhancing function. The notion of eolithism is meant to alert us to the limitations of goals-based evaluation designs, while making us aware of, not simply an alternative technique, but a totally different way of proceeding and perceiving.
p.113 The eolithic alternative was introduced into evaluation by David Hawkins... Hawkins draws on the work of American engineer/novelist Hans Otto Storm to differentiate the principle of eolithism from the principle of design... The principle of eolithism... directs the investigator to consider how ends can flow from means. One begins by seeing what exists in the natural setting and then attains whatever outcomes one can with the resources at hand. Storm adopted the term "eolithism" for this approach in order to focus our attention on the eoliths that are available all around us... An eolith is "literally a piece of junk remaining from the stone age, often enough rescued from some ancient burial heap... Stones, picked up and used by man, and even fashioned a little for his use" (Hawkins, 1976: 91)... the principle of eolithism calls to mind a child (or stone-age human) happening upon some object of interest and pondering, "Now what could this be used for?"
p.114 Eolithic programs are those programs whose participants are guided by the principle of eolithism; i.e., they look around them to see what's available and then do whatever they can with whatever they find. What they do moves them toward emerging goals that are discovered in and grow out of the environment in which they find themselves, or are inherent in the materials available to them.
p.114 evaluators need alternatives to goals-based approaches if they are to be situationally responsive.
p.117 The challenge of eolithism is to understand the uses and functions of what may appear, on first glance, to be junk
p.123 The eolithic evaluation question was: In what ways, to what ends, and for what purposes could these things be put together?
p.137 Though analogy is often misleading, it is the least misleading thing we have. Samuel Butler
p.139 Judge a man by his questions rather than by his answers. Voltaire
p.140 One doesn't need a doctorate to write questionnaires. What one does need is some clear thinking about the kind of information that is needed and an understanding of what kinds of questions to ask to get the needed information. I would hazard a guess that questionnaires are probably employed more often than any other technique in evaluation.
p.141 The process of figuring out what should be asked on a questionnaire is a process that lays the foundation for later analysis and utilization of findings... Understanding is enhanced by direct involvement in the painstaking process of making decisions about what data to collect and how to collect them.
p.210 my experience suggests that finding solutions to enormously complex problems with limited resources is the epitome of the evaluation challenge, requiring hard work, flexibility, and all the pains and insights of creative problem solving grounded in evaluation fundamentals.
p.224
Evaluations should have clear closure points; they should not go on eternally.
p.229 To use those data for evaluation purposes one has to ask questions of the data. The data have to be processed and organized, arranged and analyzed, studied and interpreted to be useful for evaluation purposes... As I talk with people who are frustrated with management information systems I note that commonly they have approached the data without any questions and yet they complain that the data don't tell them anything. If there is nothing you are trying to find out, there is nothing you will find out... The management information system, then, is not an endpoint but a beginning point in raising significant issues for additional, more in-depth study.
p.230-231 Another technique that can help make management information systems more useful is the formulation of decision rules for acting on the data. A decision rule is part of a framework that can be applied to routine data to indicate when some significant threshold or action point has been reached. The decision rule is like the red light that flashes on a boiler when too much pressure has built up. It tells the machine operator that it is time to relieve the pressure, that some action is indicated. Routine monitoring systems for all kinds of machinery and chemical systems specify acceptable ranges of activity as well as danger points that indicate when the system needs special attention. Decision rules for management information systems serve this same purpose.
p.231 When one has decision rules to use to examine the data the examination process takes on considerably more meaning and interest than it does when one simply looks at the data month after month without having a firm framework for interpreting trends.
p.232 at a minimum, when the data indicate some threshold has been reached on the basis of carefully predetermined criteria, then it is at least necessary... to consider action alternatives and decide what kind of action, if any, ought to be taken.
p.233 My experience suggests that if people in a program are unable to specify in advance what constitutes a decision threshold, they will probably be unable to recognize such a threshold if it actually occurs – despite careful attention to the data... The practical problem here is that without a decision rule it is easy to postpone action for months and months while waiting to see if the trend continues and then planning to act “next” time. There are always lots of reasons for postponing decisions at a particular moment in time. The function of setting decision rules in advance is to increase the pressure to take action when it logically and rationally should be taken rather than being seduced by endless postponement until a full crisis has emerged. Moreover, it is important to keep in mind that there are different kinds of decision rules. Some decision rules are of the order: “We are approaching a critical stage and we need additional information to inform action.” Other decision rules are of the order: “We are at a critical point; action must be taken now!”
p.301 the conceptualization and implementation of practical evaluations challenge the evaluator to produce useful information under typically sever constraints of limited time, limited finances, limited knowledge, and seemingly unlimited politics. The elegance of practical solutions often disguises the creative work, skilled diplomacy, and disciplined talent contained therein.
|
