|
xi I wrote this book to correct a costly myth that permeates many organizations today: that certain things can't be measured... Often, an important decision requires better knowledge of the alleged intangible but when an executive believes something to be immeasurable, attempts to measure it will not even be considered.
As a result, decisions are less informed than they could be... Any important decision maker could benefit from learning that anything they really need to know is measurable.
p.3 When you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind; it may be the beginning of knowledge, but you have scarcely in your thoughts advanced to the state of science. - Lord Kelvin
Anything can be measured. If a thing can be observed in any way at all, it lends itself to some type of measurement method. No matter how "fuzzy" the measurement is, it's still a measurement if it told you more than you knew before. And those very things most likely to be seen as immeasurable are, virtually always, solved by relatively simple measurement methods.
P.5-6 The focus here is on measurements that are relevant - even critical - to major organizational decisions and yet don't seem to lend themselves to an obvious and practical measurement solution... I have one recommendation for a useful exercise to try. As you read through the chapters, write down those things you believe are immeasurable or, at least, you are not sure how to measure. After reading this book, my goal is that you are able to identify methods for measuring each and every one of them. [JLJ - I am writing down "how to measure the promise or potential present in a chess position"]
p.8,9 Our first mentor of measurement did something that was probably thought by many in his day to be impossible. An ancient Greek named Eratosthenes (ca. 276-194 BC) made the first recorded measurement of the circumference of Earth... He observed that the shadows in Alexandria at noon at that time of year made an angle that was equal to an arc of one-fiftieth of a circle. Therefore, if the distance between Syene and Alexandria was one-fiftieth of an arc, the circumference of Earth must be 50 times that distance... Here is the lesson for business: Eratosthenes made what might seem an impossible measurement by making a clever calculation on some simple observations. When I ask participants in my measurement and risk analysis seminars how they would make this estimate, without modern tools, they usually identify one of the "hard ways" to do it (e.g., circumnavigation). But Eratosthenes, in fact, may not have even left the vicinity of the library to make this calculation... He wrung more information out of the few facts he could confirm instead of assuming the hard way was the only way.
p.9,10 Another example from outside business that might inspire measurements within business is Enrico Fermi (1901-1954), a physicist who won the Nobel Prize in physics in 1938. He had a well-developed knack for intuitive, even casual-sounding measurements. One renowned example of his measurement skills was demonstrated at the first detonation of the atomic bomb, the Trinity Test site, on July 16, 1945... While final adjustments were being made to instruments used to measure the yield of the blast, Fermi was making confetti out of a page of notebook paper. As the wind from the initial blast wave began to blow through the camp, he slowly dribbled the confetti into the air, observing how far back it was scattered by the blast... Fermi concluded that the yield must be greater than 10 kilotons... After much analysis of the instrument readings, the final yield estimate was determined to be 18.6 kilotons. Like Eratosthenes, Fermi was aware of a rule relating one simple observation - the scattering of confetti in the wind - to a quantity he wanted to measure.
p.10 The value of quick estimates was something Fermi was familiar with throughout his career. He was famous for teaching his students skills at approximation of fanciful-sounding quantities that, at first glance, they might presume they knew nothing about... Fermi was trying to teach his students how to solve problems where the ability to confirm the results would not be so easy. He wanted them to figure out that they knew something about the quantity in question.
p.11 The lesson for business is to avoid the quagmire that uncertainty is impenetrable and beyond analysis. Instead of being overwhelmed by the apparent uncertainty in such a problem, start to ask what things about it you do know. As we will ask later, assessing what you currently know about a quantity is a very important step for measurement of those things that do not seem as if you can measure them at all.
p.19 There are three reasons why people think that something can't be measured. Each of these three reasons is actually based on misconceptions about different aspects of measurement: concept, object, and method.
Concept of measurement. The definition of measurement itself is widely misunderstood. If one understands what it actually means, a lot more things become measurable.
Object of measurement. The thing being measured is not well defined. Sloppy and ambiguous language gets in the way of measurement.
Methods of measurement. Many procedures of empirical observation are not well known. If people were familiar with some of these basic methods, it would become apparent that many things thought to be immeasurable are not only measurable but may already have been measured.
p.21 A Definition of Measurement
Measurement: A set of observations that reduce uncertainty where the result is expressed as a quantity
p.22-23 [American electrical engineer and mathematician Claude] Shannon proposed a mathematical definition of information as the amount of uncertainty reduction in a signal... Major decisions made under a state of uncertainty... can be made better, even if just slightly, by reducing uncertainty... So a measurement doesn't have to eliminate uncertainty after all. A mere reduction in uncertainty counts as a measurement and possibly can be worth much more than the cost of the measurement.
p.25 If someone asks how to measure [a certain concept] I simply ask: "What do you mean, exactly?" It is interesting how often people further refine their use of the term in a way that almost answers the measurement question by itself.
p.26 Clarification Chain
1. If it matters at all, it is detectable/observable.
2. If it is detectable, it can be detected as an amount (or range of possible amounts).
3. If it can be detected as a range of possible amounts, it can be measured.
p.27 Identifying the object of measurement really is the beginning of almost any scientific inquiry, including the truly revolutionary ones... some things seem intangible only because the managers just haven't defined what they are talking about. Figure out what you mean and you are halfway to measuring it.
p.39 Even with all the different types of measurements there are to make, we can still construct a set of steps that apply to virtually any type of measurement...
1. What are you trying to measure? What is the real meaning of the alleged "intangible"?
2. Why do you care - what's the decision and where is the "threshold"?
3. How much do you know now - what ranges or probabilities represent your uncertainty about this?
4. What is the value of information? What are the consequences of being wrong and the chance of being wrong, and what, if any, measurement effort would be justified?
5. Within a cost justified by the information value, which observations would confirm or eliminate different possibilities? For each possible scenario, what is the simplest thing we should see if the scenario were true?
6. How do you conduct the measurement that accounts for various types of avoidable errors (again, where the cost is less than the value of the information)?
p.40 Humans possess a basic instinct to measure, yet this instinct is suppressed in an environment that emphasizes committees and consensus over making simple observations. It simply won't occur to many managers that an "intangible" can be measured with simple, cleverly designed observations... as we saw, "intangibles" are a myth. The measurement dilemma can be solved.
p.43 Before we measure we should ask five questions:
1. What is the decision this is supposed to support?
2. What really is the thing being measured?
3. Why does this thing matter to the decision being asked?
4. What do you know about it now?
5. What is the value to measuring it further?
p.45 in order to measure something, it helps to figure out exactly what we are talking about and why we care about it.
p.85-86 If we could measure the value of information itself, we could use that to determine the value of conducting measurements. If we did compute this value, we would probably choose to measure completely different things. We would probably spend more effort and money measuring things we never measured before, and we would probably ignore some things we routinely measured in the past... We can make better... decisions... when we can reduce uncertainty... Knowing the value of the measurement affects how we might measure something or even whether we need to measure it at all.
p.88 All measurements that have value result in the reduction of uncertainty of some quantity that affects some decision with economic consequences.
p.104 we are focusing on those things that are often considered to be immeasurable in business. Fortunately, the approach to addressing many of these issues does not involve the most sophisticated methods. It's worth restating the objective for this book is to show that many of the things a manager might consider immeasurable are actually measurable... A few relatively simple methods will suffice to measure most of these issues. The real obstacles to measurement, as we are discovering, are mostly conceptual, not the lack of understanding of dozens of much more complicated methods.
p.104,105 we will ask a few questions so that we might be able to determine the appropriate category of measurement methods. Those questions are:
- What are the parts of the thing we're uncertain about?...
- How has this (or its decomposed parts) been measured by others?...
- How do the "observables" identified lend themselves to measurement?...
- How much do we really need to measure it?...
- What are the sources of error?...
- What instrument do we select?
p.107,108 Part of the solution for this initial lack of imagination about measurement instruments may be to try to recapture the fascination Galileo and Fahrenheit had for observing the "secrets" of their environment. They didn't think of devices for measurement as complex contraptions to be used by esoteric specialists in arcane research. The devices were simple and obvious. Nor were they, like some managers today, dismissive of instruments because they had limitations and errors of their own. Of course they have errors. The question is "Compared to what?" Compared to the unaided human? Compared to no attempt at measurement at all? Keep the purpose of measurement in mind: uncertainty reduction, not necessarily uncertainty elimination.
Instruments generally have six advantages...
1. Instruments detect what you can't detect...
2. Instruments are more consistent...
3. Instruments can be calibrated to account for error...
4. Instruments deliberately don't see some things. Instruments are useful when they ignore factors that bias human observations...
5. Instruments record... [JLJ - produce a continuous stream of sensory information]
6. Instruments take a measurement faster and cheaper than a human.
p.110 Instead of being exasperated by a difficult estimate, [physicist Enrico] Fermi just started to break it down and answer each part. While his original question (the number of piano tuners in Chicago) seemed unknowable, it is just a function of other easier-to-estimate variables.
p.115 If you've identified your uncertainty, identified any relevant thresholds, and computed the value of information, then you've already identified something that is observable in principle. Consider these... questions about the nature of the observation...
1. Does it leave a trail of any kind? Just about every imaginable phenomenon leaves some evidence that it occurred. Think like a forensic investigator...
2. If the trail doesn't already exist, then can you observe it directly, or at least a sample of it? ...
3. If it doesn't appear to leave behind a detectable trail of any kind, and direct observation does not seem fathomable without some additional aid, can you devise a way to begin to track it now? ...
4. If tracking the existing conditions does not suffice (with either existing or newly collected data), can the phenomenon be "forced" to occur under conditions that allow easier observation (i.e., an experiment)?
p.124-126 Let's summarize how to identify the [measurement] instrument:
1. Decompose the measurement so that it can be estimated from other measurements...
2. Consider your findings from secondary research: Look at how others measured similar issues...
3. Place one or more of the elements from the decomposition in one or more of the methods of observation: trails left behind, direct observation, tracking with "tags," or experiments. Think of at least three ways you detect it, and then follow its trail forensically...
4. Keep the concept "just enough" squarely in mind. You don't need great precision if all you need is more certainty that a productivity improvement will be over the minimum threshold needed to justify a project...
5. Think about the errors specific to that problem...
consider these tips...
Work through the consequences. If the value you are seeking is surprisingly high, what should you see? If the value is surprisingly low, what should you see? ...
Imagine how others would do it...
Be iterative. Don't try to eliminate uncertainty in one giant study. Start making a few observations...
Consider multiple approaches...
What's the really simple question that makes the rest of the measurement moot? ... What is the basic question you need to ask to see if you need to measure any more?
Just do it.
p.128 We may only need a very small number of samples to draw useful conclusions about the rest of the unsampled population.
p.147 remember, usually you want to measure something because it supports some decision. And these decisions tend to have thresholds where, if a value is above it, one action is required, and another is required if the value is below it. But most statistical methods aren't about asking the most relevant question: "When is X enough to warrant a different course of action?"
p.151 The idea of performing an experiment to measure some important business quantity, unfortunately, does not come often enough to many managers... The word "experiment" could be broadly used to mean any phenomena deliberately created for the purposes of observation. You "experiment" when you run a security test to see if and how quickly a threat is responded to.
p.262 A question like "what is the value of ___?" is about as loaded as a measurement question gets. Usually, the perceived difficulty in measuring value is really the lack of a clear definition of why it is being measured... All valuation problems in business or government are about a comparison of alternatives... No value question will ever be asked that doesn't ultimately imply alternatives. If you have the right alternatives defined and the true decision defined, the value question will be much more obvious.
|
