Cleaning the Crystal Ball

How intelligent forecasting can lead to better decision making.

strategy+business magazine

by Tim Laseter, Casey Lichtendahl, and Yael Grushka-Cockayne

Illustration by Lars Leetaru

Peter Drucker once commented that “trying to predict the future is like trying to drive down a country road at night with no lights while looking out the back window.” Though we agree with Drucker that forecasting is hard, managers are constantly asked to predict the future … Good forecasts hold the key to good plans. Simply complaining about the difficulty does not help.

Nonetheless, few forecasters receive any formal training, or even expert apprenticeship. … This lack of attention to the quality of forecasting is a shame, because an effective vehicle for looking ahead can make all the difference in the success of a long-term investment or strategic decision.

Competence in forecasting does not mean being able to predict the future with certainty. It means accepting the role that uncertainty plays in the world, engaging in a continuous improvement process of building your firm’s forecasting capability, and paving the way for corporate success. …

… By using the language of probability, a well-designed forecast helps managers understand future uncertainty so they can make better plans that inform ongoing decision making. We will explore the many approaches that forecasters can take to make their recommendations robust, even as they embrace the uncertainty of the real world.

The Flaw of Averages

In forecasting the future, most companies focus on single-point estimates: … [We] often forget that a point forecast is almost certainly wrong; an exact realization of a specific number is nearly impossible.

This problem is described at length by Sam Savage, an academic and consultant based at Stanford University, in The Flaw of Averages: Why We Underestimate Risk in the Face of Uncertainty (Wiley, 2009). He notes how focusing on an average without understanding the impact of the range can lead to flawed estimates. Better decisions result from taking the time to anticipate the likelihood of overshooting or undershooting the point, and then considering what to do today, given the range of possibilities in the future.

Savage highlights the simple example of a manager estimating the demand for 100,000 units of a product — based on a range of possible market conditions — and then extrapolating that average to produce a profit estimate. … As a result, the profits at an average demand level will be much different from an average of the profits across the range of possibilities. Rather than a simple average, a better forecast would present a wide range of scenarios coupled with a set of potential actions to influence the demand and profitability. …

Reflecting risk in forecasts is a simple concept and one that may seem easy to put into practice, but managers commonly ignore the uncertainties and simply collapse their forecasts into averages instead. … Consider a project with 10 parallel tasks. Each task should take between three and nine months, with an average completion time of six months for all of them. If the 10 tasks are independent and the durations are distributed according to a triangular distribution, chances are less than one in 1,000 that the project will be completed in six months, and the duration will be close to eight months. But using the six-month figure instead offers an almost irresistible temptation; after all, that’s the average input. …

In short, forecasting should not be treated as a game of chance, in which we win by getting closest to the eventual outcome. … Instead, it’s better to use the range of possible outcomes as a learning tool: a way to explore scenarios and to prepare for an inherently uncertain future.

Drivers of Uncertainty

The most useful forecasts do not merely document the range of uncertainties; they explain why the future may turn in different directions. … Just asking “Why might this happen?” and “What would happen as a result?” helps to uncover possible outcomes that were previously unknown. Recasting the driving forces as metrics, in turn, leads to better forecasts.

For example, the general business cycle is a driving force that determines much of the demand in the appliance industry. Key economic metrics, such as housing starts, affect the sales of new units, but a consumer’s decision to replace or repair a broken dishwasher also depends on other factors related to the business cycle, such as levels of unemployment and consumer confidence. With metrics estimating these factors in hand, companies in that industry … use sophisticated macroeconomic models to predict overall industry sales and, ultimately, their share of the sales.

… Whirlpool’s planners use their industry forecast models to focus executive attention, not replace it. The planners present the model for the upcoming year or quarter, describing the logic that has led them to choose these particular levels of demand and the reason the outcomes are meaningful. Executives can set plans that disagree with the forecasters’ predictions, but everyone has to agree on which input variables reflect an overly optimistic or pessimistic future. Even more important, managers can begin influencing some of the driving forces: For example, they can work with retail partners to encourage remodeling-driven demand to offset a drop in housing starts.

Black Boxes and Intuition

As the Whirlpool example demonstrates, mathematical models can help focus discussions and serve as a foundation for effective decision making. Thanks to the increasing power of personal computers and the Internet, we have a host of advanced mathematical tools and readily available data at our disposal for developing sophisticated models.

Unfortunately, such models can quickly prove to be a “black box,” whose core relationships and key assumptions cannot be understood by even a sophisticated user. … Without a clear understanding of the drivers of the model, executives will not be attuned to the changes in the environment that influence the actual results. …

A lack of understanding of the black boxes tempts many managers to dismiss the planners’ models and simply “go with the gut” in predicting possible challenges and opportunities. … Back in the early 1970s, Nobel laureate Daniel Kahneman and his longtime collaborator Amos Tversky began a research stream employing cognitive psychology techniques to examine individual decision making under uncertainty. Their work helped popularize the field of behavioral economics and finance. (See “Daniel Kahneman: The Thought Leader Interview,” by Michael Schrage, s+b, Winter 2003.) Work in this field has demonstrated that real-life decision makers don’t behave like the purely rational person assumed in classic decision theory and in most mathematical models.

…[Our] brains seek out patterns. … Though critical in evolutionary survival, this skill can also lead us to see patterns where they do not exist. For example, when asked to create a random sequence of heads and tails as if they were flipping a fair coin 100 times, students inevitably produce a pattern that is easily discernible. The counterintuitive reality is that a random sequence of 100 coin flips has a 97 percent chance of including one or more runs of at least five heads or five tails in a row. Virtually no one assumes that will happen in an invented “random” sequence. …

Our tendency to see patterns even in random data contributes to a key problem in forecasting: overconfidence. Intuition leads people to consistently put too much confidence in their ability to predict the future. As professors, we … challenge [our MBA students] to predict, with a 90 percent confidence level, a range of values for a set of key indicators such as the S&P 500, the box office revenues for a new movie, or the local temperature on a certain day. If the exercise is done correctly, only one out of 10 outcomes will fall outside the predicted range. Inevitably, however, the forecasts fail to capture the actual outcome much more frequently than most of the students expect. Fortunately, the bias toward overconfidence diminishes over time as students learn to control their self-assurance.

History Matters

Although Peter Drucker fretted about looking out the rear window of the car, in reality too many forecasters fail to examine history adequately. Consider the subprime mortgage crisis. In 1998, AIG began selling credit default swaps to insure counterparties against the risk of losing principal and interest on residential mortgage-backed securities. …

At the end of the fourth quarter of 1998, the delinquency rate for U.S. subprime adjustable-rate mortgages stood at just over 13 percent. By the end of the fourth quarter of 2008, this rate had almost doubled, to an astonishing 24 percent. … Although a 24 percent default rate seemed unprecedented to most bankers, a look back beyond their own lifetimes would have indicated the possibility. In 1934, at the height of the Great Depression, approximately 50 percent of all urban house mortgages were in default.

That is why looking back at past forecasts and their realizations can prove so valuable; it can help prevent overconfidence and suggest places where unexpected factors may emerge. Recently, researchers Victor Jose, Bob Nau, and Bob Winkler at Duke University proposed new rules to score and reward good forecasts. An effective “scoring rule” provides incentives to discourage the forecaster from sandbagging, a proverbial problem in corporate life. … By assessing forecasting accuracy, the rules penalize sales above the forecast number as well as sales shortfalls. …

… A recent survey by decision analysis consultant Douglas Hubbard found that only one out of 35 companies with experienced modelers had ever attempted to check actual outcomes against original forecasts — and that company could not present any evidence to back up the claim. …

Wisdom of Crowds

… Journalist James Surowiecki presented the case [for conventional wisdom] in his bestseller, The Wisdom of Crowds: Why the Many Are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies, and Nations (Doubleday, 2004). Furthermore, research into forecasting in a wide range of fields by Wharton professor J. Scott Armstrong showed no important advantage for expertise. In fact, research by James Shanteau, distinguished professor of psychology at Kansas State University, has shown that expert judgments often demonstrate logically inconsistent results. For example, medical pathologists presented with the same evidence twice would reach a different conclusion 50 percent of the time.

The old game of estimating the number of jelly beans in a jar illustrates the innate wisdom of the crowd. In a class of 50 to 60 students, the average of the individual guesses will typically be better than all but one or two of the individual guesses. …[That] result raises the question of why you shouldn’t use the best single guesser as your expert forecaster. The problem is that we have no good way to identify that person in advance — and worse yet, that “expert” may not be the best individual for the next jar because the first result likely reflected a bit of random luck and not a truly superior methodology.

For this reason, teams of forecasters often generate better results (and decisions) than individuals, but the teams need to include a sufficient degree of diversity of information and perspectives. …

Group dynamics can produce a different sort of challenge in bringing together a team; … Typically, a dominant personality steps forth and drives the process toward his or her predetermined view, making little or no use of the wisdom of the crowd. In The Drunkard’s Walk: How Randomness Rules Our Lives (Pantheon, 2009), physicist and writer Leonard Mlodinow describes a number of research studies that show how most people put too much confidence in the most senior or highest-paid person. …

Culture and Capability

To become proficient at forecasting, a company must develop capabilities for both achieving insight and converting that insight into effective decision making. The firm need not seek out the star forecaster, but instead should invest in cultivating an open atmosphere … that brings to the fore a more complete picture of the expert knowledge that already resides in many of its existing employees.

The resulting culture will be one in which managers recognize and deal with uncertainty more easily; …

In the end, overcoming the problems and traps in forecasting probably requires the use of all of these approaches together, within a supportive culture. …

… Too many managers dismiss the inherent uncertainty in the world and therefore fail to consider improbable outcomes or invest sufficient effort in contingency plans. The world is full of unknowns, even rare and difficult-to-predict “black swan” events, … Overreliant on either their intuition or their mathematical models, companies can become complacent about the future.

Consider, for example, the 2002 dock strike on the West Coast of the U.S., which disrupted normal shipping in ports from San Diego to the border with Canada for a couple of weeks. A survey conducted by the Institute for Supply Management shortly afterward found that 41 percent of the respondents had experienced supply chain problems because of the strike — but only 25 percent were developing contingency plans to deal with future dock strikes.

We can train our intuition to offer a better guide in decision making. To do so, we must be aware of our biases and remember that all models start with assumptions. Engaging a diverse set of parties, … forces us to articulate and challenge those assumptions by seeking empirical data. …Rather than seeking the ultimate model or expert, managers should adopt the axiom cited by General Dwight D. Eisenhower …“plans are nothing; planning is everything.” A good forecast informs decisions today, but equally important, forces us to consider and plan for other possibilities.

Reprint No. 10202

Author Profiles:

• Tim Laseter holds teaching appointments at an evolving mix of leading business schools, currently including the Darden School at the University of Virginia and the Tuck School at Dartmouth College. He is the author of Balanced Sourcing (Jossey-Bass, 1998) and Strategic Product Creation (with Ronald Kerber; McGraw-Hill, 2007), and is an author of the newest edition of The Portable MBA (Wiley, 2010). Formerly a partner with Booz & Company, he has more than 20 years of experience in operations strategy.
• Casey Lichtendahl is an assistant professor of business administration at the University of Virginia’s Darden Graduate School of Business. His research focuses on forecasting and decision analysis.
• Yael Grushka-Cockayne is an assistant professor of business administration at the University of Virginia’s Darden Graduate School of Business. Her research focuses on project management, strategic and behavioral decision making, and new product development.