Revenue forecasting errors dominate project evaluations
When evaluating a new product development project decision trees and capital asset budgeting models suffer from an often overwhelming problem. Consider the following facts:
- Revenue forecasts dominate the evaluation, as shown below.
- Revenues occur further in the future than any other aspect of a project, so they are the most difficult variable to estimate and contain the greatest estimation errors.
Together, these problems can wreck project evaluation and cause project selection to commit many errors, canceling valuable projects and funding poor ones.
Addressing these problems, let's first see how the revenue forecasts and their errors dominate project evaluations. The decision tree of Figure 1 evaluates a drug that is being considered for phase I clinical trials. The four chance nodes (branching points) represent the three phases of clinical trials and FDA approval. The top branch of each node represents success and the bottom branch represents failure. For example, the drug has a 70% chance of success in phase I clinical trials, and if successful, it has a 45% chance of success in phase II trials. The red numbers within the decision tree represent the costs of each stage, while the green number represents revenue, all presented as present values. For example, phase I clinical trials costs $3 million and phase II trials costs $6.5 million.
Each path through the tree ends in a terminal branch. The two columns on the extreme right of the figure show (1) the probability of each path through the tree and (2) the expected value of each path. To calculate the expected value of the project, sum the expected values of each path, adding the numbers in the rightmost column. The estimate of the project's expected value is $118.21 million.
Figure 1: A decision tree for a compound entering clinical trials in drug development.
Look closely at the rightmost column. The topmost value represents success: the drug passes clinical trials, is approved by the FDA and produces $700 million in revenues (present value). The probability of traversing this path is 19.85%, so the expected value of the path is $128.6 million. Compare the absolute value of this number to those of the other terminal branches. The revenue branch dominates all other branches. This relationship must be true. If the expected value of revenues did not dominate expected costs, the product development project would be rejected outright.
To see how the error in the revenue estimate dominates the tree, suppose the estimated revenues are 10% higher or 10% lower than the true value, shown in Figure 1. The estimated revenues would be $700±70 million, and the expected value of success (the topmost branch) would be $128.6±13.9 million. The error in the expected value, ±$13.9 million, is larger than the expected value of all other paths through the tree added together. This result is outstanding. The estimated costs are largely irrelevant because they are overwhelmed by the error in estimated revenues. (In terms of signal and noise, the noise from the revenue estimate is greater than the signals from all other variables - combined.)
The problem is actually worse because the estimates of projects' probabilities of success may be only minimally effective at distinguishing valuable projects from less valuable ones. (See my discussion, "Estimating probabilities of success - it's not so successful.") For many project selection decisions the idea of decision analysis identifying the most valuable projects by trading-off revenues, costs and probabilities is wishful thinking.
Let's look more closely at the error in the estimated revenue. Is a 10% error typical? A study of drug development by McKinsey & Company measured the average error in the demand forecasts of new pharmaceuticals. Figure 2 displays the results.
Figure 2: Average error in forecasting sales of new druges. Data from: Cha, M., B. Rifai, R. Sarraf (2013), "Pharmaceutical forecasting: throwing darts?" Nature Reviews Drug Discovery, vol. 12, pp. 737-738.
The average error in the demand forecasts made two years prior to market launch is 75% of the actual demand. FDA approval lasts for two years, so the 75% error applies to drugs that are entering the FDA's review process. These drugs have already completed phase III clinical trials, so the 75% forecasting error occurs after all selection decisions are completed. The forecasting error for drugs that are being considered for clinical trials must be even larger than 75%. For example, the decision to send a drug to phase I clinical trials occurs about ten years before launch. The demand forecasts for these drugs may be worthless.
One can see that revenue estimates dominate decision trees and that errors in revenue estimates are exceptionally large. These two facts cause decision trees (and capital asset budgeting models) to make large project evaluation errors, which produce project selection errors. Project selection errors occur when a valuable project is canceled (false-negative) or a poor project is funded (false-positive). (See my discussion, "How erroneous data causes project selection errors.")
Making matters worse, sophisticated selection methods are sensitive to evaluation errors, so inputting erroneous project evaluations into sophisticated selection techniques can create even more project selection errors. (See the section, "Reliability and project selection," in my discussion, "Where's the feedback?") Yet the combination of decision trees and complex optimization models is considered a project portfolio management (PPM) "best" practice.
After reading my discussions, many managers wish to share their experiences, thoughts and critiques of my ideas. I always welcome and reply to their comments.
Please share your thoughts with me by using form below. I will send reply to you via email. If you prefer to be contacted by phone, fax or postal mail, please send your comments via my contact page.
© 2014 Pipeline Physics. All rights reserved.