Pipeline Physics

Pipeline Physics produces profit
Gary Summers, PhD	1700 University Blvd, #936
President, Pipeline Physics LLC	Round Rock, TX 78665-8016
gary.summers@PipelinePhysics.com	503-332-4095

---

How to make drug development more productive

Decision analysis is a project portfolio management (PPM) best practice, and it proposes a curious relationship: good decisions can produce bad results. According to decision analysis, correct decisions maximize expected utility (or value), but because of bad luck, these "good" decisions can produce bad results. In fact, bad results often arise from poor management of uncertainty, and mismanaging uncertainty is not luck. It is a problem managers can fix.

The relationship proposed by decision analysis does not hold because several qualities of drug development violate the assumptions that found decision analysis. These violations include:

1. Even the best project evaluation and project selection models fail to account for a significant portion of the uncertainty that afflicts drug development. (See my discussions of uncertainty and PPM.)
2. Many PPM practices, including portfolio optimization, ignore a most important quality of projects: uncertainty decreases as projects progress down a pipeline. This quality is the reason why pipelines, not portfolios, optimize the selection of compounds. (See my discussion, "How PPM differs from modern portfolio theory.")
3. Decision analysis accepts a set of projects as "givens" to an optimization problem. Rather than merely accepting the choices, improving the set of choices that enter the front-end of a pipeline makes every part of drug development more successful. (See my discussion, "Where's the front-end)?"
4. Dismissing failure as bad luck prevents managers from learning from feedback, which perpetuates problems and errors. (See my discussion, "How to count cards in blackjack.")

How can one better manage uncertainty? Uncertainty affects decisions and business processes by causing decision errors. To better manage uncertainty, (1) identify decision errors, to see where uncertainty adversely affects a process, and (2) modify the process to reduce their severity or to eliminate the errors. Iteratively applying these steps creates constant improvement, which creates value.

Drug development produces two types of decisions errors: canceling marketable (safe and effective) compounds (called false-negatives) and advancing unmarketable compounds downstream (called false-positives). As described by my pipeline physics model (introduced below), the rates of these errors depend critically each phase's throughput. To improve drug development, one must consider both throughput and errors, and doing so produces the following three-fold goal for improving drug development:

Increase pipeline throughput while simultaneously eliminating false-positives and false-negatives.

Completely eliminating false-positives and false-negatives is impossible, but striving for this ideal creates value, similar to the ideals that guide other business processes, such as zero inventories in just-in-time manufacturing and zero waste in lean product development.

Preferably, one would measure the rates of false-positives and false-negatives for each phase of a pipeline, but until recently this measurement was impossible. One never learns the results of canceled compounds, and without classifying canceled compounds one cannot estimate false-positive and false-negative rates.

My new pipeline model enables a statistical analysis that overcomes this problem. Given sufficient data, the model and statistical analysis estimate, for each phase:

1. The fraction of unmarketable compounds, evaluated by the phase, that are mistakenly advanced (false-positive rate)
2. The fraction of marketable compounds, evaluated by the phase, that are mistakenly canceled (false-negative rate)
3. The fraction of compounds the phase evaluates that are marketable (base rate)
4. The phase's ability to distinguish marketable compounds to unmarketable ones (resolution).

Given sufficient data, the statistical analysis estimates the resolution produced by NPVs, expected values and raw clinical data, and it compares a company's performance at compound selection to the aforementioned metrics. To learn about these pipeline diagnostic tools, see my pipeline physics research proposal. (Contact me for the password needed to view the proposal.)

The above metrics are useful, but one can improve a pipeline without them. False-positives and false-negatives arise from five features of drug development pipelines, and by continuously improving these features one can achieve the above three-fold goal. Figure 1 illustrates the five features. They are: (1) a pipeline's front-end, (2) data, (3) project evaluation methods, (4) project selection techniques and (5) pipeline management. Let's consider each feature, and in the process, introduce my research on improving drug development.

Figure 1: The sources of decision errors in drug development.

1. Front-end: PPM's current best practices treat projects as "givens" to an optimization problem, which ignores the quality of the choices. Yet, the quality of compounds entering a pipeline affects (1) the value a pipeline can create, (2) the resolution required of each phase to achieve success and (3) each phase's false-positive and false-negatives rates. Using industry data and reasonable assumptions about the phases' resolutions, my new pipeline model reveals the front-end's impact. Reasonable increases in the quality of compounds entering phase I decreases development costs by 10%-20%.
2. Data: All data contains error, called estimation error. When used with a project evaluation model or metric, these estimation errors cause project evaluation errors, so the estimated value of a compound equals its true value plus an error term. For example:

   Estimated NPV = True NPV + Project Evaluation Error

   Project evaluation errors foil project selection, causing selection techniques to commit errors: false-positives and false-negatives.

   For analysis of some important estimation errors, see my discussions, "Revenue forecasting errors dominate decision trees," "Overconfidence and underestimating risk," and "Estimating probabilities of success: it's not so successful." To see how estimation errors affect cause evaluation errors and selection errors, read my discussion, "How erroneous data causes project selection errors."

3. Project evaluation model: Project evaluation models, such as scoring models, capital asset budgeting models and decision trees cause project evaluation errors via three mechanisms:
   1. Some models require data that has large estimation errors.
   2. Some models poorly manage estimation errors, propagating errors through their calculations to produce large project evaluation errors.
   3. All project evaluation models are both incomplete and imperfect, and these modeling limitations increase project evaluation errors.

To learn a method of estimating project evaluation errors, see my discussion "Where's the feedback?")

4. Project selection technique: Selection techniques reduce resolution via three mechanisms.
   1. First, selection techniques have varying levels of sensitivity to project evaluation errors. If sophisticated techniques contain few modeling errors (see below), they have high fidelity, meaning they represent a situation well. However, sophisticated models can be sensitive to project evaluation errors, so evaluation errors create more project selection errors (false-positives and false-negatives). Simpler selection techniques are less complete, so they have lower fidelity, but they are less sensitive to project evaluation errors.
      
      (Sensitivity to evaluation errors is a common problem. It causes overfitting in linear regression and optimization failures in finance's modern portfolio theory. See my discussions, "Action flexibility and state flexibility in PPM" and "What modern portfolio theory reveals about PPM.")
      
      Table 1 summarizes the relationship between project evaluation errors, project selection methods and performance. If project evaluation errors are small, project selection should exploit the fine details of a situation. Sophisticated techniques, if they contain few modeling errors, accomplish the task, but simpler selection techniques are too coarse to do it. If project evaluation errors are large, simpler techniques perform best. Sophisticated techniques are too sensitive to these errors, while simpler techniques are robust to them.

      Table 1: How to match your selection technique to your project evaluation errors.

      		Project Selection Technique
      		Simpler	Sophisticated
      Project Evaluation Errors	Small	Poor result (value left on the table)	Best result (achieves action flexibility)
      	Large	Good result (achieves state flexibility)	Poor result (too many avoidable errors)

      Applying these relationships to drug development requires research. My research addresses the following questions:

      • Can project portfolio optimization models be too complex?
      • Which features of optimization models are robust to uncertainty? Which ones are sensitive to uncertainty?
      • How much uncertainty is needed for simpler selection techniques, such as rankings and cutoff values, to outperform optimization models?
   2. Project selection techniques reduce resolution in a second way. Like all decision models and methods, selection techniques are imperfect, containing modeling errors, which cause project selection errors. Sophistication can make the problem worse because sophisticated models provide more opportunities for making modeling errors. In contrast, simpler models, which suffer from incompleteness, provide fewer opportunities to err. (To learn about modeling errors in project portfolio optimization, see my discussion, "When is optimization suboptimal?")

      Both factors affect a model's performance, and new research in decision theory addresses them via the following equation:

      Total loss = Loss from incompleteness + Loss from imperfections

      Imperfections include both modeling errors and sensitivity to estimation errors in data.

      The best model minimizes the total loss. Model sophistication creates a trade-off, in which building more sophisticated models reduces the loss from incompleteness but increases the loss from imperfections. Now consider the impact of uncertainty. High uncertainty increases estimation errors in data and causes modeling errors, both of which increase the loss from imperfections. When uncertainty increases, one should simply models to reduce this loss. The loss from incompleteness increases, but considering both sources of loss, the trade-off is beneficial. To learn more about these relationships, see my discussions, "Action flexibility and state flexibility in PPM" and "Why some C-level executives are skeptical of PPM."

   3. Project selection techniques can reduce resolution in a third way: emphasizing the wrong selection criteria. Let's return to decision analysis, mentioned at the start of this article. Decision analysis teaches two techniques for creating value: (1) selecting choices to maximize value and (2) investing to decrease uncertainty (which decision analysis calls the expected value of information).

      Table 2: Two choices. (Dollar values are millions.)

      	Compound A	Compound B
      Cost	190	240
      NPV (if successful)	1,000	1,500
      Prob. of Success	35%	30%
      eNPV	227	282
      Standard Deviation	568	797
      Difference in eNPV	55

      Table 2 helps illustrate a simple metric that compares the value the two techniques can potentially produce. It presents two compounds being considered for phase II trials. Note the following qualities:

      • The difference in the compounds' eNPVs estimates the value one can gain by striving to select the compound with the greater value. In this example the value is $55 million.
      • The projects' standard deviations estimate the amount of value one can gain by resolving uncertainty.

      The ratio of these values shows which approach has a greater potential for producing value:

      Projects' Standard Deviations
      Differences in eNPVs

      In this example the ratio is about 683 / 55 = 12.

      The amount of value gained from resolving uncertainty depends on the cost of resolving uncertainty and amount of uncertainty resolved, and a precise comparison considers portfolios, not individual projects. Nonetheless, the ratio shows where opportunity lies. For drug development, the greater opportunity is resolving uncertainty.

      Mimicking finance's modern portfolio theory (MPT), PPM emphasizes asset selection, but compounds differ from financial assets in a crucial way.

      • Investing in financial assets does not resolve uncertainty; tomorrow's stock prices are as uncertain as today's prices were yesterday. In MPT one cannot invest to resolve risk, so asset selection is the best method of creating value.
      • Drug development investments resolve uncertainty, and for drug development, resolving uncertainty can create more value than project selection.

      This distinction is the reason why financial assets are managed with portfolios while drug development is managed with pipelines. (To learn more about this distinction, see my discussion, "How PPM differs from modern portfolio theory.")

      Using Bayes' law, one can create a project selection model that addresses both the value created from compound selection and the value created from resolving uncertainty, as does an excellent article on designing clinical trials.1 One can then perform portfolio optimization to maximize total value.

      While the approach is theoretically sound, its results may be suboptimal. If resolving uncertainty creates the most value, compound evaluation and selection should use the information that maximizes resolution. Consider two types of information:

      1. Clinical trial data: Raw clinical trial data may provide the best estimates of technical success. Subjective probabilities likely have less resolution (they can't have more) and discount rates have no resolution, since most companies apply the same discount rate to each compound.
      2. Revenue forecasts: Research shows that revenue forecasts for compounds in development are extremely unreliable, with the average errors exceeding seventy percent of their true values.2

      Revenue forecasts dominate the two most common metrics used in portfolio optimization: NPV (financial theory) and expected value (decision analysis). Meanwhile both metrics eschew raw clinical data and address technical risk with subjective probabilities and discount rates, which may further reduce resolution. (See my discussions, "Revenue forecasting errors dominate decision trees" and "Estimating probabilities of success: it's not so successful.")

      Potentially, noncompensatory metrics, called Fast & Frugal heuristics, may be the best metrics for evaluating and selecting compounds. These metrics apply clinical trial data and revenue forecasts separately, applying the more reliable clinical data first, so it remains uncorrupted by erroneous revenue estimates. Meanwhile, with Fast & Frugal heuristics one can adjust technical selection criteria to be more stringent than the marketing criteria (or vice versa) to best exploit their respective resolutions. One can adjust the criteria for each phase as well, applying relaxed marketing criteria upstream, where revenues estimates are especially unreliable, and strengthening the criteria downstream. Alternatively, one can apply the opposite progression, depending on whether one wishes to keep options open or focus investment. With Fast & Frugal heuristics once can organize compound evaluation and selection, throughout discovery and development, to best manage uncertainty, making a pipeline more effective at managing uncertainty - which creates value.

      In contrast, expected values (decision theory) and NPVs (finance) combine all information and use all information in the same manner at every phase of discovery and development. These methods are theoretically sound only if drug development satisfies their assumptions about uncertainty, which is questionable. Potentially, pipelines may be more effective at managing uncertainty than project portfolios and Monte Carlo analysis, the techniques of decision theory. Which tool manages uncertainty better and creates more value: pipelines or portfolios? My research is answering this question.

5. Pipeline management:

   Consider the pipeline model implied by the following quote:

   "The hypothesis was simple: if one drug was launched for every ten candidates entering clinical development, then doubling or tripling the number of candidates entering development should double or triple the number of drugs approved. However, this did not happen; consequentially, R&D costs increased while output - as measured by launched drugs - remained static."3

   According to this hypothesis, sending more compounds to clinical trials leaves downstream attrition rates unaffected. In mathematical terms, the hypothesis assumes each phase's attrition rate is conditionally independent the preceding phases' attrition rates. This is a static pipeline model, static because the pipeline's shape is fixed. Many pipeline analyses assume a static pipeline, but the assumption is wrong. Let's look at an oft-cited analysis, by Paul et al.4

   Paul et al. present the industry averages for the cost, duration and probability of technical success (success rate) for each phase in discovery and development. To identify the best strategy for raising productivity, Paul et al. vary each variable, individually, while holding all other variables constant - a static pipeline model. With all variables set at their industry averages, the cost of developing a drug is $1.78 billion. Their analysis identified the most impactful variable as the phase II's success rate. Its industry average is 34%. Raising its value to 50% reduces the cost of a drug by 25% to $1.33 billion. Reducing its value to 25% raises the cost of a drug by 29% to 2.3 billion.

   Consider their key assumption: changing phase II's success rate has no impact on phase III's success rate. This assumption is incorrect because phase II's throughput (success rate) affects phase III's success rate as follows:

   • With strict criteria executives are more selective, advancing only the best compounds to phase III. Phase II has lower throughput but phase III receives better compounds. A higher percent of them are marketable, so phase III's attrition rate falls and its success rate rises.
   • With relaxed criteria executives are less selective, advancing more compounds to phase III. Phase II has greater throughput, but phase III receives worse compounds. A smaller percent of them are marketable, so phase III's attrition rate raises and its success rate falls.

   Figure 2 illustrates the correct physics of pipelines. For a reasonable scenario (see below), it shows phase II's impact on phase III. Decreasing (increasing) phase II's throughput sends better (worse) compounds to phase III. As a result, phase III's success rate is inversely proportional to phase II's throughput.

   

   Figure 2: How phase II's throughput (success rate) affects the quality of compounds entering phase III. The analysis assumes 35% of phase II compounds are marketable and a reasonable resolution by phase II trials: a signal-to-noise ratio of 1.2.

   The relationship mitigates the impact of phase II's throughput on productivity. Raising phase II's throughput increases productivity, but it reduces phase III's success rate, which lowers productivity. Lowering phase II's throughput reduces productivity, but it increases phase III's success rate, which increases productivity. The total effect is small. Repeating Paul et al.'s analysis with the correct pipeline physics produces the following predictions:

   • If executives relax selection criteria, raising phase II's throughput to 50%, the cost of creating a drug decreases 4% to $1.71 billion.
   • If executives strengthen selection criteria, lowering phase II's throughput to 25%, the cost of creating a drug increase by 6% to $1.89 billion.

   These results will disappoint when compared to the expectations set by Paul et al., which suggest raising and lowering pipeline productivity by 25% and 29%.

   While always qualitatively correct, Figure 2's numbers involved depend on two key variables: the percent of phase II compounds that are marketable and phase II's resolution. These values are not published by Paul et al. or any industry analysis. These variables, false-negative rates and false positives rate are not part of static pipeline model, and until recently, these variables could not be estimated from pipeline data. Fortunately, given sufficient data, my new pipeline model and statistical analysis can estimate these values for each phase of drug discovery and development. Since I do not yet have data to analyze, the above example uses reasonable assumptions that are compatible with Paul et al.'s data. It assumes 35% of phase II compounds are marketable and phase II's resolution has a signal-to-noise ratio of 1.2.

   With the new pipeline model, I perform pipeline sensitivity analyses, derive pipeline's optimal shapes and assess pipeline management strategies. Here are two examples:

   • One prescription for raising productivity is the "quick kill" approach. It proposes higher standards for advancing compounds from phase IIA to phase IIB, to reduce phase III's attrition rate. As Figure 2 illustrates, the strategy sends better compounds to phase III, which reduces phase III's attrition, but the strategy reduces phase II throughput as well. As illustrated above, these affects largely offset each other, so the "quick kill" strategy provides only small benefits, at best.
   • Pipeline physics reveals that increasing phase II's resolution reduces the cost of creating a drug by about 10%. Several companies are pursing this strategy.3,5,6 Shaping a pipeline to better exploit phase II's higher resolution reduces costs by another 10%.

   My new pipeline model integrates the ideas of this paper to fully describe how pipelines behave. You can learn more about my model by reading my discussion, "Managing drug development pipelines." For a mathematical introduction, see my pipeline physics research proposal. (Contact me for the password needed to view the proposal.)

Conclusion

Table 3 summarizes the differences between the pipeline physics framework and current PPM's best practices. The distinctions arise from alternative assumptions about uncertainty. PPM's current best practices arise from decision theory, which until recently assumed models correctly and completely modeled risk and uncertainty.

Recent advances in decision theory relax this assumption and study the impact of uncertainty that cannot be modeled, such as unknown-unknowns. (See my discussion, "The difference between theory and practice: it's disappearing.") These new theories of uncertainty follow changes in management, in which major business functions successively switched from a paradigm of forecasting and optimizing to frameworks that better manage uncertainty, such as just-in-time manufacturing, agile project management, flexible supply chains and lean product development. (See my discussion, "Does PPM need a new paradigm?")

Currently, PPM's best practices require forecasting (expected values and NPVs) and optimizing (portfolio optimization), but perhaps, like other business functions, drug development can manage uncertainty better. I am developing pipeline physics to achieve this goal.

Table 3: Pipeline Physics' and PPM's paradigms.

Distinction	Pipeline Physics	PPM's Current Best Practices
Fundamental assumption	No model incorporates all uncertainty.	A model's probabilities completely and correctly represent the uncertainty that affects a decision in drug development.
Strategy for creating value	Increase pipeline throughput while simultaneously reducing the false-positive and false-negative rates (dynamic, pipeline model).	Maximize portfolio value (static, portfolio model).
Pipeline management	Coordinate phases I, II and III to maximize productivity.	Assume a pipeline's shape is static: each phase's attrition rate is conditionally independent of previous phases' attrition rates.
Front-end	The front-end affects the (1) optimal throughput for each phase (optimal pipeline shape), (2) each phase's false-positive and false-negative rates and (3) the amount of resolution required for success. Improve the front-end by extending the pipeline physics framework to discovery.	Accept one's choices as "givens" to an optimization problem.
Compound selection	Use a selection technique that best fits the amount of uncertainty you face. When uncertainty is sufficiently high, simplicity is a virtue (see Table 1).	Use portfolio optimization and model all impactful details of a situation.
Compound evaluation	Maximize resolution, which may require using technical and market information separately, which prevents uncertain revenue forecasts from corrupting clinical trial data. Representing clinical trial data with probabilities may add noise to data, which reduces resolution.	Estimate compounds' eNPVs by representing clinical trial data with probabilities and combining the probabilities with revenue forecasts.
Feedback	Find and fix errors, perhaps by using my new statistical analyses and pipeline diagnostic metrics.	Instead of feedback, evaluate PPM with maturity models. Periodically re-estimate compound evaluations and re-optimize a drug development portfolio.

1 Julious, S. and D. Swank (2005), "Moving statistics beyond the individual clinical trial: applying decision science to optimize a clinical development plan," Pharmaceutical Statistics, 4(1), pp. 37-46.

2 Cha, M., B. Rifai and R. Sarraf (2013), "Pharmaceutical forecasting: throwing darts?" Nature Reviews Drug Discovery, 12(10), pp. 737-738.

3 Cook, D., D. Brown, R. Alexander, R. March, P. Morgan, G. Satterthwaite and M. Pangalos (2014), "Lessons learned from the fate of AstraZeneca's drug development pipeline: a five-dimensional framework," Nature Reviews Drug Discovery, 12(6), p. 419.

4 Paul, S., D. Mytelka, C. Dunwiddie, C. Persinger, B. Munos, S. Lindborg and A. Schacht (2010), "How to improve R&D productivity: the pharmaceutical industry's grand challenge," Nature Reviews Drug Discovery, 9(3), pp. 203-14.

5 Morgan, P, P. Van Der Graaf, J. Arrowsmith, D. Feltner, K. Drummond, C. Wegner and S. Street (2012), "Can the flow of medicines be improved? Fundamental pharmacokinetic and pharmacological principles toward improving phase II survival," Drug Discovery Today, 17(9/10), pp. 419-424.

6 Owens, P., E. Raddad, J. Miller, J. Stille, K. Olovich, N. Smith, R. Jones and J. Scherer (2015), "A decade of innovation in pharmaceutical R&D: the Chorus model," Nature Reviews Drug Discovery, 14(1), pp. 17-28.

© 2015 Pipeline Physics. All rights reserved.