Pipeline Physics

Pipeline Physics produces profit
Gary Summers, PhD	1700 University Blvd, #936
President, Pipeline Physics LLC	Round Rock, TX 78665-8016
gary.summers@PipelinePhysics.com	503-332-4095

---

Managing drug development pipelines

Decision theory, the theoretical foundation of project portfolio management (PPM), is evolving to treat uncertainty differently than it treats risk. (See my discussion, "The difference between theory and practice: it's disappearing.") These advances suggest changing PPM and drug development in significant ways:

• The common, static, portfolio model should be replaced with a dynamic, pipeline model.
• Instead of optimizing a forecast of portfolio value, better management of uncertainty, which reduces the frequency of decision errors, creates more value.

Consider each of these changes.

Pipelines versus portfolios: PPM's current best practices - decision analysis and portfolio optimization - mimic finance's modern portfolio theory (MPT), but compounds differ from financial assets in a crucial way:

• Investing in financial assets does not resolve uncertainty; tomorrow's stock prices are as uncertain as today's prices were yesterday.
• Drug development investments resolve uncertainty, and for drug development, resolving uncertainty is the greatest source of value.

This distinction is the reason why financial assets are managed with portfolios while drug development is managed with pipelines. (To learn more about this distinction, see my discussion, "How PPM differs from MPT.")

Managing uncertainty versus maximizing value: Previously, managers managed manufacturing, supply chain management, project management and product development by using a two-step process of forecasting and optimization, or colloquially, predict-and-plan. The results were dismal. Subsequently, each function adopted new frameworks that succeeded by managing uncertainty better, and these new frameworks are now well-known: just-in-time, theory of constraints, flexibility, lean and agile. PPM lags behind other business functions by adhering to the old, harmful predict-and-plan approach. (To learn more about the predict-and-plan and the manage uncertainty frameworks, see my discussion, "Does PPM need a new paradigm?")

Managers must organize compound evaluations, compound selection, the design of clinical trials and their pipeline management to better manage uncertainty. This discussion addresses pipeline management, showing how pipelines behave, especially how upstream processes (discovery, phase I and phase II) affect phase III's attrition rate and the cost of developing a drug. Pipeline physics, a new mathematical model of drug development pipelines, predicts the relationships presented below.

While this discussion addresses pipelines, some executives may wish to learn more about uncertainty's impact on PPM. These executives may appreciate the discussions mentioned above and the following ones:

• Why some C-level executives are skeptical of PPM.
• Action flexibility and state flexibility in PPM.
• What modern portfolio theory reveals about PPM.
• How to count cards in blackjack.

Executives who wish to know more about managing uncertainty in drug development may appreciate my discussion:

• How to make drug development more productive.

Definitions and pipeline

How exactly does better management of uncertainty create value? Uncertainty affects drug development by causing decision errors, so creating value requires managers to (1) identify decision errors, to see where uncertainty adversely affects drug development, and (2) modify drug development to eliminate the errors or reduce their severity. To apply this process, we must define a decision error in drug development and several additional concepts.

• Marketable compound: A compound that is safe and effective.
• Unmarketable compound: A compound that is unsafe or ineffective.
• False-positive: Advancing an unmarketable compound to the next phase.
• False-negative: Canceling a marketable compound.
• False-positive rate: A phase's false-positive rate is the fraction of the unmarketable compounds it evaluates that it advances to the next phase.
• False-negative rate: A phase's false-negative rate is the fraction of the marketable compounds it evaluates that it cancels.
• Base rate: A phase's base rate is the percent of compounds it evaluates that are marketable.
• Resolution: A phase's ability to distinguish marketable compounds from unmarketable ones.
• Throughput: The percent of compounds a phase advances, which is typically called a phase's success rate. (The term throughput is preferred to success rate because some advancing compounds are unmarketable. Advancing such compounds makes a pipeline less successful.)
• Phase III's success rate: The percent of compounds entering phase III that become successful drugs.

With these definitions let's explore the pipeline presented by Figure 1 and Table 1. To better show how upstream processes (discovery, phase I and phase II) effect productivity the pipeline combines phase III and the new drug application (NDA). This simplification leaves the qualitative relationships presented below untouched and has only minor effects on the forthcoming calculations.

Table 1: Estimates of industry-wide pipeline statistics.

	Phase I	Phase II	Phase III & NDA
Throughput	54%	34%	64%
Attrition rate	46%	66%	36%
Resolution (signal-to-noise ratio)	0.4	1.2	∞
Base rate	29%	35%	64%
Capitalized cost per compound ($ millions)	32.14	69.68	234.05
Cost per drug ($ millions)			959

Table 1 compliments Figure 1 by providing industry-wide data on development. Rows 1 and 2 present each phase's throughput (success rate) and attrition rate. Row 5 presents the capitalized cost of each phase. This data comes from the well-cited article:

Paul, S., D. Mytelka, C. Dunwiddie, C. Persinger, B. Munos, S. Lindborg and A. Schacht (2010), "How to improve R&D productivity: the pharmaceutical industry's grand challenge," Nature Reviews Drug Discovery, 9(3), pp. 203-14.

Rows 3 and 4 present metrics from pipeline physics, which is a new mathematical model of drug development pipelines and phase-gate systems. Row 3 presents reasonable values for each phase's resolution, with phase II's resolution exceeding phase I's resolution, as appropriate. These values are guesses and one would prefer to have estimates produced by analyzing pipeline data, as I mention below. The table assigns infinite resolution to phase III and the NDA, meaning they perfectly distinguish marketable compounds from unmarketable ones. While unrealistic, this assumption simplifies the example without affecting its results too much.

In Row 4 the base rate for phase I is estimated from the industry data and the model. Industry-wide, 12% of phase I compounds become new drugs (from Paul et al.). Given the industry throughput rates, the assumed resolutions and the equations of pipeline physics, the 12% pipeline success rate implies that 29% of phase I compounds are marketable (phase I's base rate). The other base rates follow from the phases' throughputs and resolutions.

Pipeline capacity

Every pipeline has a limited capacity to create value. A pipeline's capacity does not refer to its resource constraints, which increase with investment. Uncertainty limits a pipeline's capapcity to create value by producing the following relationship:

Phase III's attrition rate varies inversely with pipeline throughput.

Figure 2 illustrates this relationship, which arises from the following qualities of pipelines:

• Phase II has a finite resolution, so it makes errors, canceling some marketable compounds and advancing some unmarketable ones.
• Managers can raise phase II's throughput by lowering the standard of advancement. More compounds reach phase III, but because phase II is less selective, a greater percentage of them are unmarketable and phase III's success rate declines.
• Managers can reduce phase II's throughput by raising the standard for advancement. Fewer compounds reach phase III, but because phase II is more selective, a greater percentage of them are marketable and phase III's success rate increases.

The impact of phase II's throughput on phase III's success have offsetting effects on the cost of drug development:

• Increasing phase II's throughput raises productivity, but decreasing phase III's success rate reduces productivity.
• Decreasing phase II's throughput reduces productivity, but increasing phase III's success rate raises productivity.

Using the data from Table 1, including a phase I throughput of 54%, Figure 3 shows how the cost of developing a drug varys with phase II's throughput. The minimum cost for developing a drug is $891 million, which occurs when phase II's throughput is 55%. Despite its appearance, the cost curve is quite flat. Development cost is within 5% of its minimum value if phase II's throughput is between 37% and 78%. This broad range arises because of the offsetting effects described above. Importantly, inefficiencies escalate more from having too little throughput than from having too much throughput. A cautious executive will error by setting phase II's throughput too high, rather than too low.

Varying both phase I's and phase II's throughput to optimally shape Figure 1's pipeline sets phase I's and phase II's throughput at 63% and 55%, respectively. Development costs are then $888 million. Further reducing costs requires achieving combinations of phase II's throughput and phase III's success rate that lie above the curve of Figure 2. It requires:

• increasing phase III's success rate while maintaining phase II's throughput,
• raising phase II's throughput while maintaining phase III's success rate, or in the best case,
• simultaneously raising phase II's throughput and phase III's success rate.

To achieve these results executives must shift Figure 2's curve upward, and only two accomplishments achieve this result:

• Improving the quality of compounds sent to phase I.
• Increasing phase I's or phase II's resolution.

Let's see the impact of each approach.

Better phase I compounds

Let's begin by assuming phase I's throughput remains unchanged. Sending better compounds to phase I, raising phase I's base rate, affects phase I in two ways. First, a higher percentage of the compounds phase I advances to phase II are marketable, which raises phase II's base rate. This effect increases pipeline productivity and reduces the cost of developing a drug. Second, phase I has more false-negatives. To reduce these errors an executive should increase phase I's throughput. This effect increases productivity and sends additional marketable compounds to phase II, some of which become new drugs.

Now recall that increasing phase I's base rate raised phase II's base rate. With a higher base rate, phase II benefits from the two affects just described. It can increase its throughput while also raising phase III's base rate. Phase III's higher base rate raises phase III's success rate. In total, improving the quality of compounds that discovery sends to phase I improves all phases of development.

To illustrate some of these effects assume that discovery sends better compounds to the pipeline of Figure 1 and Table 1, raising phase I's base rate from 29% to 35%. If phase I's throughput remains unchanged, phase II's base rate rises to 42%. Figure 4 shows the impact on the pipeline's capacity. The curve describing the trade-off between phase II's throughput and phase III's success rate shifts upward. Compared to the original pipeline, executives can increase phase II's throughput while simultaneously obtaining a higher phase III success rate. The cost curve in Figure 3 shifts downward, reducing the minimum drug development cost by 14% to $763 million.

The impact of improving phase I's base rate is more dramatic when the base rate is low. For example, in an oncology pipeline, about 13.4% of the phase I compounds are marketable. Raising this value to 16.75%, an additional 2.35 marketable compounds per 100 phase I compounds, reduces the cost of an oncology drug by $300 million. To see this example, which is utilizes the pipeline physics framework, as does the present discussion, see my essay, "How discovery impacts development.")

Increase phase II's resolution

The curves in Figures 2 and 4 demonstrate phase II's limited ability to distinguish marketable compounds from unmarketable ones. If phase II advances few compounds, it commits few false-positives but many false-negatives. As throughput increases the false-negative rates falls, but the false-positive rate rises. By adjusting phase II's throughput executives can trade false-negatives for false-positives. However they adjust phase II's throughput, they face a offsetting effect that limit their ability to increase productivity. If they raise phase II's throughput, to increase productivity, phase's III success rate declines, which reduces productivity. If they reduce phase II's thoughput, to increase phase III's success rate, which raises productivity, the lower phase II throughput reduces productivity.

The exact trade-off depends on phase II's resolution, and higher resolution has a profoundly good impact on drug development. With higher resolution, for every level of throughput, phase II commits fewer false-positives and fewer false-negatives. The trade-off is less constraining, the curve in Figure 2 rises and drug development costs fall.

Figure 5 illustrates this result. The top curve represents a 20% increase in resolution, as measured by the signal-to-noise ratio (S-N-R). The curve shifts upward, allowing executives to simultaneously increase phase II's throughput and raise phase III's success rate, and both effects increase productivity. In this example, the minimum cost of drug development drops 6%, from $888 million to $838 million. The impact is more dramatic than it seems because the signal-to-noise ratio overstates the improvement in resolution. Another key metric measures the same improvement in resolution at a more mundane increase of 5%.

How do one increase phase II's resolution? One can increase the number of patients phase II trials, or one can design phase II to more incisively evaluate compounds' disease targets, pharmacological activity, safety profiles and intended patient populations. Several large pharmaceutical companies are implementing the latter approach.1

Additionally, executives can evaluate and select compounds by using methods that manage uncertainty well. PPM's current best practices may mismanage uncertainty and reduce resolution, which would harm drug development. Consider the following problems with current best practices:

• Expected values and NPVs, two common metrics, require revenue forecasts, but revenue forecasts contain large errors. A study by McKinsey and Company found that two years prior to launch, when a compound is entering FDA review, the average error in revenue forecasts is 75% of the revenues the drugs subsequently produced.2 Likely, the revenue forecasts for compounds further upstream contain even larger errors because they require forecasting further into the future.
• Expected values and NPVs require estimating compounds probabilities of technical success, but the process of producing these estimates reduces resolution (see the discussion listed below). In contrast raw clinical data may provide higher resolution than expected values and eNPVs.
• Sophisticated selection techniques are more sensitive to compound evaluation errors, such as those described above, and this sensitivity can increase false-positive rates and false-negative rates, reducing resolution. Additionally, portfolio optimization can misshape pipelines, which raises drug development costs.

To learn more about project evaluations errors, see my discussions:

• Revenue forecasting errors dominate decision trees.
• Estimating probabilities of success: it's not so successful.
• How erroneous data causes project selection errors.

To learn how selection techniques can err, especially those that are too sophisticated, see my discussions:

• Where's the feedback?
• When is optimization suboptimal?
• How PPM differs from modern portfolio theory.

To see an alternative framework for managing drug development, see my discussion:

• How to make drug development more productive.

Conclusion

Uncertainty prevents one form perfectly distinguishing marketable compounds from unmarketable ones, so false-positives and false-negatives are inevitable. Executives can trade false-positives for false-negatives by varying pipeline throughput. They can have higher pipeline throughput with lower a phase III success rate or lower pipeline throughput with a higher phase III success rate. Neither choice is satisfactory.

Dramatically improving productivity requires increasing pipeline throughput while raising phase III's success rate, and only two methods achieve the result: (1) improving discovery, so it sends better compounds to development, and (2) increasing phase I's or phase II's resolution. Executives can raise the phases' resolutions by designing more incisive clinical trials. Additionally, common PPM metrics and practices may mismanage uncertainty and reduce resolution, especially evaluating compounds with expected values, NPVs and eNPVs and selecting compounds with the more sophisticated portfolio optimization models. Executives can increase productivity by fixing these problems.

Recall how the above analysis guessed phase I's and phase II's resolutions. Preferably, one would measure these resolutions from pipeline data, but until recently this measurement was impossible. One never learns the results of canceled compounds, and without classifying canceled compounds, as marketable or unmarketable, one cannot estimate the phases' resolutions.

The pipeline physics model, which founds the above analysis, enables statistical techniques to overcome this problem. Given sufficient data, the techniques estimate phase I's, phase II's and the preclinical phase's (1) base rate, (2) false-positive rate, (3) false-negative rate and (4) resolution. The statistical analysis estimates the resolution produced by NPVs, expected values and raw clinical data as well, and it compares a company's performance at compound selection to the resolution produced by the these metrics.

To learn more about pipeline physics and the statistical analyses, see my pipeline physics research proposal. (Contact me for the password needed to view the proposal.)

1 Cook, D., D. Brown, R. Alexander, R. March, P. Morgan, G. Satterthwaite and M. Pangalos (2014), "Lessons learned from the fate of AstraZeneca's drug development pipeline: a five-dimensional framework," Nature Reviews Drug Discovery, 12(6), p. 419.

Morgan, P, P. Van Der Graaf, J. Arrowsmith, D. Feltner, K. Drummond, C. Wegner and S. Street (2012), "Can the flow of medicines be improved? Fundamental pharmacokinetic and pharmacological principles toward improving phase II survival," Drug Discovery Today, 17(9/10), pp. 419-424.

Owens, P., E. Raddad, J. Miller, J. Stille, K. Olovich, N. Smith, R. Jones and J. Scherer (2015), "A decade of innovation in pharmaceutical R&D: the Chorus model," Nature Reviews Drug Discovery, 14(1), pp. 17-28

2 Cha, M., B. Rifai and R. Sarraf (2013), "Pharmaceutical forecasting: throwing darts?" Nature Reviews Drug Discovery, 12(10), pp. 737-738.

© 2014 Pipeline Physics. All rights reserved.