Multiple Indications: estimating probabilities
When compounds in development have multiple indications, should you test them simultaneously or serially? If serially, in what order?
By using decision analysis, you can evaluate the possibilities, manage risk, and maximize value, but these models require estimates of payoffs and probabilities.
- Payoffs: You can estimate payoffs with traditional methods, but consider adjusting the results for optimizer's curse (described here).
- Probabilities: Estimating probabilities is problematic. The results of each indication, success or failure, are correlated. How do you estimate the probabilities of correlated results?
To present a solution, suppose an oncology compound has three indications awaiting phase III trials: A, B and C. For convenience, let S and F indicate success and failure, so that A=S and A=F indicate success and failure for indication A. We can produce all the probabilities and correlations needed for any model by estimating the seven probabilities in Table 1.
| Probability | Event | Type of probability |
|---|---|---|
| p(A=S) | the probability that A is successful | marginal |
| p(B=S) | the probability that B is successful | marginal |
| p(C=S) | the probability that C is successful | marginal |
| p(B=S|A=S) | the probability that B is successful given that A is successful |
conditional |
| p(C=S|A=S) | the probability that C is successful given that A is successful |
conditional |
| p(C=S|B=S) | the probability that C is successful given that B is successful |
conditional |
| p(C=S|A=S, B=S) | the probability that C is successful given that both A and B are successful |
conditional |
When estimating these probabilities, consider using these perspectives:
- Class: Every compound-indication is a member of a category, such as a therapeutic area and indication, with a level of innovation, such as a novel drug, label expansion or me-too compound. Form these categories, identify the reference class that best exemplifies the compound-indication.
- Case: The compound-indication is a specific case from the class.
One method of reducing estimation errors (1) estimates a probability for the class, (2) estimates a probability for the case, and (3) combines the two estimates by averaging them or using Bayes' law. This construction integrates more information, and from two different perspectives, than using the case estimate only, to produce a more precise estimate.
Some tips may help with estimating the class and case probabilities:
Class: If possible, estimate the class probabilities with historical data. Whether you use published estimates or calculate your own, consider using the path-by-path method described in:
Won CH, Siah KW, Lo AW (2019), "Estimation of clinical trial success rates and related parameters," Biostatistics, 20(2), pp. 273-286.
For oncology indications, you can find some of the conditional probabilities in the following paper, but check for mathematical inconsistencies (see below):
DiMasi JA, Reichert JM, Feldman L, Malins A (2013), "Clinical approval success rates for investigational cancer drugs," Clinical Pharmacology & Therapeutics, 94(3), pp. 329-335.
Case: A large literature shows that people poorly estimate correlations, so estimate conditional probabilities instead. Consider p(B=S|A=S). Decomposition can help you estimate this probability. Build a model that shows the key elements that compound-indications A and B have in common, the key elements in which they differ, and the range of values for these elements. This model will reveal the maximum correlation of the indications' results, which bounds the conditional probabilities, which limits the estimation errors. Estimate the values of the model's variables with empirical measurements, using expert judgment when empirical data is unobtainable.
Two additional issues are helpful:
Mathematical consistency: Consider indications A and B. The marginal probabilities plus any conditional probability determine all the remaining conditional probabilities. For example, p(A=S), p(B=S), and p(B=S|A=S) determines p(B=S|A=F), p(A=S|B=S), and p(A=S|B=F). If you estimate more than the marginals and one conditional probability, you will likely overdetermine the probabilities to produce several inconsistencies. For example, an inconsistency exists if the estimate of p(B=S|A=F) does not equal the valued calculated from other probabilities.
Mathematical inconsistencies occur with the other conditionals as well. For example, estimates can produce the illogical situation of p(C=S|A=S,B=S) < p(C=S|A=S,B=F).
Of course, you should try to derive consistent probabilities, but you can use inconsistencies that arise to your advantage. Adjusting estimates to remove inconsistencies reduces estimation errors. Via these adjustments, produce a consistent set of class probabilities and a consistent set of case probabilities. Then combine the class and case estimates.
Threeway covariates: With three indications, there is covariance between each pair of indications: Cov(A, B), Cov(A, C), and Cov(B, C) and a threeway "covariance," Cov(A, B, C). Models should use this term, which you can calculate from the probabilities in Table 1.
Monte Carlo software often includes a correlation matrix for pairwise correlations but omits the three covariance term. If you use Monte Carlo simulation, programming the correlations yourself may be helpful.