Pipeline Physics

Pipeline Physics produces profit
Gary Summers, PhD	1700 University Blvd, #936
President, Pipeline Physics LLC	Round Rock, TX 78665-8016
gary.summers@PipelinePhysics.com	503-332-4095

---

Is my scoring model good enough?

Scoring models are valid (see my discussion, "Are scoring models valid?") but does a scoring model produce good project selection? Does it keep false-positives (funding poor projects) and false-negatives (canceling valuable projects) to a minimum?

Scoring models exploit correlations between project attributes and project value. If the total correlation of scores and value is too low, the model will produce large project evaluation errors. These evaluation errors cause project selection errors, which destroy value.

Is the correlation of the project scores with project value high enough? All scoring models contain six errors that decrease correlation. To build a good scoring model, one must limit the damage from each error. Here are the six errors:

1. Imperfect attributes scales: Attribute scales are the most important parts of a scoring model. Poor scales increase the errors in the estimates of the attributes' values, and incorrect attribute values produce incorrect scores, which decrease correlation. Ideally, attribute scales are defined on qualities that are measured via tests, such as technical and marketing tests. In the worst case, they are "anchored" subjectively using the terms best and worst.
2. Imperfect attribute estimates: Errors in estimating attribute values produce errors in project scores, reducing the correlation between project scores and project value. The most accurate estimates come from having well-defined attribute scales and measurements made by testing qualities of a project with, for example, marketing tests. The worst attribute measures are guestimates made on poorly defined scales.
3. The wrong attributes: A scoring model should use the attributes that best predict success. How does one know which attributes to use? Think in terms of signals and noise.
   • Signal: Consider an attribute scale, such as one that measures fit with strategy. If a project's value for the attribute is measureed without error, it has a value of AT, where T stands for the true value. Evaluating multiple projects produces a variety of values, as some projects score high on the attribute while others score low. A strong signal varies considerably over projects, aptly distinguishing the best from the worst. A weak signal varies little over projects, finding little difference among them. One measures the variation, the strength of the signal, with the standard deviation of the projects' true attribute values, σAT
   • Noise: Measurements have error. One can think of measurement error as a term added to the true value, so that the measured score is the true score plus an error term: A = AT + e. The error term causes variation in the measurement, and the amount of variation depends on the standard deviation of the error term, σe.

   Together the standard deviations of the attributes' true values and of the measurement errors determine the variation of the attribute measurement:

   σA2 = σAT2 + σe2

   Now we can specify the reliability of an attribute's measurement:

   

   Reliablity determines an attribute's ability to distinguish valuable projects from less valuable ones. Consider these situations:

   • If an attribute (signal) varies little over a set of projects, so σAT is small, the signal is weak and reliability is low.
   • If the measuresment error (noise) is large, so σe is large, the reliability is low.
   • If the signal is large and the noise is small, so σAT is large and σe is small, reliability is high.

   When reliability is low, an attribute contributes little to project selection, but when it is high, an attribute helps to identify the most valuable projects. To select the best attributes for the scoring model, calculate the reliability of each attribute and use the attributes that are the most reliable.

   To estimate an attribute's reliability one must calculate the numerator and denominator of above fraction. Calculating the denominator is easy. It's the standard deviation of an attribute's value over of the projects, σA2. Measuring the numerator requires more subtle techniques. I am developing and testing them in my research on scoring models. (You can learn more about reliability from my discussion, "Where's the feedback?")

4. Too many or two few attributes: A larger model with more attributes can bring great benefits. Potentially, it can average away the errors in the attribute estimates, so the imprecision in a project's score can be less than those in its attributes. However, two qualities of scoring models mitigate this benefit.

The first problem comes from correlation. Correlated errors do not average away. Errors are correlated when one measures attributes with the same information. For example, if one includes four marketing attributes measured via the same marketing study, the same marketing model or the same executives' guestimates, the errors will be highly correlated.

The second problem comes from the attribute weights. With optimal weights, adding more attributes (with uncorrelated errors) always improves the scoring model. Unfortunately, suboptimal weights thwart the effect. The more the weights differ from the optimal ones, the fewer attributes should be included in the model.

One should have the right number of attributes, not too many, not too few. My current research studies this issue and will identify the right number of attributes for the scoring model.
5. Imperfect attribute weights: For every scoring model there is an optimal set of weights, weights that maxiize reliability, but unfortunately, no one knows these weights. All one can do is make a good guess. Research shows that weights that are reasonably close to the optimal ones perform very well, achieving near optimal performance, but weights that are far from the optimal ones are harmful. When guessing the weights, the goal is not perfection. The goal is to minimize the likelihood of being wrong by too much.

With this in mind, common practices commit at least one error. Most methods of assigning weights, such as AHP, recommend setting weights according to an attribute's perceived importance, so more important attributes get larger weights. However, importance is only one of two key factors. Imprecise measurements matter as well. If attributes' values are uncorrelated with each other, imprecisely measured attributes should have smaller weights.

Correlations between attribute values are critical. Some research suggests that attribute values are uncorrelated with each other. For example, among new product development projects, the values of marketing attributes might be uncorrelated with the values of technology attributes. However, the values of other attributes may be correlated. For example, newness to the firm, the degree of technological advance and the competitive advantage over other products might be positively correlated.

There are two cases:
• If attributes are uncorrelated, skewed weights are best. When assigning these weights one should consider both an attribute's importance and the imprecision in its measurement. The weight should increase with importance and decrease with imprecision.
• If attributes are correlated, especially if some are negatively correlated, flat weights are best. Give every attribute the same weight.

If there are a sufficient number of projects, one can assess the correlations via a correlation matrix.
6. Imperfect correlation: Scoring models exploit correlations but the correlations of project attributes with project value are always imperfect. If the five problems above vanished, the scoring model would still imperfectly predict project value, and even in this best of all possible worlds, the correlation could be too low to produce good project selection.

My current research is testing methods of estimating the correlation of scoring models, including all six of the above errors. It is producing knowledge and metrics to help build better scoring models.

© 2014 Pipeline Physics. All rights reserved.