Evaluating and Choosing Preferred Projects Richard de Neufville Professor of Engineering Systems and of Civil and Environmental Engineering MIT

Outline THE THOUGHT Fundamental Question: To what extent is it meaningful to look for “the best”? What can we expect to do? Value Functions (simple form of Utility) THE METHODS Target Curves Dominant Designs Tables

The Thought To what extent is it meaningful to look for “the best?” What defines “best”? Extreme (high or low) of all other possibilities This supposes what? Either (1): we have one metric of performance Or (2): metrics can be put on single scale Is (1) realistic? Is (2)? Under What Conditions?

Implied Need: “Value Function” Definition: V(X) is a means of ranking the relative preference of an individual for a bundle on consequences, X A non-quantitative form of Utility Function

Conditions for a “Value Function” Basic Axioms (1) Completeness or Complete Preorder People have preferences over all Xi Transitivity If X1 is preferred to X2; and X2 is preferred to X3 ; Then X1 is preferred to X3 Caution: Assumed True for Individuals; NOT Groups (discussion below)

Basic Axioms (2) Monotonicity or Archimedean Principle For any Xi (X*  Xi  X*) there is a w (0 < w < 1) such that V(Xi) = w V(X*) + (1 - w) V(X*) That is, More is Better (or Worse)

Consequence of V(X) Axioms Existence of V(X) Ranking Only Strategic Equivalence of Many Forms of V(X) Any Monotonic Transform of a V(X) is Still an Equivalent V(X) For example: V(X1, X2) = X12 X2 = 2 log (X1) + log (X2)

Value Functions

Does this apply to groups? Do all members in a group have same preferences? Possibly…. In general, however: Groups composed of stakeholders with different interests (builders, owners, users…) Their interests almost certainly diverge Can we expect them to agree? NO!

Example Intransitivity for Groups WHO WINS ELECTION? Left against Center: Left wins 2:1 Center against Right: Center wins 2:1 So: Left is preferred to Right? Wrong!!! Left against Right: Right wins 2:1 !!!

Where does this leave us? Under certain assumptions (conditions), Individuals can rank alternatives (from least to most preferred) This does not apply to groups If they agree on a process (set of voting rules) Then, they might be able to agree on a result Arrow’s Impossibility Theorem (or Paradox) [No “fair” voting system, without a dictator, satisfies everyone’s preferences…] So, concept of “best” not meaningful for design of complex systems => “preferred”

Analysis of Outcomes What criteria? Target Curve, concept and construction Robustness? Other dimensions Hassan satellite example

What can we expect to do? First, consider the nature of Problem for Evaluation and Choice Evaluation Many dimensions, metrics of performance Uncertainty about them, many states of metrics Best is not defined We can screen out dominated solutions Choice Any single person, must see, make TRADEOFFS Groups inevitably have to NEGOTIATE DEAL

What Criteria? “Expected Value” has been the index of choice for valuation… Why is this appropriate? Is this measure sufficient?

When do we see expected value? Expected Value based on recognizing possibility of Carbon Tax = 10.8 When do we meet this value?

Conclusion about E(V) A useful single metric But Insufficient Cannot describe the range of effects This is your A , B, C…

Other dimensions to explore The worst that could happen People are “risk averse”, sensitive to loss With some notion of probability of loss The best that might occur Upside also important Capex (capital expenditure = investment) Some measure of Benefit-cost

P5 , P10 or VAR P5 , P10 are values for 5%, 10% lowest end of a distribution. The percentage = probability losses do not exceed a particular level. VAR is a standard concept in finance = “Value at Risk” P10 = 10% VAR Motivated by lenders, who are mainly concerned about getting repaid

P90 , P95 or Value at Gain We have developed this concept as counterpart of “VAR” It represents the upside potential of a project Motivated by investors, interested in amount they may gain (not especially interesting to bankers…)

Target curve Target curve is the cumulative distribution of outcomes Going from worst case at x% probability To best case with y% probability combines VAR and “Value at Gain”

Target curve for Gulf Oil Case 12% VAR of only 16k; 4% VAG of ~ 30k

How do we construct Target Curve? See Decision Analysis example

What are elements of Target Curve? What are possible outcomes? Answer: 18 and 6 … and associated probabilities? Ans: 0.4 and 0.6 So Target Curve is as on next slide.

Construction of Target Curve

VARG Useful to Compare Alternatives TARGET CURVES can show relative merit of alternative designs

Concept of “Dominance” Idea: One alternative better than others on all dimensions Locus of dominant alternatives Dominated Alternatives If alternatives are dominated, they can be discarded

Dominance in Target Curves If one Target Curve is always to right of another… Does it dominate? Yes… but Does it mean that one alternative always perform better than the other? No! Frequency of occurrence do not translate that way!

Concept of “Robustness” Popular Basis for Design (“Taguchi method”) Robust design ≡ “a product whose performance is minimally sensitive to factors causing variability…” Robustness measured by standard deviation of distribution of outcomes

Illustration of Robustness Probability More Robust Smaller standard deviation Outcome

Do we want robustness? When might robustness be a good measure of performance? When we really want a particular result Tuning into a signal Fitting parts together, etc Is this what we want for maximizing value? No!! We want to limit downside but make upside as large as possible => higher σ

Robustness does not maximize expected value Probability Less Robust Higher Expected Value Outcome

Other Dimensions for Evaluation In addition to Expected Value Minimum and Maximum Values Capex = Initial Capital Expenditure = Investment “Benefit-Cost” ratio of EV / Capex Value Added by Flexibility Others? … depends on users

Example: Hassan Satellite Case

Take-Aways: Thoughts Evaluation is complex Many metrics of performance Plus Uncertainties Concept of “Best” is problematic Individuals may have a value function But groups are unlikely to do so Especially stakeholders with different interests Preferred is more realistic concept Need to show dominating alternatives; Help Decision-Makers see trade-offs

Take-Aways: Method “Expected Value” not sufficient Measure Target Curve powerful visual image Shows Maximum and Minimum Compares alternatives Tables usefully show Capex Benefit-Cost of “Expected Value / Capex” Value of Flexibility (more on this later)