A notional science prioritisation Morton, Alec and Bird, D. and Jones, A. and White, M. (2011) Decision conferencing for science prioritisation in the UK public sector: a dual case study. Journal of the Operational Research Society, 62 (1). pp ISSN

Evaluation and process lessons National Measurement System Environment Agency Science Department MissionSupport measurement infrastructure Support EA in protecting and improving the environment BudgetAbout £60m paAbout £15m pa Number programmes FiveEight GovernancePolicy Unit with programme- level WGs Overall Programme Board with individual Programme Boards CustomersUK industry and societyEnvironment Agency policy and operations SuppliersFour National Measurement Institutes Diverse supplier base including in- house Planning cycle3 years5 years MCDA process for R&D prioritisation introduced at two public science funders

A value tree for buying a house Buying a house Costs Financial cost Benefits SpaceCharacterCloseness

A good set of criteria has to be: Complete: criteria completely define the objectives Operational: criteria are meaningful Decomposable: The criteria can be analyzed one at a time, do not depend on each other. Absent of redundancy: criteria are mutually exclusive, do not mean the same thing. Of minimum size: Decision makers cannot handle large number of criteria. (Keeney and Raiffa, 1976)

Criterion a particular perspective according to which decision alternatives may be compared, usually representing a particular interest, concern or point of view Attribute a quantitative measure of performance associated with a particular criterion according to which an alternative is to be evaluated Belton and Stewart (2002) operationalises

Buying a house Costs Financial cost Benefits SpaceCharacterCloseness Purchase price Sq footage? TfL Zone

Alternatives x attribute matrix “Most yes 700 Preferred”

Look carefully at the options: can you say anything about your choice?

Attribute Levels Criteria Scores

Eliciting intermediate values Consider the difference in value you experience if you were to move from living in a flat in in Zone 3 to living in a flat in Zone 1 Relative to this quantum of value, IF you were to move from a flat in Zone 3 to a flat in Zone 2, how much value would you get from this move? Zone 1Zone 3 70% 30% Zone 2

Bisection method “a most preferred stimulus and a least preferred stimulus are identified, and subsequently a midpoint stimulus is found that is equidistant from both extremes.” (von Winterfeldt and Edwards, 1986) Find a purchase price £xK such that the difference in value between a purchase price of £220K and £xK is equal in value to the difference between £xK and £120K £220K £120K £180K

When all values are in, make consistency checks, by asking if same value differences ‘feel’ the same.

Criteria scores are measured on different (interval) scales

Weighting 1: Swing Weights Imagine you live in a flat at its worst value on all criteria (220, Z3, no, 500 sq ft)

Swing Weight Example

Weighting 2: Tradeoff method Compare An option with the best level of “cost” and the worst level of “space” An option with the worst level of “cost” and the best level of “space” Adjust the cost of the first option until you are indifferent between the two options Option2 £220k, 500 sq ft £220k, 700 sq ft Option1 £120k, 500 sq ft £220k, 500 sq ft V cost =5 0 £180k ~

Weighting 3: Asking for direct importance judgements This method doesn’t force people to think hard about tradeoffs Psychologically not clear what such questions “plug in to” Experimental research shows that judgements elicited in this manner do not “behave like” swing and tradeoff weights Moral: this isn’t a good way to elicit criteria weights!

Calculating values of options The judged value of an option is the weighted additive value For the ith option, Where J is an index set of criterion weights

Calculating values of options

Understanding the options

Subaggregation and efficiency Options 1 – 4 are efficient; option 5 is inefficient Flat 5

Sensitivity analysis No weight on “closeness” All weight on “closeness”