Public Procurement Evaluation by Evidence-based Multiple Criteria Decision Analysis — From conventional scoring to systematic profiling Professor Jian-Bo Yang Director of Decision and Cognitive Sciences Research Centre Manchester Business School (MBS) The University of Manchester Tel: (Ext: 63427), (O2) Web: Public Procurement Evaluation and MCDA by J B Yang of MBS

Outline of This Presentation  Public procurement evaluation and Multiple Criteria Decision Analysis (MCDA)  Typical MCDA models –Decision matrix – Scoring –Pairwise comparison decision matrix - Rating –Belief decision matrix – Profiling or grading  Evaluation aggregation based on scores –Linear aggregation or weighted sum –Reference point approach using nonlinear distance measures  Evaluation aggregation based on beliefs (IDS) –Evidence collection, mapping and grading –Evidential reasoning for generating bidder profiles –Expected utility as score for ranking –Sensitivity analysis for testing the robustness of ranking –Communication based on both bidder profiles and scores Public Procurement Evaluation and MCDA by J B Yang of MBS

Public Procurement Evaluation and MCDA Procurement evaluation criteria (weight)  Contractor’s organisation (0.1)  Financial considerations (0.3)  Management resources (0.2)  Past experience (0.2)  Past performance (0.2) –Failure of a contract (0.25) –Overruns: time (0.25) –Overruns: cost (0.25) –Actual quality achieved (0.25) –…… Public Procurement Evaluation and MCDA by J B Yang of MBS

Multiple Criteria Decision Analysis Under uncertainty – Summary of main features  A hierarchy of performance or risk criteria  Quantitative and qualitative criteria  Precise data and uncertain numbers  Subjective judgements with uncertainty  Possible absence of data  Non-commensurability among criteria  Conflict among criteria  Ranking may not be precise Public Procurement Evaluation and MCDA by J B Yang of MBS

Modelling for Procurement Evaluation  Transparency and fairness via knowledge sharing  Objectivity via data collection and management  Systematic analysis via information aggregation  Panoramic view of bidder profile  Sensitivity analysis for uncertainty clarification  Consistency in evaluation  Simulation for improvement and feedback  Communication with evidence (original / aggregated) Public Procurement Evaluation and MCDA by J B Yang of MBS

MCDA Models for Procurement Evaluation Scoring based model – decision matrix MCDA problem with numbers: Decision maker is faced with assessing and ranking several alternatives with all attributes being considered simultaneously, with no attribute being absolutely more important than others. The problem can be represented as follows Criterion 1Criterion 2…Criterion m Bidder 1y 11 y 12 …y1my1m Bidder 2y 21 y 22 …y1my1m …………… Bidder lyl1yl1 yl2yl2 …y lm Decision Matrix (Table) How should the assessment and ranking be made ? Public Procurement Evaluation and MCDA by J B Yang of MBS

Scoring-based Decision Matrix – Job evaluation Decision Matrix for Job Evaluation CriteriaJob offer 1Job offer 2Job offer 3 Salary£32,500£28,500£26,000 Quality of life Average (50%) Poor (25%) Good (75%) Interest of work Poor (25%) Good (75%) Average (50%) Location Poor (25%) Good (75%) Public Procurement Evaluation and MCDA by J B Yang of MBS

Pairwise Comparison Matrix Compare each pair of job offers on a criterion Pairwise Comparison Matrix for Job Evaluation Quality of life Job offer 1Job offer 2Job offer 3 Job offer Job offer Job offer 3241 Job 1 is judged (rated) twice as good as Job 2 in terms of “Quality of life” (Interval comparison ?) Public Procurement Evaluation and MCDA by J B Yang of MBS

Evidence-based Belief Decision Matrix – Take into account judgmental information Criterion 1Criterion 2…Criterion m Alternative 1y 11 S 12 …S1mS1m Alternative 2y 21 S 22 …S1mS1m …………… Alternative lyl1yl1 Sl2Sl2 …S lm Belief Decision Matrix MCDA problem with both numbers and judgements: Belief distribution : S ij ={(H 1, β ij1 ), (H 2, β ij2 ), ……, (H N, β ijN )} Public Procurement Evaluation and MCDA by J B Yang of MBS

House Criteria House 1 in Altrincham House 2 in Heaton House 3 in Mercy House 4 in Didsbury Location {(G, 0.5), (E, 0.5)} {(G, 0.5)} {(A, 0.2), (G, 0.8)} {(G, 0.2), (E, 0.8)} Distance (mile) Asking Price (£) 113,000110,000118,000150,000 Attractive- ness {(P, 0.05), (G, 0.35), (E, 0.60)} {(A, 0.4), (G, 0.6)} {(G, 0.3), (E, 0.7)} {(G, 0.6), (E, 0.4)} Belief Decision Matrix Assessment based on evidence collected Belief is generated from the assessment of evidence Public Procurement Evaluation and MCDA by J B Yang of MBS

Belief Decision Matrix Assessment based on evidence collected and mapped Assessing the Location of House 1 in Altrincham using the collected evidence against the agreed assessment standards (mapping) Public Procurement Evaluation and MCDA by J B Yang of MBS

Procurement Evaluation Aggregation – Weighted sum or Multiple Attribute Value Function General form of an additive (linear) value function is given by: Conditions for use of Additive MAVF: 1.Satisfaction of preferential independence among any groups of attributes. This is only a necessary condition. 2.Satisfaction of the corresponding trade-off, or Thomsen condition. 3.Interval scale property for constructing marginal value function. 4.Weights of attributes need to be assessed as scaling constants (trade-offs), or swing weights, not necessarily relative importance. 5.Linear & complete compensation among criteria without any limit. Public Procurement Evaluation and MCDA by J B Yang of MBS

Chinese Restaurant Menu : Combination of soup and main dish Attribute 1: Choose soup Attribute 2: Choose main dish Are you preferentially independent when choosing soup and main dish? MCDA – Value Measurement Theory – Preferential independence violation example SoupValue scoreMain dishValue score Mixed veg & bean curd 8Bean curd10 Egg and tomato 3 Pork with Spring Onions 7 If you are preferentially independent in choosing soup and main dish, you would ask for a main dish without considering what soup you have taken. However, is this the case for you? Would you really choose both Mixed veg & bean curd as soup and Bean curd as main dish? Public Procurement Evaluation and MCDA by J B Yang of MBS

Limitation or Bias of Additive MAVF Efficient frontier: A, B, D, E, F, G Efficient convex hull: A, E, G Additive MAVF cannot find B or F as the most preferred solution ωsvs+ωpvp=vωsvs+ωpvp=v Public Procurement Evaluation and MCDA by J B Yang of MBS

Distance-based Aggregation Ideal point models (minimax distance) Ideal point models: Set an ideal reference point and find an alternative closest to the ideal point in certain distance measure. Reference point Ideal point Set criterion weights Public Procurement Evaluation and MCDA by J B Yang of MBS

Evidential Reasoning MCDA Assessment distribution by a belief structure ER Example 1 : A qualitative assessment that the quality y q of a bidder A is assessed to be “ Good ” or “ Excellent ” by an equal number of assessors, with no assessment below “ Average ”, can be described by the following distribution S(y q (A)) ={(Bad, 0), (Average, 0), (Good, 0.5), (Excellent, 0.5)} which is termed as a belief distribution of assessment, with “ Bad ”, “ Average ”, “ Good ” and “ Excellent ” defined as “assessment grade” and 0 (0%) and 0.5 (50%) as “belief degree” (frequency to which “ Good” or “ Excellent ” is ticked by the assessors). The above distribution shows the quality profile of the bidder. Public Procurement Evaluation and MCDA by J B Yang of MBS

Bidder 1 Score 76% 6.1 Give examples of STRATEGIC Partnering, Alliances and Collaborative Working Bidders ({Best, 28%}, {Good, 51%}, {Average,17%}, {Poor, 4%}, {Worst, 0%} Bidder 2 Score 76% ({Best, 46%}, {Good, 29%}, {Average,15%}, {Poor, 3%}, {Worst, 7%} Traditionally, only scores are used ER uses both scores and belief degrees Evidential Reasoning MCDA Assessment Using ER – What’s different Public Procurement Evaluation and MCDA by J B Yang of MBS

Assessment Using a Decision Support System – Intelligent Decision System (IDS)  IDS is supported by the Evidential Reasoning (ER) approach  ER has been developed over a period of over 15 years  ER results from multi-discipline research - Decision Sciences - Artificial Intelligence - Statistical Analysis - Fuzzy Sets  ER addresses subjectivity and uncertainties  ER can handle heterogeneous information  ER guarantees to generate rational results  ER is gaining popularity in both academia and industry Public Procurement Evaluation and MCDA by J B Yang of MBS

Assessment Using IDS – Advantages  Structured and natural No modification needed in IDS for procurement evaluation modelling  Flexible in modelling Model can be modify, attributes changed, added and deleted easily  Improved consistency and efficiency Through knowledge management and using an systematic evidence mapping process Public Procurement Evaluation and MCDA by J B Yang of MBS

Assessment Using IDS – Advantages  No unnecessary assumption No need to use scores for subjective judgement No need to assume missing data  Transparent Candidates compared on any attribute at any level Weaknesses and strengths of each candidate identified  Rational, convincing and informative Examine impact of changes in any factor on decisions easily so that the decisions are made in a more rational, convincing and informative way Public Procurement Evaluation and MCDA by J B Yang of MBS

Assessment Using IDS – Modelling Build an evaluation criteria hierarchy Public Procurement Evaluation and MCDA by J B Yang of MBS

Assessment Using IDS – Modelling Define qualitative attribute : Number of grades can be changed Public Procurement Evaluation and MCDA by J B Yang of MBS

Assessment Using IDS – Modelling Define grades to assess a qualitative attribute: Preference value of grades can be changed Wording of grades can be changed Public Procurement Evaluation and MCDA by J B Yang of MBS

Assessment Using IDS – Modelling Define grade standard to assess a qualitative attribute: This can be used as guidelines to help improve consistency in assessment Public Procurement Evaluation and MCDA by J B Yang of MBS

Assessment Using IDS – Modelling Define quantitative attribute: Public Procurement Evaluation and MCDA by J B Yang of MBS

Assessment Using IDS – Modelling Assign weights Drag and drop to change weight Or type weight here Public Procurement Evaluation and MCDA by J B Yang of MBS

Assessment Using IDS – Review Document Evidence classified and recorded Public Procurement Evaluation and MCDA by J B Yang of MBS

Assessment Using IDS – Mange Knowledge Evidence examined and comments provided Public Procurement Evaluation and MCDA by J B Yang of MBS

Assessment Using IDS – Make assessment Evidence mapped and belief degrees assigned to grades Grade guidelines entered earlier Optional More than one grades may be selected Public Procurement Evaluation and MCDA by J B Yang of MBS

Assessment Using IDS – View Results Performance distribution (profile) of bidder generated Unknown element due to lack of data in Economic Test & Interview Can be any attribute in the hierarchy Public Procurement Evaluation and MCDA by J B Yang of MBS

Assessment Using IDS – View Results Performance scores – when there is missing data Public Procurement Evaluation and MCDA by J B Yang of MBS

Assessment Using IDS – View Results Compare candidates on multiple criteria – by scores Public Procurement Evaluation and MCDA by J B Yang of MBS

Assessment Using IDS – View Results Compare candidates – by performance profile Public Procurement Evaluation and MCDA by J B Yang of MBS

Other Applications - Siemens UK Supplier pre-qualification assessment Public Procurement Evaluation and MCDA by J B Yang of MBS

Other Applications  Product design and evaluation car, motorcycle, ship, aircraft, computer, …  Safety and risk assessment  Quality management  Supply chain management  Environmental management  Financial services and investment  Customer satisfaction survey  Web based survey  Data collection only – remote or onsite audit Public Procurement Evaluation and MCDA by J B Yang of MBS

Summary and Conclusions  Public procurement evaluation and multiple criteria decision analysis (MCDA)  Typical MCDA procurement evaluation models –Decision matrix – Scoring –Pairwise comparison decision matrix - Rating –Belief decision matrix – Profiling or grading  Evaluation aggregation based on scores –Linear aggregation or weighted sum –Reference point approach using distance measures  Evaluation aggregation based on beliefs (IDS) –Evidence collection and mapping or grading –Evidential reasoning for generating bidder profile –Expected utility as score for ranking –Sensitivity analysis for testing the robustness of ranking –Communication based on both bidder profile and score Public Procurement Evaluation and MCDA by J B Yang of MBS