1. Applying the Churchman/Ackoff Value Estimation Procedure to Spatial Modeling Susan L. Ose MGIS Capstone Presentation Penn State University - World Campus 17 June 2008

2. GIS Models - A Refresher GIS models provide –Decision Support –Prediction –Cost assessment, etc. Weighting values –Emphasize/De-emphasize impact of inputs –Model adjusted via tweaking weights How does an analyst determine weights? Figure source: Joseph K. Berry, University of Denver

3. Estimating Weights - Gathering Expert Opinions via Delphi Process ? ? Compile Results Experts Questionnaire Discussion Anonymous Questionnaires Statistical aggregation Controlled feedback Avoids groupthink Consensus determines number of rounds

4. Questionnaire Format The ideal questionnaire: –Guides the user through the weight estimation process –Focuses on expert knowledge - not statistics –Emphasizes relative not numeric values Software module –Stores and compiles results –Allows easy distribution to expert group –Can be used in online meeting

5. Churchman/Ackoff Procedure Focuses on relative value of layers –Compares layer against combination of other layers –Uses recursive procedures –Breaks down larger groups of layers for easier assessment Recommended for group decision making Project adapted method to browser-based software module

6. Churchman/Ackoff Procedure Step 1: Rank Layers Layers are dragged and dropped into desired order

7. Churchman/Ackoff Procedure Step 2: Assign Initial Weights Layers divided subdivided into equal groups of no more than four. Control layer randomly chosen, assigned value of 50, added to each group Expert inputs initial estimate of weight Group 1 Group 2

8. Churchman/Ackoff Procedure Step 3: Judge Importance Expert chooses one of the three conditions and clicks on it. Algorithm adjusts values accordingly.

9. Churchman/Ackoff Procedure : Results Screen

10. Procedure Test Case Potential for rain-fed agriculture in Liberia taking into account cost to market Expert panel consisted of MDA Federal, Inc. employees experienced in modeling Used two Delphi rounds –Compiled statistics –Ran model using average, high, and low values

11. Model Diagram

12. First Round Results Reference layers received lowest scores Proximity to water and land layers received highest scores Model output using average score and 8 highest weighted layers consistent with results obtained using different methodology Discussion focused on experience with procedure Panel agreed on top 7 layers to include in final model

13. Second Round Results Results reflected post-first round discussion –Access to water considered key, water layers grouped near top of ranking –Land cover shows potential land to be transformed to agriculture, thus higher score than previous Higher variance in scores than in first round –Probably due to score distribution among less layers –First round lower variance may not have happened if expert could discard layers Model more definitive around water features

14. Round Comparison

15. Model Results - Average 2nd round1st round

16. Does the method work? Model result viable Considering one vs. many values more difficult than considering one vs. one (pairwise method) –Deliberately designed that way to challenge one's opinions –May frustrate participants - "weighting fatigue" Random groupings confused some participants –Difficult to recall original ranking, weight –However randomness focuses judgment on subgroup Can assist individual in examining own conclusions

17. What happens when someone joins the group later? One expert did not attend discussions, used tool with minimal guidance First round results changed slightly Average Z-score highest of the group

18. Second Round with Additional Person Pronounced difference in results Greater difference in layer order in mid to lower layers Average Z-score still highest of the group Demonstrates importance of group discussion Z-Scores New expertExpert 1Expert 2Expert 3Expert 4Expert 5 First Round1.210.450.660.670.650.88 Second Round1.200.460.830.880.550.77

19. Further Development/Study Improve the software tool –Rework the math algorithm that calculates the weights –Offer opportunity to discard layers –Provide comparison of start/finish results –Provide progress bar Application of the method –Categorical values within layers need to be rated first –Orientation meetings important Conduct an experiment using both this method and a pairwise approach and compare results

20. Acknowledgements Dr. Gregory Koeln, President, MDA Federal Inc. for sponsoring this project Michael Schreiber, Webmaster, MDA Federal Inc. for transforming the procedure from paper to software Dr. Todd Bacastow and Dr. Douglas Miller, Penn State World Campus, for their advice and guidance Dr. Douglas Way, MDA Federal Inc. for his invaluable guidance David Cunningham, Dr. Anna Oldak, Dr. François Smith, and Dr. Andrew Ralowicz, MDA Federal Inc. for their expertise Gregory A. Ose, for his unfailing support