1. Interactive Goal Model Analysis Applied - Systematic Procedures versus Ad hoc Analysis Jennifer Horkoff 1 Eric Yu 2 Arup Ghose 1 Department of Computer Science 1 Faculty of Information 2 jenhork@cs.utoronto.cajenhork@cs.utoronto.ca yu@ischool.utoronto.ca arup.ghose@utoronto.cayu@ischool.utoronto.ca arup.ghose@utoronto.ca University of Toronto November 10, 2010 PoEM’10

2. Goal Modeling  Used as a tool for system analysis and design in an enterprise  Captures social-driven goals which motivate design or redesign  First sub-model of Enterprise Knowledge Development (EKD) method  Used in several Requirements Engineering frameworks i* (Yu, 97) Tropos (Bresciani et al., 94) GBRAM (Antón et al., 98) KAOS (Dardenne & van Lamsweerde, 93) GRL (Liu & Yu, 03) Etc. Interactive Goal Model Analysis Applied - Horkoff, Yu, Ghose 2

3. Goal Model Analysis  Work has argued that more utility can be gained from goal models by applying systematic analysis  Many different types of analysis procedures have been introduced (metrics, model checking, simulation, planning, satisfaction propagation)  Most of the work in goal model analysis focuses on the analytical power and mechanisms of the procedures  What are the benefits of goal model analysis?  Do these benefits apply only to a systematic procedure? Or also to ad-hoc (no systematic procedure) analysis?  Focus: interactive satisfaction propagation Interactive Goal Model Analysis Applied - Horkoff, Yu, Ghose 3

4. Hypotheses: Benefits of Systematic, Interactive Goal Model Analysis  Previous work by the authors has introduced interactive, qualitative goal model analysis aimed for early enterprise analysis (CAiSE’09 Forum, PoEM’09, IJISMD)  Hypotheses concerning benefits of interactive analysis developed through application of several case studies (PoEM’09, PST’06, REFSQ’08, HICSS’07, RE’05) Analysis: aids in finding non-obvious answers to domain analysis questions Model Iteration: prompts improvements in the model Elicitation: leads to further elicitation of information in the domain Domain Knowledge: leads to a better understanding of the domain  In this work we design and administer studies to test these hypotheses Interactive Goal Model Analysis Applied - Horkoff, Yu, Ghose 4

5. Background: i* Models  We use i* as an example goal modeling framework Interactive Goal Model Analysis Applied - Horkoff, Yu, Ghose 5

6. “Real” Example: inflo Case Study Interactive Goal Model Analysis Applied - Horkoff, Yu, Ghose 6

7. Background: Interactive Satisfaction Analysis  Forward: A question/ scenario/ alternative is placed on the model and its affects are propagated “forward” through model links  Interactive: user input (human judgment) is used to decide on partial or conflicting evidence “What is the resulting value?”  Publications: CAiSE’09 Forum, PoEM’09, IJISMD  Additional procedure for “backward” analysis, allows “is this possible?” questions  Publications: istar’08, ER’10 Interactive Goal Model Analysis Applied - Horkoff, Yu, Ghose 7 Human Judgment What if…?

8. Case Study Design  One group study involving “inflo” “back-of-the-envelope” calculation and modeling tool (case = group) Four grad students, 1 professor, and 1 facilitator Three two hour modeling sessions + one hour analysis session Most of each session devoted to developing the model & discussion with analysis at the end of each session  Ten two-hour sessions with an individual and a facilitator (case = individual) Five used systematic forward and backward analysis implemented in OpenOME Five were allowed to analyze the models as they liked  Individual study design was modified midway through Divided into Round 1 and Round 2  Studies were both exploratory and confirmatory Interactive Goal Model Analysis Applied - Horkoff, Yu, Ghose 8

9. Individual Studies (Round 1)  Participants: students who had i* experience in system analysis courses or through i*-related projects  Purposive selection: wanted subjects with some i* knowledge but not much analysis experience  Training: Participants given 10 minutes of i* training (including analysis labels) Systematic participants given 15 minutes of analysis training using the tool  Model Domain: ICSE Greening models, large to medium models created by others  Analysis Questions: 12 questions provided 2 for each analysis direction (forward, backward) per model * 3 models Interactive Goal Model Analysis Applied - Horkoff, Yu, Ghose 9

10. ICSE Greening Example: Conference Experience Chair Interactive Goal Model Analysis Applied - Horkoff, Yu, Ghose 10

11. Individual Studies  Intermediate (Round 1) results: Models were too complicated Too many analysis questions Participants unfamiliar with domain Didn’t “care” about judgment decisions Made very few changes to models (too afraid to change other’s work? too intimidated to change complex models?) Interactive Goal Model Analysis Applied - Horkoff, Yu, Ghose 11

12. Individual Studies (Round 2)  Round 2 Changes (last 4/10 participants) Model Domain: Asked participants to create their own models describing student life Group case study showed that participants had trouble finding analysis questions over their own model Created Analysis Methodology to help guide the analysis  Extreme test conditions (all alternatives/targets satisfied/denied)  Analyze likely alternatives/targets  Analyze domain-driven questions Interactive Goal Model Analysis Applied - Horkoff, Yu, Ghose 12

13. Data Capture  Analysis: captured answers to analysis questions  Model Iteration: quantitative counts of model changes for each stage in the studies  Elicitation: captured lists of questions asked about the domain in each stage  Domain Knowledge: follow-up questions about experience  Recorded and analyzed other interesting qualitative findings Interactive Goal Model Analysis Applied - Horkoff, Yu, Ghose 13

14. Results

15. Analysis  Analysis: aids in finding non-obvious answers to domain analysis questions  Some participants gave explicit answers, others had difficultly producing answers  Some referred to analysis labels in the model as answers to the question  Only some participants were able to interpret analysis results in the context of the domain  Generally, difficulty in mapping the model to the domain  Conclusion: knowledge of i* and the domain may have a significant effect on the ability to apply and interpret analysis Interactive Goal Model Analysis Applied - Horkoff, Yu, Ghose 15

16. Model Iteration & Elicitation  Model Iteration: prompts improvements in the model  Elicitation: leads to further elicitation of information in the domain  Few changes, few differences between ad hoc & systematic, familiar and unfamiliar domain, forward backward Interactive Goal Model Analysis Applied - Horkoff, Yu, Ghose 16 # Model Changes# Questions Asked TreatmentPartic. Forward Questions Backward Questions Forward Questions Backward QuestionsRound Ad-hoc P15910 1 1 P40010 P551366 P72500 2 P9 050 0 Systematic P20023 1 P30020 P60351 P80022 2 P10 000 1

17. Model Iteration & Elicitation  Conflicts with previous results (PoEM’09, PST’06, etc.), Why?  Underlying theory: interactive analysis prompts users to notice differences between mental domain model and physical model Evaluation did not reveal differences between the mental and physical model, or these differences existed, but were not used to modify the model Interactive Goal Model Analysis Applied - Horkoff, Yu, Ghose 17

18. Model Iteration & Elicitation  Previous studies were conducted by i*/modeling “experts” who had commitment to the project  Conclusion: Model iteration and elicitation relies on: More extensive knowledge of syntax and analysis procedures More extensive knowledge of the domain “buy-in”/caring about a real problem Interactive Goal Model Analysis Applied - Horkoff, Yu, Ghose 18

19. Domain Knowledge  Domain Knowledge: leads to a better understanding of the domain  Follow-up question: “do you feel that you have a better understanding of the model and the domain after this exercise?”  7/10 participants said yes (mix of ad-hoc and systematic participants)  Conclusion: both ad-hoc and systematic knowledge can help improve domain knowledge Interactive Goal Model Analysis Applied - Horkoff, Yu, Ghose 19

20. Additional Findings  Promoted Discussion in Group Setting: human judgment caused discussion among participants Example: “what is meant by Flexibility?”  Model Interpretation Consistency i* syntax leaves room for interpretation Results shows a variety of interpretations when propagating analysis labels with ad-hoc analysis Conclusion: systematic analysis provokes a more consistent interpretation of the model  Coverage of Model Analysis Results show significant differences in the coverage of analysis across the model with systematic vs. ad-hoc analysis  Model Completeness and Analysis Analysis may not be useful until the model is sufficiently complete Some participants noticed incompleteness in the model(s) after applying analysis Interactive Goal Model Analysis Applied - Horkoff, Yu, Ghose 20

21. Conclusions and Future Work  Designed and administered studies to test perceived benefits of interactive goal model analysis  Initial Hypotheses: Analysis, Model Iteration, Elicitation, Domain Knowledge Benefits dependent on:  Knowledge of i* and i* evaluation  Presence of an experienced facilitator  Domain expertise/buy-in  The presence of a real motivating problem  Discovered benefits: Interpretation Consistency, Coverage of Model Analysis, Model Completeness  Several threats to validity (construct, internal, external, reliability) described in the paper  Future Work More realistic action-research type studies Better tool support – make the tool the expert? Interactive Goal Model Analysis Applied - Horkoff, Yu, Ghose 21

22. Thank you Questions?  jenhork@cs.utoronto.ca jenhork@cs.utoronto.ca www.cs.utoronto.ca/~jenhork  yu@ischool.utoronto.ca yu@ischool.utoronto.ca www.cs.utoronto.ca/~eric  arup.ghose@utoronto.ca arup.ghose@utoronto.ca  OpenOME: https://se.cs.toronto.edu/trac/ome Interactive Goal Model Analysis Applied - Horkoff, Yu, Ghose 22

23. Outline  Goal Modeling  Goal Model Analysis  Hypotheses: Benefits of Systematic, Interactive Goal Model Analysis  Background: i* Syntax  Background: Interactive Goal Model Analysis  Case Study Design Group study Individual Studies  Results  Threats to Validity  Conclusions and Future Work Interactive Goal Model Analysis Applied - Horkoff, Yu, Ghose 23

24. Goal Model Analysis  Work has argued that more utility can be gained from goal models by applying systematic analysis  Many different types of analysis procedures have been introduced Metrics (Franch, 06) (Kaiya, 02) Model checking (Fuxman et al., 03) (Giorgini et al., 04) Simulation (Gans et al., 03) (Wang & Lesperance, 01) Planning (Bryl et al., 06) (Asnar et al., 07) Satisfaction Propagation (Chung et al., 00) (Giorgini et al., 05)  Most of this work focuses on the analytical power and mechanisms of the procedures  What are the benefits of goal model analysis?  Do these benefits apply only to a systematic procedure? Or also to ad-hoc (no systematic procedure) analysis? Interactive Goal Model Analysis Applied - Horkoff, Yu, Ghose 24

25. inflo (Group) Case Study  inflo: “back-of-the-envelope” calculation and modeling tool Support informed debate over issues like carbon footprint calculations  Four grad students, 1 professor, and 1 facilitator  Three two hour modeling sessions + one hour analysis session  Most of each session devoted to developing the model & discussion  Used systematic model analysis at the end of each session Interactive Goal Model Analysis Applied - Horkoff, Yu, Ghose 25

26. Individual Studies (Round 1)  Analysis Questions: 12 questions provided 4 per model (3 models) 2 for each analysis direction (forward, backward) per model  Example (forward): “If every task of the Sustainability Chair and Local Chair is performed, will goals related to sustainability be sufficiently satisfied?”  Example (backward): “What must be done in order to Encourage informal and spontaneous introductions and Make conference participation fun?” Interactive Goal Model Analysis Applied - Horkoff, Yu, Ghose 26

27. Analysis Methodology  1. Alternative Effects (Forward Analysis) a) Implement as much as possible: all leaves are satisfied b) Implement as little as possible: all leaves are denied c) Reasonable Implementation Alternatives: Evaluate likely alternatives  2. Achievement Possibilities (Backward Analysis) a) Maximum targets: all roots must be fully satisfied. Is this possible? How? b) Minimum targets: lowest permissible values for the roots. Is this possible? How? c) Iteration over minimum targets: try gradually increasing the targets in order to find maximum targets which still allow a solution.  3. Domain-Driven Analysis (Mixed) a) Use the model to answer interesting domain-driven questions Interactive Goal Model Analysis Applied - Horkoff, Yu, Ghose 27

28. Threats to Validity  Construct Validity Model changes may not be beneficial  Internal Validity Presence of facilitator Think-aloud protocol Choice of model domain  External Validity Used students Used i* - generalize to other goal model frameworks?  Reliability Facilitator was i* & evaluation expert Interactive Goal Model Analysis Applied - Horkoff, Yu, Ghose 28