Analyzing Goal Models – Different Approaches and How to Choose Among Them Jennifer Horkoff 1 Eric Yu 2 1 Department of Computer Science 2 Faculty of Information University of Toronto, Canada SAC’11 RE Track

Goal-Oriented Requirements Engineering (GORE)  GORE has received much attention in RE research as a means of: Understanding the motivations for system requirements Helping to ensure that the right system is built  Generally, GORE frameworks allow for: Representation of stakeholder goals Goals may be assigned to an agent (stakeholder or system) Goals may have relationships to other goals, often describing achievement  Several goal modeling frameworks KAOS, GBRAM, AGORA, NFR, i*, Tropos, GRL, … 2 Analyzing Goal Models, Horkoff & Yu

Goal-Oriented Requirements Engineering (GORE) 3 Analyzing Goal Models, Horkoff & Yu Example: Counseling Organization i* Model (Horkoff & Yu, 2009) We can analyze the contents of goal models systematically…

Analyzing Goal Models, Horkoff & Yu 4 Example: Qualitative, Interactive Forward Satisfaction Analysis of Goal-Oriented Models Horkoff, Yu: Evaluating Goal Achievement in Enterprise Modeling What is the effect of using a Cybercafe/ Portal/ Chat Room?”

Models and Analysis become Complex Analyzing Goal Models, Horkoff & Yu 5

Goal Model Analysis  Many different analysis techniques for goal models have been introduced: Propagate satisfaction values through the model Measure metrics over the model Apply planning techniques Run simulations Perform checks over models  Abundance of approaches is encouraging from a research perspective, but…  From a user or practitioner perspective can be confusing What are the differences? When would I use one and not another?  Limits adoption 6 Analyzing Goal Models, Horkoff & Yu

Motivating Questions  Survey of methods What methods are available? What types of analysis questions can these methods answer? What types of goal modeling constructs do the procedures support? What information is needed in order to use the methods?  Analysis benefits What are some of the potential benefits of goal model analysis in the requirements process?  Mapping and Selection What available methods can be applied to achieve which kinds of usage objectives? How can we use this information to advise on selection? 7 Analyzing Goal Models, Horkoff & Yu

Survey of Goal Model Analysis Procedures Analyzing Goal Models, Horkoff & Yu 8 ApproachAnalysis ResultsAdditional Notation Supported PaperSatisf Forwds Satisf Backwds Human Interv MetricsPlan- ning Simu- lation Model Check QualQuantBinaryDepend -encies Soft- goals Contribut -ion Links Chung et al. [9] YNYNNNNYNYNYY Giorgini et al. [21] YNNNNNNYYYNMY Giorgini et al.[22] YYNNNNNYNYNMY Giorgini et al. [23] YYNNNNNYNYMYY Horkoff & Yu [26] YNYNNNNYNYYYY Maiden et al. [33] YNYNNNNNNYYYY Amyot et al. [1] YNNNNNNYYYYYY Asnar & Giorgini [3] YYNNNNNYMYNMY Letier & vLams. [31] YYNNNNNNYYMNN Horkoff & Yu [27] YYYNNNNYNYYYY Wang et al. [35] YYNNNNNNNYNMY Bryl et al. [6] NNNYYNNNYYYNN Bryl et al. [7] NNYYYNYMYYYNN Asnar et al. [4] YYYYYNNYNYYMY Gans et al [18] NNYNNYYNNYYNN Wang & Lesper. [34] NNNNNYNNNYNNN Gans et al. [16] [18] NNYNYYYNYYYYY Gans et al. [17] NNNYNYMNYYYNN Fuxman et al. [14] [15] NNMNNNYNNYYYN Giorgini et al.[20] NNNNNNYNNYYNN Bryl et al.[8] NNNNYNYNNYYNN Procedures Summary Dimensions

A Survey of GORE Analysis Techniques  Summarize results over the following points: Algorithm Approach  Satisfaction Forwards, Backwards, Human Intervention, Metrics, Planning, Simulation, and Model Checking Format of analysis results  Qualitative ( ), quantitative (0.37), binary (T/F) Goal-oriented concepts supported (beyond AND/OR)  Dependencies, softgoals, contribution links Additional information required beyond typical goal model constructs  e.g., priority, probabilities, events, delegations, and trust. 9 Analyzing Goal Models, Horkoff & Yu ApproachAnalysis ResultsAdditional Notation Supported PaperSatisf Forwds Satisf Backwds Human Interv MetricsPlan- ning Simu- lation Model Check QualQuantBinaryDepend -encies Soft- goals Contribut -ion Links Smith et al. [1]YNYNNNNYNYNYY

Satisfaction Analysis  Example Analysis Questions: What is the effect of this alternative? Can this goal be satisfied?  Evaluates the satisfaction or denial of goals given a functional or design alternative  Values are propagated forward or backward throughout the model  Qualitative or quantitative approaches  Techniques take different approaches to resolving multiple values for incoming goals: Adding evidence, combine using probabilistic rules, separate evidence, fixed rules, human judgment 10 Analyzing Goal Models, Horkoff & Yu

Satisfaction Analysis 11 Analyzing Goal Models, Horkoff & Yu ApproachAnalysis ResultsAdditional Notation Supported PaperSatisf Forwds Satisf Backwds Human Interv MetricsPlan- ning Simu- lation Model Check QualQuantBinaryDepend -encies Soft- goals Contribut -ion Links Chung et al. [9] YNYNNNNYNYNYY Giorgini et al. [21] YNNNNNNYYYNMY Giorgini et al.[22] YYNNNNNYNYNMY Giorgini et al. [23] YYNNNNNYNYMYY Horkoff & Yu [26] YNYNNNNYNYYYY Maiden et al. [33] YNYNNNNNNYYYY Amyot et al. [1] YNNNNNNYYYYYY Asnar & Giorgini [3] YYNNNNNYMYNMY Letier & vLams. [31] YYNNNNNNYYMNN Horkoff & Yu [27] YYYNNNNYNYYYY Wang et al. [35] YYNNNNNNNYNMY

Metrics  Example Analysis Questions: How secure is the system represented by the model? How risky is a particular alternative for a stakeholder?  Structural properties of the model and construct classifications are used to calculate metrics Example: counts of dependency classifications (instance, model, duplicate, hidden) in a Strategic Dependency (SD)  Metrics often represent non-functional requirements Examples: predictability, security, privacy, accuracy, etc.  They can also represent model properties: Examples: completeness, consistency and correctness  Metrics can be local or global 12 Analyzing Goal Models, Horkoff & Yu

Metrics 13 Analyzing Goal Models, Horkoff & Yu ApproachAnalysis ResultsAdditional Notation Supported PaperSatisf Forwds Satisf Backwds Human Interv MetricsPlan- ning Simu- lation Model Check QualQuantBinaryDepend -encies Soft- goals Contribut -ion Links Franch & Maiden [12] NNNYNNNNYNYYN Franch et al. [13] NNNYNNNMYNYYN Franch [11]NNYYNNNYYNYYY Kaiya et al. [30] NNNYNNNNYYNNM

Planning  Example Analysis Questions: What actions must be taken to satisfy goals? What are the best plans according to certain criteria?  Work has applied AI-type planning to find satisfactory sequences of actions in models  Requires definition of axioms that express possible goal decompositions and delegations Expresses the capabilities of actors in a model  A planner finds a delegation of goals to actors which fulfills model goals  Plans are evaluated by some criteria 14 Analyzing Goal Models, Horkoff & Yu

Planning 15 Analyzing Goal Models, Horkoff & Yu ApproachAnalysis ResultsAdditional Notation Supported PaperSatisf Forwds Satisf Backwds Human Interv MetricsPlan- ning Simu- lation Model Check QualQuantBinaryDepend -encies Soft- goals Contribut -ion Links Bryl et al. [6]NNNYYNNNYYYNN Bryl et al. [7]NNYYYNYMYYYNN Asnar et al. [4]YYYYYNNYNYYMY

Simulation  Example Analysis Questions: What happens when an alternative is selected? Are there unexpected properties in a simulation?  Adds temporal information including pre- and post- conditions to models Translated to ConGolog (situation calculus) programs for simulation  Extensions simulate confidence, trust and distrust 16 Analyzing Goal Models, Horkoff & Yu

Simulation 17 Analyzing Goal Models, Horkoff & Yu ApproachAnalysis ResultsAdditional Notation Supported PaperSatisf Forwds Satisf Backwds Human Interv MetricsPlan- ning Simu- lation Model Check QualQuantBinaryDepend -encies Soft- goals Contribut -ion Links Gans et al [18]NNYNNYYNNYYNN Wang & Lesper. [34] NNNNNYNNNYNNN Gans et al. [16] [18] NNYNYYYNYYYYY Gans et al. [17] NNNYNYMNYYYNN

Model Checking  Example Analysis Questions: Is it possible to achieve a particular goal? Is the model consistent?  Models are expanded/converted to a temporal formalism Includes expressions of creation, fulfillment and invariant properties  First order temporal logic statements are used to represent desired constraints  Model checker is used to validate properties and check for consistency  Further work adds in checks for security and trust 18 Analyzing Goal Models, Horkoff & Yu

Model Checking 19 Analyzing Goal Models, Horkoff & Yu ApproachAnalysis ResultsAdditional Notation Supported PaperSatisf Forwds Satisf Backwds Human Interv MetricsPlan- ning Simu- lation Model Check QualQuantBinaryDepend -encies Soft- goals Contribut -ion Links Fuxman et al. [14] [15] NNMNNNYNNYYYN Giorgini et al.[20] NNNNNNYNNYYNN Bryl et al.[8]NNNNYNYNNYYNN

Tabular Summary Analyzing Goal Models, Horkoff & Yu 20 ApproachAnalysis ResultsAdditional Notation Supported PaperSatisf Forwds Satisf Backwds Human Interv MetricsPlan- ning Simu- lation Model Check QualQuantBinaryDepend -encies Soft- goals Contribut -ion Links Chung et al. [9] YNYNNNNYNYNYY Giorgini et al. [21] YNNNNNNYYYNMY Giorgini et al.[22] YYNNNNNYNYNMY Giorgini et al. [23] YYNNNNNYNYMYY Horkoff & Yu [26] YNYNNNNYNYYYY Maiden et al. [33] YNYNNNNNNYYYY Amyot et al. [1] YNNNNNNYYYYYY Asnar & Giorgini [3] YYNNNNNYMYNMY Letier & vLams. [31] YYNNNNNNYYMNN Horkoff & Yu [27] YYYNNNNYNYYYY Wang et al. [35] YYNNNNNNNYNMY Bryl et al. [6] NNNYYNNNYYYNN Bryl et al. [7] NNYYYNYMYYYNN Asnar et al. [4] YYYYYNNYNYYMY Gans et al [18] NNYNNYYNNYYNN Wang & Lesper. [34] NNNNNYNNNYNNN Gans et al. [16] [18] NNYNYYYNYYYYY Gans et al. [17] NNNYNYMNYYYNN Fuxman et al. [14] [15] NNMNNNYNNYYYN Giorgini et al.[20] NNNNNNYNNYYNN Bryl et al.[8] NNNNYNYNNYYNN

Information Required by each Procedure Additional InformationRequired by 1Goal CostSatisfaction Analysis: [23][4][22][3], Planning: [6] 2RiskSatisfaction Analysis: [3], Planning: [4] 3Textual ArgumentsSatisfaction Analysis:[33], Metrics, Model Checking: [30] 4Probabilistic InformationSatisfaction Analysis: [23] [31] 5Events and TreatmentsSatisfaction Analysis: [3] 6Importance/PrioritySatisfaction Analysis: [1], Metrics: [13] [1], Simulation: [34] 7Actor CapabilitiesPlanning: [6] [7] [4], Model Checking: [8] 8(Pre/Post) Conditions/ Temporal Information Simulation: [34] [18] [18] [16] [17], Model Checking: [15] [14] 9Delegation/OwnershipModel Checking: [19] [8] 10TrustPlanning: [4], Simulation: [17], Model Checking: [20][8] 11Speech ActsSimulation: [17] 12Confidence and DistrustSimulation: [17] 13PreferencesModel Checking: [30] 14CardinalitiesSimulation:[34], Model Checking: [14] Analyzing Goal Models, Horkoff & Yu 21

Goal Model Analysis Objectives  Using capabilities of techniques in our survey, as well as our own experience in modeling and analysis, we list categories of objectives for goal model analysis List is likely not complete  Objective Categories (goal model analysis can help…): Understand the domain Communicate Improve the model Make scoping decisions Prompt requirements elicitation Improve requirements Design a system Analyzing Goal Models, Horkoff & Yu 22

Mapping Procedures to Objectives  We have made suggestions concerning what procedures may map to what objectives Each mapping can be considered as a hypothesis  We have included guiding questions with each objective to help motivate the mapping and guide users  Example mappings:  Downloadable interactive mapping table: Analyzing Goal Models, Horkoff & Yu 23 CategoryGuidelinesRecommended Procedures Domain Understanding QU1. Does the domain contain a high degree of social interaction, have many stakeholders with differing goals, or involve many interacting systems? Yes. Try: Agent Approaches: i*/GRL Satisfaction Analysis ([1][26][27][33]) i* Metrics ([11][12][13]) Tropos Metrics, Planning, or Model Checking ([4][6][7][8][14][15][19]) SNET ([16][17][18]) Requirements Improvement QR1. Are you working with a system where safety/security/ privacy/risks or other specific properties are critical considerations? Yes. Try: Analysis over Specific Constructs or Metric Approaches: KAOS ([31]) i* Metrics ([11][12][13]) AGORA ([30]) Tropos Risk, Trust, and Security ([3][4] [8][19]) SNET Trust ([17])

Analyzing Goal Models, Horkoff & Yu 24 CategoryGuidelinesRecommended Procedures Domain Understanding QU1. Does the domain contain a high degree of social interaction, have many stakeholders with differing goals, or involve many interacting systems? Yes. Try: Agent Approaches: i*/GRL Satisfaction Analysis ([1][26][27][33]) i* Metrics ([11][12][13]) Tropos Metrics, Planning, or Model Checking ([4][6][7][8][14][15][19]) SNET([16][17][18]) QU2. Do you need to understand details of the system at this point? Do you have access to detailed information such as cost, probabilities, and conditions? Can you express necessary or desired domain properties? Yes. Try: Quantitative or Detailed Information: Tropos Probabilistic Satisfaction Analysis ([3][21][22][23]) KAOS Satisfaction Analysis ([31]) GRL Quant. Analysis ([1]) i* Quant. Metrics ([11][12][13]) Tropos Planning ([4][6][7][8]) Tropos Modeling Checking ([8][14][15][19]) SNET([16][17][18][18]) i* Simulation([34]), or Model Checking: Tropos ([8][14][15][19]) SNET([16][18]) CommunicationQC1. Do you need to communicate with stakeholders? Validate requirements in the model? Justify recommendations? Yes. Try: Forward Satisfaction Approaches: NFR([9]) Tropos([3][21][22][23]) KAOS([31]) i*([26][33]) GRL([1]) Model Improvement QM1. Are you confident in the accuracy, structure, and completeness of domain knowledge and models? No. Try: Interactive Approaches: NFR([9]) i*([26][27][33]) Tropos([4][7]) SNET([16][18]) i* Metrics([11]) QM2. Would you like to verify critical properties over the model?Yes. Try: Model Checking: Tropos([8][14][15][19]) SNET([16][18]) ScopingQS1. Do you need to determine system scope?Yes. Try: Agent Approaches: i*/GRL Satisfaction Analysis ([1][26] [27][33]) i* Metrics ([11][12][13]) Tropos Metrics, Planning, or Model Checking ([4][6][7][8][14][15][19]) SNET ([16][18]) Requirements Elicitation QE1. Do you need to find more high-level requirements? Are you looking for ways to prompt further elicitation? Yes. Try: Interactive Approaches: NFR([9]) i*([27][27][33]) Tropos([4][7]) SNET([16][18]) i* Metrics([11]) QE2. Do you need to find detailed system requirements?Yes. Try: Quantitative or Detailed Information: Tropos Probabalistic Satisfaction Analysis ([3][21][22][23]) KAOS Satisfaction Analysis ([31]) GRL Quant. Analysis ([1]) i* Quant. Metrics ([11][12][13]) Tropos Planning ([4][6][7][8]) Tropos Modeling Checking ([8][14][15][19]) SNET([16][17][18][18]) i* Simulation([34]) QE3. Do you need to consider non-functional requirements difficult to quantify?Yes. Try: Approaches supporting softgoals or contributions: NFR([9]) i* Satisfaction Analysis ([26][27][33]) Tropos Satisfaction Analysis ([3][21][22][23]) Tropos Model Checking([14][15]) GRL([1]) i* Metrics([11][12][13]) SNET([16][17][18]) QE4. Do you need to capture domain assumptions?Yes. Try: Approaches using Satisfaction Arguments: i* Satisfaction Arguments [33] Requirements Improvement QR1. Are you working with a system where safety/security/ privacy/risks or other specific properties are critical considerations? Yes. Try: Analysis over Specific Constructs or Metric Approaches: KAOS([31]) i* Metrics([11][12][13]) AGORA([30]) Tropos Risk, Trust, and Security([3][4] [8][19]) SNET Trust([17]) QR2. Do you need to find errors and inconsistencies in requirements?Yes. Try: Model Checking: Tropos([8][14][15][19]) SNET([16][18]) DesignQD1. Are you aware of a sufficient number of high-level design alternatives?No. Try: Agent, Planning, Forward and Backward Satisfaction Approaches: NFR([9]) i* Satisfaction Analysis ([26][27][33]) Tropos Planning([4][6][7][8]) KAOS([31]) GRL Forward Satisfaction Analysis([1]) SNET Planning([16][18]) QD2. Are you aware of a sufficient number of detailed design alternatives?No. Try: Quantitative Planning, Forward and Backward Satisfaction Approaches: KAOS Satisfaction Analysis ([31]) GRL Forward Satisfaction Analysis([1]) Tropos Planning([6][7]) SNET Planning([16][18]) QD3. Do you need to evaluate and choose between high-level design alternatives?Yes. Try: Satisfaction Analysis, Metrics and Agent Approaches: KAOS Satisfaction Analysis([31]) i* Forward Satisfaction([26][33]) GRL Satisfaction Analysis([1]) i* Metrics([11][12][13]) Tropos Risk([4]) QD4. Do you need to evaluate and choose between detailed design alternatives?Yes. Try: Quantitative or Detailed Information: Tropos Probabalistic Satisfaction Analysis ([3][21][22][23]) KAOS Satisfaction Analysis ([31]) GRL Quant. Analysis ([1]) i* Quant. Metrics ([11][12][13]) Tropos Planning ([4][6][7][8]) Tropos Modeling Checking ([8][14][15][19]) SNET([16][17][18][18]) i* Simulation([34]) QD5. Do you need to find acceptable processes?Yes. Try: Planning Approaches: Tropos Planning([4][6][7][8]) SNET Planning([16][18]) QD6. Do you need to test run-time operation before implementation?Yes. Try: Simulation Approaches: SNET([16][17][18]) i* Simulation([34]) Mapping of Procedures to Objectives Objectives Procedures

Guideline Usage Examples  Example: Online Counseling Domain (Horkoff & Yu, 2009) Online counseling alternatives: text messaging or chat room? Apply guiding questions…  High degree of social interaction (QU1)  Do no yet understand details, not yet confident in the accuracy and completeness of models (Qu2, QM1)  Communication is important, scoping is challenging (QS1, QU2)  Etc… Recommendations:  Interactive, agent-oriented techniques for forward satisfaction analysis supporting softgoals  Analysis for anonymity or privacy with the same techniques or with GRL Satisfaction Analysis, and/or i* Metrics  If the required detailed information is available, apply planning and/or simulation techniques Analyzing Goal Models, Horkoff & Yu 25

Conclusions  First step towards making goal model analysis techniques more accessible to modelers  Enable potential users to user their knowledge of the domain and analysis objectives to select one or more procedures  We have attempted to be neutral in our analysis Each procedure has unique abilities  Future work is needed to undertake studies to validate or refute the claims made by our guidelines Hope that guidelines will be expanded and refined as more application experiences are available Analyzing Goal Models, Horkoff & Yu 26

Thank you   Analyzing Goal Models, Horkoff & Yu

Outline  Goal Models  Goal Model Analysis  Motivation: Abundance of Approaches  Survey of Goal Model Analysis Approaches  Survey Results Summary  Objectives of Goal Model Analysis  Mapping of Procedures to Objectives  Example Selection  Conclusions & Future Work 28 Analyzing Goal Models, Horkoff & Yu

Survey of GORE Analysis Techniques: Selection  Article selection: Started with a set of known relevant papers Linked work through references Stopped with picture of breadth was captured (24 papers)  Alternative selection methods: Search for specific key words… … in specific journals, conferences, portals … during specific time periods  Challenge: work in goal model analysis appears in a range of venues with a range of keywords Venues: Books, RE, REJ, Agent-related conferences, CAiSE, AI related journal, FSE, PoEM, Journal of Information Systems, Trust-related Conference, ASE, etc… Keywords: agent-oriented software development, goal-oriented requirements analysis, early requirements analysis, multi-agent systems, agent-oriented software engineering, agent-oriented methodologies, risk analysis, countermeasure identification, goal modeling, goal-oriented analysis, quality metrics, etc…. 29 Analyzing Goal Models, Horkoff & Yu

Guideline Usage Examples  Example: Wireless service from Amyot et al New wireless service added to existing network Where should the data and service be located? Apply guiding questions…  Domain contains some interacting systems (QU1), no emphasis on communication (QC1)  Aware of alternatives, but need to select one (QD1, QD3)  Do not yet understand details, detailed alternatives, don’t have access to specific information, don’t want to find processes or perform simulations (QU2, QR2, QD2, QD5, QD6)  Domain is well understood, scope is clear, models are sufficiently complete (QE1, QS1, QM1)  Must consider non-functional requirements (QE3), data privacy (QR1) Recommendations:  Agent-oriented approaches supporting softgoals to consider social and non-functional nature of the problem  Satisfaction analysis or metrics to chose between alternatives Analyzing Goal Models, Horkoff & Yu 30

Guideline Usage Examples  Example: Online Counseling Domain from Horkoff & Yu, 2009 Online counseling alternatives: text messaging or chat room? Apply guiding questions…  High degree of social interaction (QU1)  Do no yet understand details, not yet confident in the accuracy and completeness of models (Qu2, QM1)  Communication is important, scoping is challenging (QS1, QU2)  Consider many non-functional requirements, privacy and security especially, capture assumptions (QE3, QE4, QR1)  Need to find and evaluate alternatives (QD1, QD3)  Could be useful to find the most successful process (plan) for counseling or simulate throughput (QD4, QD5) Recommendations:  Interactive, agent-oriented techniques for forward satisfaction analysis supporting softgoals  Analysis for anonymity or privacy with the same techniques or with GRL Satisfaction Analysis, and/or i* Metrics  If the required detailed information is available, apply planning and/or simulation techniques Analyzing Goal Models, Horkoff & Yu 31