Dept. of Computation, UMIST A Design Complexity Evaluation Framework for Agent-Based System Engineering Methodologies Anthony Karageorgos Nikolay Mehandjiev Dept. of Computation, UMIST {karageorgos,mehandjiev}@co.umist.ac.uk ESAW 2003 London, 29-31 October 2003 OVERVIEW 1. Design Complexity Issues 2. Evaluation Framework 3. Case Study 4. Discussion

Complexity in ABS Engineering Complexity: difficulty in understanding Software engineering complexity: difficulty in understanding, developing and verifying software ABS Design complexity: the difficulty in understanding and manipulating, concepts, models and techniques associated with the ABS design process High software complexity results to high development effort, less reusability and low software quality The structure and properties of agents increase design complexity To reduce design complexity of ABS engineering methodologies we need to assess it Design complexity can be application dependent, i.e. an agent monitoring a device involves low design complexity

Evaluating ABS Engineering methodologies Develop a case study Use historical data from past projects Other evaluation techniques Descriptive user-based: Ask potential users to fill questionnaires framework-based: Compare methodologies with respect to aspects of interest grouped in a framework meta-model based: Compare meta-model descriptions of methodologies Experimental Replicated experiments: carry out the same task in multiple ways and examine the differences, i.e. using Gaia and SODA Run-time analysis: Test the software product on run-time. Simulation. Execute the software product with a model of the real environment Experimental techniques require more effort and resources Descriptive: User-based. Framework-based. Metamodel-based: (Qualitative) In a replicated experiment, several projects (or subjects) are staffed to perform a task in multiple ways. Researchers set control variables, such as duration, staff level, and methods used. By using such a method, researchers can establish statistical validity more easily than by relying on case studies. In a replicated experiment, researchers replace a given task with another task. They might replace Ada with C++, eliminate walkthroughs, or add independent verification and validation. Researchers form several treatments that implement products using either the old or new task. Then they collect data on both approaches and compare the results. This method represents the classical scientific experiment. If there are enough replications—perhaps 20 to 40—researchers can establish the statistical validity of the method under consideration. While replicated experiments evaluate the process, Run-time analysis and simulation evaluate the product. In run-time analysis we can have debuggers and profilers etc Experimental methods have the disadvantage that they require effort and resources.

Complexity Evaluation Framework Concept definition Design in scope Heuristics support Design perspective Support for reuse Design automation Concepts Models Agent-Based System Design Process Each view includes a number of conceptually linked aspects and examines an ABS methodology from different perspectives. A qualitative ranking scheme is applied for each aspect. Concepts: which modelling concepts are used by the methodology. Organisational settings Collective behaviour Non-functional aspects Generality Abstractability Tool support Pragmatics

Concepts and Models Evaluation Framework Aspects Concepts Models Description/Ranking Values Concepts Concept definition How is the approach characterised regarding restrictions in the definitions of agents and agent-based systems? [limited (), bounded (<>), open (><)] Design in scope Does the engineering approach provide explicit support for the design phase of the agent-based system engineering lifecycle? [yes (), no()] Heuristics support Does the approach provide formal techniques to support application of design heuristics? Models Organisational settings Are organisational settings considered as first-class design constructs? [yes (), no()] Collective behaviour Are collective behaviours considered as first-class design constructs? Non-functional aspects Are non-functional aspects explicitly modelled and considered in the agent-based system design?

Process and Pragmatics Evaluation Framework Aspects Description/Ranking Values Process Design perspective What is the development perspective of the approach? [bottom-up (), top-down (), both ( )] Support for reuse Does the approach provide guidelines and techniques to reuse existing design knowledge? [yes (), no()] Design automation Do suitable formal underpinnings exist that can automate the design process to a certain extent? Pragmatics Generality What is the generality (possibility of being applied to many application domains) of the approach? [low (○), medium (), high ()] Abstractability Does the approach provide formal support for reasoning at different levels of abstraction? Tool support Is there any assistance to the agent-based system designers by some software tool? It must be noted that some aspects are interrelated. For example, low or limited concept definition is likely to be combined with low or medium generality, as is the case in RAPPID [Parunak et al. 1999]. However, this is not always the case, For example, Tropos [Bresciani et al. 2001] is bounded to only BDI agents and it is still applicable in many application domains.

Case Study Material Customers : Suppliers VE Administration Warehouses Retailers Manufacturers Information Agent Must reuse appropriate designs that already exist in VE members Need a software tool to carry out the design. Similar designs are needed for other application domains Authority relationships between VE Administration and VE members Certain patterns of interaction, such as contract-net or auction protocols should be followed to establish service-level agreements Security, privacy and performance requirements Proven heuristics need to be followed, for example all VE financial transactions should be carried by a specialised trusted agent

Framework Application Agent-Based System Engineering We classify current ABS engineering methodologies with respect to how design is done in a classification scheme We select one methodology from each class in the classification scheme Approaches Static Dynamic e.g. KARMA Ad-hoc Formal Informal e.g. RAPPID e.g. DESIRE IS OOSE K. Eng. Tool Based e.g. Tropos e.g. GAIA e.g. MESSAGE e.g. Zeus RAPPID: domain specific targeting collaborative design DESIRE: a compositional framework for engineering and verification of agent-based systems Gaia: combines agent theoretic concepts and object-oriented engineering principles Tropos: originated from information systems engineering MESSAGE: applies best practices from knowledge engineering methodologies to ABS development Zeus: a popular methodology based on the Zeus toolkit Karma/TEAMCORE: locates and recruits existing agents in the cyberspace

Evaluation Results Concepts Models Process Pragmatics RAPPID DESIRE Gaia MESSAGE Tropos Zeus KARMA/TEAMCORE Concept definition  <> >< Design in scope ─  Heuristics support Models Organisational settings Collective Behaviour Non-functional aspects Process Design Perspective   Support for reuse Design Automation Pragmatics Generality ○   Abstractability Tool Support

Research Implications I Support design heuristics Describe heuristics as formal design constraints Informally considered in Zeus but involve high design complexity Formally in DESIRE but no systematic guidelines as to how to use them Organisational Settings When considered they model only organisational relations No rigorous methods for combining organisational settings with application functionality Collective Behaviour No rigorous methods to reuse collective behaviour

Research Implications II Non-functional aspects Describe heuristics as formal design constraints Informally considered in Zeus but involve high design complexity Formally in DESIRE but no systematic guidelines as to how to use them Automating the design process Generally achieved through successive transformations of analysis models KARMA supports automatic ABS formation but it assumes agents already exist Working at different abstraction levels Informal support by MESSAGE and ZEUS Partial support by DESIRE and KARMA However, the focus is to ensure correct representation of analysis models in the design components and satisfying the application requirements in the design.

Future Work Extend the framework to cover other phases of the ABS engineering lifecycle Develop a systematic ABS methodology selection method based on the theory of the Analytic Hierarchy Process Develop a methodology selection library and software tool