1. Dept. of Computation, UMIST
A Design Complexity Evaluation Framework for Agent-Based System Engineering Methodologies
Anthony Karageorgos
Nikolay Mehandjiev
Dept. of Computation, UMIST
ESAW 2003
London, October 2003
OVERVIEW
1. Design Complexity Issues
2. Evaluation Framework
3. Case Study
4. Discussion

2. Complexity in ABS Engineering
Complexity: difficulty in understanding
Software engineering complexity: difficulty in understanding, developing and verifying software
ABS Design complexity: the difficulty in understanding and manipulating, concepts, models and techniques associated with the ABS design process
High software complexity results to high development effort, less reusability and low software quality
The structure and properties of agents increase design complexity
To reduce design complexity of ABS engineering methodologies we need to assess it
Design complexity can be application dependent, i.e. an agent monitoring a device involves low design complexity

3. Evaluating ABS Engineering methodologies
Develop a case study
Use historical data from past projects
Other evaluation techniques
Descriptive
user-based: Ask potential users to fill questionnaires
framework-based: Compare methodologies with respect to aspects of interest grouped in a framework
meta-model based: Compare meta-model descriptions of methodologies
Experimental
Replicated experiments: carry out the same task in multiple ways and examine the differences, i.e. using Gaia and SODA
Run-time analysis: Test the software product on run-time.
Simulation. Execute the software product with a model of the real environment
Experimental techniques require more effort and resources
Descriptive: User-based. Framework-based. Metamodel-based: (Qualitative)
In a replicated experiment, several projects (or subjects) are staffed to perform a task in multiple ways. Researchers set
control variables, such as duration, staff level, and methods used. By using such a method, researchers can establish statistical validity more easily than by relying on case studies.
In a replicated experiment, researchers replace a given task with another task. They might replace Ada with C++, eliminate walkthroughs, or add independent verification and validation. Researchers form several treatments that implement products using either
the old or new task. Then they collect data on both approaches and compare the results. This method represents
the classical scientific experiment. If there are enough replications—perhaps 20 to 40—researchers
can establish the statistical validity of the method under consideration.
While replicated experiments evaluate the process, Run-time analysis and simulation evaluate the product.
In run-time analysis we can have debuggers and profilers etc
Experimental methods have the disadvantage that they require effort and resources.

4. Complexity Evaluation Framework
Concept definition
Design in scope
Heuristics support
Design perspective
Support for reuse
Design automation
Concepts
Models
Agent-Based
System Design
Process
Each view includes a number of conceptually linked aspects and examines an ABS methodology from different perspectives.
A qualitative ranking scheme is applied for each aspect.
Concepts: which modelling concepts are used by the methodology.
Organisational settings
Collective behaviour
Non-functional aspects
Generality
Abstractability
Tool support
Pragmatics

5. Concepts and Models Evaluation Framework Aspects Concepts Models
Description/Ranking Values
Concepts
Concept definition
How is the approach characterised regarding restrictions in the definitions of agents and agent-based systems?
[limited (), bounded (<>), open (><)]
Design in scope
Does the engineering approach provide explicit support for the design phase of the agent-based system engineering lifecycle?
[yes (), no()]
Heuristics support
Does the approach provide formal techniques to support application of design heuristics?
Models
Organisational settings
Are organisational settings considered as first-class design constructs?
[yes (), no()]
Collective behaviour
Are collective behaviours considered as first-class design constructs?
Non-functional aspects
Are non-functional aspects explicitly modelled and considered in the agent-based system design?

6. Process and Pragmatics
Evaluation Framework Aspects
Description/Ranking Values
Process
Design perspective
What is the development perspective of the approach?
[bottom-up (), top-down (), both ( )]
Support for reuse
Does the approach provide guidelines and techniques to reuse existing design knowledge?
[yes (), no()]
Design automation
Do suitable formal underpinnings exist that can automate the design process to a certain extent?
Pragmatics
Generality
What is the generality (possibility of being applied to many application domains) of the approach?
[low (○), medium (), high ()]
Abstractability
Does the approach provide formal support for reasoning at different levels of abstraction?
Tool support
Is there any assistance to the agent-based system designers by some software tool?
It must be noted that some aspects are interrelated.
For example, low or limited concept definition is likely to be combined with low or medium generality, as is the case in RAPPID [Parunak et al. 1999]. However, this is not always the case, For example, Tropos [Bresciani et al. 2001] is bounded to only BDI agents and it is still applicable in many application domains.

7. Case Study
Material
Customers :
Suppliers VE
Administration
Warehouses
Retailers
Manufacturers
Information
Agent
Must reuse appropriate designs that already exist in VE members
Need a software tool to carry out the design. Similar designs are needed for other application domains
Authority relationships between VE Administration and VE members
Certain patterns of interaction, such as contract-net or auction protocols should be followed to establish service-level agreements
Security, privacy and performance requirements
Proven heuristics need to be followed, for example all VE financial transactions should be carried by a specialised trusted agent

8. Framework Application
Agent-Based System Engineering
We classify current ABS engineering methodologies with respect to how design is done in a classification scheme
We select one methodology from each class in the classification scheme
Approaches
Static
Dynamic
e.g. KARMA
Ad-hoc
Formal
Informal
e.g. RAPPID
e.g. DESIRE IS
OOSE
K. Eng.
Tool Based
e.g. Tropos
e.g. GAIA
e.g. MESSAGE
e.g. Zeus
RAPPID: domain specific targeting collaborative design
DESIRE: a compositional framework for engineering and verification of agent-based systems
Gaia: combines agent theoretic concepts and object-oriented engineering principles
Tropos: originated from information systems engineering
MESSAGE: applies best practices from knowledge engineering methodologies to ABS development
Zeus: a popular methodology based on the Zeus toolkit
Karma/TEAMCORE: locates and recruits existing agents in the cyberspace

9. Evaluation Results Concepts Models Process Pragmatics
RAPPID
DESIRE
Gaia
MESSAGE
Tropos
Zeus
KARMA/TEAMCORE
Concept definition 
<>
><
Design in scope ─ 
Heuristics support
Models
Organisational settings
Collective Behaviour
Non-functional aspects
Process
Design Perspective  
Support for reuse
Design Automation
Pragmatics
Generality ○  
Abstractability
Tool Support

10. Research Implications I
Support design heuristics
Describe heuristics as formal design constraints
Informally considered in Zeus but involve high design complexity
Formally in DESIRE but no systematic guidelines as to how to use them
Organisational Settings
When considered they model only organisational relations
No rigorous methods for combining organisational settings with application functionality
Collective Behaviour
No rigorous methods to reuse collective behaviour

11. Research Implications II
Non-functional aspects
Describe heuristics as formal design constraints
Informally considered in Zeus but involve high design complexity
Formally in DESIRE but no systematic guidelines as to how to use them
Automating the design process
Generally achieved through successive transformations of analysis models
KARMA supports automatic ABS formation but it assumes agents already exist
Working at different abstraction levels
Informal support by MESSAGE and ZEUS
Partial support by DESIRE and KARMA
However, the focus is to ensure correct representation of analysis models in the design components and satisfying the application requirements in the design.

12. Future Work
Extend the framework to cover other phases of the ABS engineering lifecycle
Develop a systematic ABS methodology selection method based on the theory of the Analytic Hierarchy Process
Develop a methodology selection library and software tool