1. Methodologies evaluation
Agentlink III
AOSE TFG
Budapest, 17 sep. 2005

2. Evaluation framework for AOSEM
Towards an evaluation framework for AOSEM
Previous approaches
Questionnaire results
Review
Outline and plan for document on AOSEM evaluation framework
AOSE TFG
Budapest meeting, 17/9/2005

3. An evaluation framework for AOSEM
Context
Diverse scope of application of methodologies
Several aspects: analysis, design, implementation, deployment, validation, verification, etc.
Several application domains: from closed systems to open systems, web support, etc.
Tool support
Tools for modelling and code generation
Some methodologies have no tool support at all (or in a very experimental state)
Development process not always defined
Different notations
Different agent concepts
Standardization efforts
Several approaches for integration:
A common standard agent specification language: which one?
Fragments: method engineering
AOSE TFG
Budapest meeting, 17/9/2005

4. An evaluation framework for AOSEM
Evaluation of AOSEM can help towards the success of AOSE
Clarification of concepts => towards some standardization
Integration of fragments
Definition of AOSE processes: heavy to light approaches
Promotion of tools
AOSE TFG
Budapest meeting, 17/9/2005

5. Inputs for AOSEM evaluation
A. Sturm, O. Shehory, D. Dori (2004). Evaluation of Agent-Oriented Methodologies. In: AL3 TF1-AOSE TFG
Q.N. Tran, G. Low (2005). Comparison of ten agent-oriented methodologies. In: Henderson-Sellers, B. and Giorgini, P., editors (2005). Agent-Oriented Methodologies. Idea Group Publishing. Chapter XII, pp
C. Bernon, et al. (2004). A Study of some Multi-Agent Meta-Models. Proc. AOSE 2004 (to appear in LNCS, Springer-Verlag).
L. Cernuzzi, G. Rossi (2004). On the evaluation of agent oriented methodologies. In: Proc. of the OOPSLA 2002 Workshop on Agent-Oriented Methodologies.
L. Cernuzzi, M. Cossentino, F. Zambonelli (2005). Process Models for Agent-Based Development. International Journal on Engineering Applications of Artificial Intelligence (EAAI). Elsevier. (in edition?)
AOSE TFG
Budapest meeting, 17/9/2005

6. Questionnaire
Originally from Mickael Winikoff and modified by Massimo Cossentino
Aim: assess an AOSE methodology against a range of criteria. The criteria fall into a number of areas.
Concepts/properties: The ideas that the methodology deals with, basically the ontology
Modelling: The models that are constructed and the notations used to express the models.
Process: The phases and steps that are followed as part of the methodology.
Pragmatics: Practical issues that are concerns when adopting a methodology (e.g., the availability of training materials and courses, the existence and cost of tools, etc.)
AOSE TFG
Budapest meeting, 17/9/2005

7. Questionnaire Answers from: ADELFE (Carole Bernon/creator)
INGENIAS (Jorge Gómez-Sanz & Juan Pavón/creators)
OPEN Process Framework (OPF) (Brian Henderson-Sellers/creator)
Prometheus-ROADMAP (Lin Padgham/creator)
Gaia (Giancarlo Fortino/Alfredo Garro: users!!!)
PASSI (M. Cossentino:creator, L. Sabatucci, V. Seidita/PhD Students: users/doing research on it, 8 graduating students: users)
TROPOS (3 students)
Others are always welcome!!!
Answers from users (not creators) can provide a better critical view of methodologies
AOSE TFG
Budapest meeting, 17/9/2005

8. Looking at the results of the questionnaire
It can be useful to consider changes in the questionnaire
Subjective interpretation of questions and answers
Not applicable
Missing questions
Useful? Clarifying?
Identification of methodology challenges
Let’s see what are the results and discuss…
AOSE TFG
Budapest meeting, 17/9/2005

9. Questionnaire – Concepts & Properties
Creator/PhD Students/Grad. Stud.
Questionnaire – Concepts & Properties
Concept/Property
Adelfe
Gaia
Ingenias
OPF
PASSI
Prome-theus
TROPOS
Autonomy H
H/H/M L
Mental attitudes N
L/L/M M
Proactiveness
H/M/H
Reactiveness
H/H/H
Concurrency
Teamwork and roles
M/H/H
Cooperation model
AMAS th.
Teamwork
ALL
Task del./ Teamwork
none
Negotiation/ Task del.
Protocols support
Communication modes
Async mess.
Direct
Communication language
ACL like
Speech acts
messages
Situatedness
H/M/M
Environment type
All episodic
Dynamic
Continuous
All discrete
Inacc., Non episodic, Dynam.
N: None L: Low M: Medium H: High
AOSE TFG
Budapest meeting, 17/9/2005

10. Questionnaire – Concepts & Properties
Concept/Property
Adelfe
Gaia
Ingenias
OPF
PASSI
Prometheus
TROPOS
Other agent features —
Opennes
Mobility, openness, security
Plans, agent decisions
Security, Trust, Delegation, Ownership, Dependency, Provision
Non supported features
Security & Mobility (not explicitly)
Security & Mobility (on going work)
Complex design-time social organiza-tions
Security & Mobility
Dynamic Behavior of Agent
Clear concepts A SA
SA/N/N
Overloaded concepts N D SD
D/D/N
More Agent-oriented than OO
both
SA/A/A
(Main) Supported agents
Cooperative
BDI (mainly)
Mainly: State-based, rational, reactive
ALL
BDI, Rational
Society of agents modelling No
(on going work)
A/-/-
Society structure -
Groups/WF
p2p, simple hierarchies, holons
Agent Society Pattern, such as Broker, Mediated, Matchmaker
AOSE TFG
Budapest meeting, 17/9/2005
SD: Strongly Disagree    D: Disagree    N: Neutral   A: Agree    SA: Strongly Agree

11. Questionnaire – Modelling & Notation
Creator/PhD Students/Grad. Stud.
Questionnaire – Modelling & Notation
Notation
Adelfe
Gaia
Ingenias
OPF
PASSI
Prome-theus
TROPOS
Support for static (structure) and dynamic (processing) aspects SA A —
SA/A/A D
Symbols and syntax well defined N
A/A/D
Well defined semantics
A/N/D
Clear notation
A/A/N
Easy to use notation
Easy to learn notation
N/N/N NA
A methodology is really notation independent. Yes, there is a need for a modelling language and in the FAME project we have FAML (FAME modelling language) although not yet a notation. So we can’t really answer these notation specific questions (i.e )
SD: Strongly Disagree    D: Disagree    N: Neutral   A: Agree    SA: Strongly Agree
NA: Not Applicable
AOSE TFG
Budapest meeting, 17/9/2005

12. Questionnaire – Modelling & Notation
Adelfe
Gaia
Ingenias
OPF
PASSI
Prometheus
TROPOS
Multiple views A N SA
SA/A/A —
Adequate and expressive
A/A/N
Traceability between models and between models and code D
SA/N/A
Guidelines and techniques for consistency checking SD
N/N/N
Supports refinement
Supports modularity
Supports component reusability
Extensible
A/-/-
Supports hierarchical modelling and abstraction
Other issues
AOSE TFG
SD: Strongly Disagree    D: Disagree    N: Neutral   A: Agree    SA: Strongly Agree
Budapest meeting, 17/9/2005

13. Questionnaire – Process
Lifecycle coverage
Adelfe
Gaia
Ingenias
OPF
PASSI
Prometheus
TROPOS
Planning CE
CEH
Requirements analysis
CEH/-/-
Architectural (or agent society) design
Detailed (agent) design
Implementation E P
Testing/Debugging H
PCEH
Deployment
Maintenance
Death
C: Clear definition of activities
E: Examples given
H: Heuristics given
P: Partial
AOSE TFG
Budapest meeting, 17/9/2005

14. Questionnaire – Process
Adelfe
Gaia
Ingenias
OPF
PASSI
Prometheus
TROPOS
Addresses Quality Assurance D SD N SA
N/N/A A
Estimating guidelines (cost, …) —
D/N/A
Support for decision making (e.g. when to move between phases)
N/A/A
Development approach
Iterative/
incremental
Top-down
Transformation & architectural based
ANY
Incre-mental
Incre-mental/
Spiral
Top Down
Supports patterns or reusability
SA/-/-
Degree of user implication (i.e. it does requires user-designer communication ?)
Medium
Strong
Weak M
SD: Strongly Disagree    D: Disagree    N: Neutral   A: Agree    SA: Strongly Agree
AOSE TFG
Budapest meeting, 17/9/2005

15. Questionnaire – Pragmatics
Software tools
Adelfe
Gaia
Ingenias
OPF
PASSI
Prome- theus
TROPOS
Diagram editor
OpenTool
IDK editor
PTK
GR-Tool, ST-Tool, TAOM4E
Code generator
IDK code g.
Agent Factory
Design consistency checker
IDK ATA
Prototype
GR-Tool, ST-Tool
Project Management
AdelfeToolkit
Rapid prototyping
Reverse engineering
Automatic testing
Commercial or research product
OT: comm.
AT: free
Research
Adequate level of functionalities A
A/-/- A? N
Quick and easy to learn
N/-/-
Support in raising the quality
SA/-/-
SA?
Reduces time to design/implem. SA
Other comments
GPL license, UML/ Ingenias notation
Considering other tools
AOSE TFG
Budapest meeting, 17/9/2005
SD: Strongly Disagree    D: Disagree    N: Neutral   A: Agree    SA: Strongly Agree

16. Questionnaire – Pragmatics
Adelfe
Gaia
Ingenias
OPF
PASSI
Prome-theus
TROPOS
Audience
All
All/-/-
Grad. st., experts, researcher
Complexity compared to UML/RUP
About the same
A lot simpler
About the same/-/-
Simpler
Resources: Papers X
Text books
Tutorial notes
Consulting services
Training services
Nr.applications built with meth.
1-5
21+
6-20
21+ in OO/ME
21+/-/-
Were applications real?
Yes No
Y/Y/N
Any developed by other users?
Target any specific domain
Complex systems
All but RT
No/-/-
Support scalability
A/N/- — N
Supports distributed systems
SA/A/SA
AOSE TFG
Budapest meeting, 17/9/2005

17. Evaluation framework revisited
Taking the experience of this questionnaire
Review evaluation framework criteria and their organization
Review method for evaluation: questionnaire, case studies development, ...
Refine questionnaire
Define case studies
Review metrics
How to avoid subjectivity
AOSE TFG
Budapest meeting, 17/9/2005

18. Evaluation framework revisited Criteria for AOSEM evaluation
Modelling
Autonomy, society, …
Abstraction Modularity
Domain specific concepts
Knowledge skills
Scalability
Process
Deliverables Activities
Team work
Domain specific methods
Tools
Features
Complexity
Domain
Pragmatics
AOSE TFG
Budapest meeting, 17/9/2005

19. Towards an AOSEM evaluation framework
The evaluation framework should allow:
Criteria refinement and extensions
Criteria metrics depending on the domain
E.g. agents in a web service or in robotics
Definition of standard case studies for evaluation
Evaluation of documentation and filling questionnaires is not enough …
AOSE TFG
Budapest meeting, 17/9/2005

20. Towards an AOSEM evaluation framework
The framework can be based on the definition and use of evaluation models
Case studies for putting the methodologies to work
Organized by criteria
For each criteria, define metrics
Criteria can be refined to get more insight or being more specific
For instance, agent behaviour, depending on whether BDI, neural network, CBR, reactive, or whatever model is used
New criteria can be added
Some criteria may be considered non applicable
Associate criteria to case studies
AOSE TFG
Budapest meeting, 17/9/2005

21. Outline and plan for document on AOSEM evaluation framework
Participants
Plan
AOSE TFG
Budapest meeting, 17/9/2005