Methodologies evaluation

1
Methodologies evaluation

• Agentlink III
• AOSE TFG
• Budapest, 17 sep. 2005

2
Evaluation framework for AOSEM

• Towards an evaluation framework for AOSEM
• Previous approaches
• Questionnaire results
• Review
• Outline and plan for document on AOSEM evaluation
  framework

3
An evaluation framework for AOSEM

• Context
• Diverse scope of application of methodologies
• Several aspects analysis, design,
  implementation, deployment, validation,
  verification, etc.
• Several application domains from closed systems
  to open systems, web support, etc.
• Tool support
• Tools for modelling and code generation
• Some methodologies have no tool support at all
  (or in a very experimental state)
• Development process not always defined
• Different notations
• Different agent concepts
• Standardization efforts
• Several approaches for integration
• A common standard agent specification language
  which one?
• Fragments method engineering

4
An evaluation framework for AOSEM

• Evaluation of AOSEM can help towards the success
  of AOSE
• Clarification of concepts gt towards some
  standardization
• Integration of fragments
• Definition of AOSE processes heavy to light
  approaches
• Promotion of tools

5
Inputs for AOSEM evaluation

• A. Sturm, O. Shehory, D. Dori (2004). Evaluation
  of Agent-Oriented Methodologies. In AL3 TF1-AOSE
  TFG
• Q.N. Tran, G. Low (2005). Comparison of ten
  agent-oriented methodologies. In
  Henderson-Sellers, B. and Giorgini, P., editors
  (2005). Agent-Oriented Methodologies. Idea Group
  Publishing. Chapter XII, pp. 341-367.
• C. Bernon, et al. (2004). A Study of some
  Multi-Agent Meta-Models. Proc. AOSE 2004 (to
  appear in LNCS, Springer-Verlag).
• L. Cernuzzi, G. Rossi (2004). On the evaluation
  of agent oriented methodologies. In Proc. of the
  OOPSLA 2002 Workshop on Agent-Oriented
  Methodologies.
• L. Cernuzzi, M. Cossentino, F. Zambonelli (2005).
  Process Models for Agent-Based Development.
  International Journal on Engineering Applications
  of Artificial Intelligence (EAAI). Elsevier. (in
  edition?)

6
Questionnaire

• Originally from Mickael Winikoff and modified by
  Massimo Cossentino
• Aim assess an AOSE methodology against a range
  of criteria. The criteria fall into a number of
  areas.
• Concepts/properties The ideas that the
  methodology deals with, basically the ontology
• Modelling The models that are constructed and
  the notations used to express the models.
• Process The phases and steps that are followed
  as part of the methodology.
• Pragmatics Practical issues that are concerns
  when adopting a methodology (e.g., the
  availability of training materials and courses,
  the existence and cost of tools, etc.)

7
Questionnaire

• Answers from
• ADELFE (Carole Bernon/creator)
• INGENIAS (Jorge Gómez-Sanz Juan Pavón/creators)
• OPEN Process Framework (OPF) (Brian
  Henderson-Sellers/creator)
• Prometheus-ROADMAP (Lin Padgham/creator)
• Gaia (Giancarlo Fortino/Alfredo Garro users!!!)
• PASSI (M. Cossentinocreator, L. Sabatucci, V.
  Seidita/PhD Students users/doing research on it,
  8 graduating students users)
• TROPOS (3 students)
• Others are always welcome!!!
• Answers from users (not creators) can provide a
  better critical view of methodologies

8
Questionnaire

• Looking at the results of the questionnaire
• It can be useful to consider changes in the
  questionnaire
• Subjective interpretation of questions and
  answers
• Not applicable
• Missing questions
• Useful? Clarifying?
• Identification of methodology challenges
• Lets see what are the results and discuss

9
Questionnaire Concepts Properties
Creator/PhD Students/Grad. Stud.
Concept/Property Adelfe Gaia Ingenias OPF PASSI Prome-theus TROPOS
Autonomy H H H H H/H/M H L
Mental attitudes L N H H L/L/M M M
Proactiveness M L H H H/M/H H N
Reactiveness H L H H H/H/H H N
Concurrency H M H L H/H/M H L
Teamwork and roles L H H H M/H/H L M
Cooperation model AMAS th. Teamwork ALL ALL Task del./ Teamwork none Negotiation/ Task del.
Protocols support H H H H H/M/H H N
Communication modes ALL Async mess. ALL ALL Direct N
Communication language ALL ACL like ALL ALL Speech acts messages
Situatedness H H H H H/M/M H H
Environment type All episodic Dynamic Continuous All discrete ALL ALL ALL Inacc., Non episodic, Dynam.
N None L Low M Medium H High
10
Questionnaire Concepts Properties
Concept/Property Adelfe Gaia Ingenias OPF PASSI Prometheus TROPOS
Other agent features Opennes Opennes Opennes Mobility, openness, security Plans, agent decisions Security, Trust, Delegation, Ownership, Dependency, Provision
Non supported features Security Mobility (not explicitly) Security Mobility (on going work) Complex design-time social organiza-tions Security Mobility Dynamic Behavior of Agent
Clear concepts A A SA SA SA/N/N A A
Overloaded concepts N D D SD D/D/N D N
More Agent-oriented than OO A SA SA both SA/A/A SA SA
(Main) Supported agents Cooperative BDI (mainly) BDI (mainly) Mainly State-based, rational, reactive ALL BDI, Rational
Society of agents modelling No SA SA (on going work) A/-/- No A
Society structure - - Groups/WF - p2p, simple hierarchies, holons - Agent Society Pattern, such as Broker, Mediated, Matchmaker
SD Strongly Disagree    D Disagree    N
Neutral   A Agree    SA Strongly Agree
11
Questionnaire Modelling Notation
Creator/PhD Students/Grad. Stud.
Notation Adelfe Gaia Ingenias OPF PASSI Prome-theus TROPOS
Support for static (structure) and dynamic (processing) aspects SA A SA SA/A/A SA D
Symbols and syntax well defined A N SA A/A/D A N
Well defined semantics A D SA A/N/D A A
Clear notation A A N A/A/N A N
Easy to use notation A A SA A/A/N A SA
Easy to learn notation N SA A N/N/N NA N
A methodology is really notation independent.
Yes, there is a need for a modelling language and
in the FAME project we have FAML (FAME modelling
language) although not yet a notation. So we
cant really answer these notation specific
questions (i.e. 21-26)
SD Strongly Disagree    D Disagree    N
Neutral   A Agree    SA Strongly Agree NA Not
Applicable
12
Questionnaire Modelling Notation
Modelling Adelfe Gaia Ingenias OPF PASSI Prometheus TROPOS
Multiple views A N SA SA SA/A/A A
Adequate and expressive A N SA SA A/A/N N
Traceability between models and between models and code A D SA SA SA/N/A D
Guidelines and techniques for consistency checking A SD N N N/N/N D
Supports refinement SA N SA N SA/A/A A
Supports modularity SA D A SA SA/A/A N
Supports component reusability SA SD SA SA SA/A/A SD
Extensible SA SD SA SA A/-/- SA A
Supports hierarchical modelling and abstraction SA D SA SA SA/N/A SA A
Other issues
SD Strongly Disagree    D Disagree    N
Neutral   A Agree    SA Strongly Agree
13
Questionnaire Process
Lifecycle coverage Adelfe Gaia Ingenias OPF PASSI Prometheus TROPOS
Planning CE CEH CEH
Requirements analysis CE CE CEH CEH CEH/-/- CEH CE
Architectural (or agent society) design CE CE CEH CEH CEH/-/- CEH CE
Detailed (agent) design CE CE CEH CEH CEH/-/- CEH CE
Implementation CEH CEH CEH/-/- E P
Testing/Debugging H P CEH H PCEH
Deployment P CEH CE
Maintenance CEH P
Death CEH
C Clear definition of activities E Examples
given H Heuristics given P Partial
14
Questionnaire Process
Process Adelfe Gaia Ingenias OPF PASSI Prometheus TROPOS
Addresses Quality Assurance D SD N SA N/N/A A SD
Estimating guidelines (cost, ) SD A N D/N/A N N
Support for decision making (e.g. when to move between phases) A SD SA A N/A/A N D
Development approach Iterative/ incremental Top-down Iterative/ incremental Transformation architectural based ANY Iterative/ Incre-mental Iterative/ Incre-mental/ Spiral Top Down
Supports patterns or reusability A SD D SA SA/-/- N N
Degree of user implication (i.e. it does requires user-designer communication ?) Medium Medium Strong Weak M
SD Strongly Disagree    D Disagree    N
Neutral   A Agree    SA Strongly Agree
15
Questionnaire Pragmatics
Software tools Adelfe Gaia Ingenias OPF PASSI Prome- theus TROPOS
Diagram editor OpenTool IDK editor PTK GR-Tool, ST-Tool, TAOM4E
Code generator IDK code g. Agent Factory
Design consistency checker IDK ATA Prototype PTK GR-Tool, ST-Tool
Project Management AdelfeToolkit
Rapid prototyping
Reverse engineering Agent Factory
Automatic testing
Commercial or research product OT comm. AT free Research Research Research Research Research
Adequate level of functionalities A A A/-/- A? N
Quick and easy to learn N A N/-/- A? A
Support in raising the quality A A SA/-/- SA? N
Reduces time to design/implem. A SA SA/-/- SA? A
Other comments GPL license, UML/ Ingenias notation Considering other tools
SD Strongly Disagree    D Disagree    N
Neutral   A Agree    SA Strongly Agree
16
Questionnaire Pragmatics
Pragmatics Adelfe Gaia Ingenias OPF PASSI Prome-theus TROPOS
Audience All All All All All/-/- All Grad. st., experts, researcher
Complexity compared to UML/RUP About the same About the same About the same A lot simpler About the same/-/- About the same Simpler
Resources Papers X X X X X X
Text books X X X
Tutorial notes X X X X
Consulting services X X
Training services X X
Nr.applications built with meth. 1-5 21 6-20 21 in OO/ME 21/-/- 21 1-5
Were applications real? Yes No No All Y/Y/N Yes Yes
Any developed by other users? No Yes Yes Yes Y/Y/N Yes No
Target any specific domain Complex systems No No All but RT No/-/- No No
Support scalability Yes No Yes Yes A/N/- N
Supports distributed systems Yes Yes Yes SA/A/SA N
17
Evaluation framework revisited

• Taking the experience of this questionnaire
• Review evaluation framework criteria and their
  organization
• Review method for evaluation questionnaire, case
  studies development, ...
• Refine questionnaire
• Define case studies
• Review metrics
• How to avoid subjectivity

18
Evaluation framework revisited Criteria for
AOSEM evaluation
Process DeliverablesActivities Team
work Domain specific methods Tools
Modelling Autonomy, society, AbstractionModula
rity Domain specific concepts Knowledge
skills Scalability
Features
Complexity
Domain
Pragmatics
19
Towards an AOSEM evaluation framework

• The evaluation framework should allow
• Criteria refinement and extensions
• Criteria metrics depending on the domain
• E.g. agents in a web service or in robotics
• Definition of standard case studies for
  evaluation
• Evaluation of documentation and filling
  questionnaires is not enough

20
Towards an AOSEM evaluation framework

• The framework can be based on the definition and
  use of evaluation models
• Case studies for putting the methodologies to
  work
• Organized by criteria
• For each criteria, define metrics
• Criteria can be refined to get more insight or
  being more specific
• For instance, agent behaviour, depending on
  whether BDI, neural network, CBR, reactive, or
  whatever model is used
• New criteria can be added
• Some criteria may be considered non applicable
• Associate criteria to case studies

21
Outline and plan for document on AOSEM
evaluation framework

• Outline
• Participants
• Plan