School of something FACULTY OF OTHER School of Computing FACULTY OF ENGINEERING Evaluation Kleanthous Styliani

Overview Common Evaluation Approaches Before Planning an Evaluation Evaluation for this PhD School of Computing FACULTY OF ENGINEERING

Evaluation approaches for Intelligent Systems Formative & Summative Evaluation Layered Evaluation (Specific to Adaptive Systems) Simulations Control Groups School of Computing FACULTY OF ENGINEERING

Simulations: preferred when you need large amounts of data or data is too expensive to collect or when people have to be involved and there is no available sample. (e.g. P2P Communities, Social Networks) Control Groups: allow to different samples to use a system with and without the intelligent functionality and measure which group did best or according to what they are evaluating. (Comtella) (negative: the non-intelligent version cannot be optimal in any way if the system built with intelligent functionality at the beginning ) School of Computing FACULTY OF ENGINEERING

Formative: the system should be evaluated for its usability and effectiveness in the early stages. Summative: the effectiveness of the system is determined in real environments after development or completion of a major stage. Layered approach: Layer 1 – Interaction Assessment Evaluation e.g. Are the users characteristics being successfully detected by the system and maintained in the user model? Layer 2 – Adaptation Decision Making Evaluation e.g. Are the adaptation decisions valid and meaningful for selected assessment results? School of Computing FACULTY OF ENGINEERING

School of Computing FACULTY OF ENGINEERING Before planning the evaluation: What are you evaluating? What are your research questions/hypothesis? How will you test the above with the evaluation? Plan the evaluation

What are you evaluating? Extraction of community model Pattern detections Evolution algorithms Advantages of interventions School of Computing FACULTY OF ENGINEERING

What are your research questions? Formative Evaluation Do the CM algorithms work as are intended to work? Do the pattern detection algorithms extract the correct patterns? Do the evolution algorithms pick the changes happen through time? School of Computing FACULTY OF ENGINEERING

Summative Evaluation Structured specifically for this framework with specific questions to be answered Suitability of interventions: How do members evaluate the interventions? Benefits for users: Do the users find the interventions helpful in Identifying people relevant to them? Identifying resources relevant to them? Become aware of who is working on what? Identify potential collaborators? School of Computing FACULTY OF ENGINEERING

Benefits for the community: Newcomers How quickly are they integrated in the VC? Oldtimers How active they are? Transactive Memory Do they know who to ask or where to find resources for topic A? Do they know what other members know in the VC? To whom is your knowledge important? School of Computing FACULTY OF ENGINEERING

Benefits for the community: Shared Mental Models What others are doing in this VC? What is the purpose of this VC? Cognitive Centrality Who shares the most valuable resources in this community? Have the centrality shifted more effectively between members? Any peripheral members became cognitively central? School of Computing FACULTY OF ENGINEERING

Where to evaluate? BSCW Data of Semantic Web VC (Formative) AWESOME Simulated Data (Formative) & (prove generality of approach) Active VC to evaluate the whole framework & focusing on the intelligent interventions (Summative) Qualitative (questionnaires) Quantitative (statistics) School of Computing FACULTY OF ENGINEERING

Why am I telling you these things? BSCW VC for our group I need your contribution to complete the major part of evaluation – Summative Evaluation Share some resources with the others Download resources that might interest you Try to follow the guidelines given School of Computing FACULTY OF ENGINEERING

Thank you! School of Computing FACULTY OF ENGINEERING

Mark, M. & Greer, J. (1993). Evaluation Methodologies for Intelligent Tutoring Systems. Journal of Artificial Intelligence and Education, 4 (2/3), pp Karagiannidis, C. and D.G. Sampson. Layered Evaluation of Adaptive Applications and Services. in International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems Springer-Verlag. Millan, E. and J. Perez-De-La-Cruz, A Bayesian Diagnostic Algorithm for Student Modeling and its Evaluation. User Modeling and User-Adapted Interaction, (2): p Shlomo, B., et al. Evaluating User Model Effectiveness by Simulation. in Workshop on Personalized Access on Cultural Heritage at 11th international conference on user modeling Cheng R., Vassileva J. (2006) Design and Evaluation of an Adaptive Incentive Mechanism for Sustained Educational Online Communities. User Modelling and User-Adapted Interaction, 16 (2/3), (special issue on User Modelling Supporting Collaboration and Online Communities). School of Computing FACULTY OF ENGINEERING