1 Reference Model for Evaluating Intelligent Tutoring Systems Esma Aimeur, Claude Frasson Laboratoire HERON Informatique et recherche opérationnelle Université.

Slides:



Advertisements
Similar presentations
Experimental Course for Students with LD/ADHD Diana Cassie, Ph.D. Dalhousie University.
Advertisements

Modelling with expert systems. Expert systems Modelling with expert systems Coaching modelling with expert systems Advantages and limitations of modelling.
Alina Pommeranz, MSc in Interactive System Engineering supervised by Dr. ir. Pascal Wiggers and Prof. Dr. Catholijn M. Jonker.
Team 6 Lesson 3 Gary J Brumbelow Matt DeMonbrun Elias Lopez Rita Martin.
User Interfaces 4 BTECH: IT WIKI PAGE:
Improving System Safety through Agent-Supported User/System Interfaces: Effects of Operator Behavior Model Charles SANTONI & Jean-Marc MERCANTINI (LSIS)
Bridgette Parsons Megan Tarter Eva Millan, Tomasz Loboda, Jose Luis Perez-de-la-Cruz Bayesian Networks for Student Model Engineering.
CHAPTER 1 SOFTWARE DEVELOPMENT. 2 Goals of software development Aspects of software quality Development life cycle models Basic concepts of algorithm.
Clinical Coach Standardisation Meeting August 2011.
The Training-Learning Process Mala, Jeya, Nesrin, Noura.
Técnicas de Calidad en el Software Sesión # 10. Good quality software Operations Transition Testing Portability Reusability Interoperability Maintainability.
Usability Evaluation Evaluation should occur continually through the design and implementation process. Evaluation methods are applied as the interface.
Intelligent Blended Learning Objects: Developing Student Models Trevor Barker Blended Learning Unit Introduction In Higher Education today, and in general,
6.1 Intelligent User Interfaces: achievements and challenges ISE554 The WWW for eLearning.
Design and Evaluation of Iterative Systems n For most interactive systems, the ‘design it right first’ approach is not useful. n The 3 basic steps in the.
Project Life Cycle Jon Ivins DMU. Introduction n Projects consist of many separate components n Constraints include: time, costs, staff, equipment n Assets.
Evaluation Through Expert Analysis U U U
Providing Tutoring Service through Accumulating Interaction Data Chi-Jen LIN Fo Guang University, Taiwan.
North American Coordinating Council on Japanese Library Resources Training the Trainers Emily Werrell and Sharon Domier August 7-9, 2004.
Building Knowledge-Driven DSS and Mining Data
Usability and Evaluation Dov Te’eni. Figure ‎ 7-2: Attitudes, use, performance and satisfaction AttitudesUsePerformance Satisfaction Perceived usability.
Ch 3 Usability page 1CS 368 Usability Models the authors compare three usability models and introduce their own “the extent to which a product can be used.
Usability Evaluation Methods Computer-Mediated Communication Lab week 10; Tuesday 7/11/06.
1 User Interface Design CIS 375 Bruce R. Maxim UM-Dearborn.
 A set of objectives or student learning outcomes for a course or a set of courses.  Specifies the set of concepts and skills that the student must.
Click to edit Master title style  Click to edit Master text styles  Second level  Third level  Fourth level  Fifth level  Click to edit Master text.
“Enhancing Reuse with Information Hiding” ITT Proceedings of the Workshop on Reusability in Programming, 1983 Reprinted in Software Reusability, Volume.
Demystifying the Business Analysis Body of Knowledge Central Iowa IIBA Chapter December 7, 2005.
Business Analysis and Essential Competencies
ADALE’06 DublinJune 20, 2006 Issues in developing adaptive learning management systems for higher education institutions Jesus G. Boticario, Olga C. Santos.
Cognitive Reasoning to Respond Affectively to the Student Patrícia A. Jaques Magda Bercht Rosa M. Vicari UNIVERSIDADE FEDERAL DO RIO GRANDE DO SUL BRASIL.
Evaluation of Adaptive Web Sites 3954 Doctoral Seminar 1 Evaluation of Adaptive Web Sites Elizabeth LaRue by.
11 C H A P T E R Artificial Intelligence and Expert Systems.
Formative Evaluation cs3724: HCI. Problem scenarios summative evaluation Information scenarios claims about current practice analysis of stakeholders,
Multimedia Specification Design and Production 2012 / Semester 1 / week 5 Lecturer: Dr. Nikos Gazepidis
SEG3120 User Interfaces Design and Implementation
Comp 15 - Usability & Human Factors Unit 8a - Approaches to Design This material was developed by Columbia University, funded by the Department of Health.
 Architecture and Description Of Module Architecture and Description Of Module  KNOWLEDGE BASE KNOWLEDGE BASE  PRODUCTION RULES PRODUCTION RULES 
Introduction Complex and large SW. SW crises Expensive HW. Custom SW. Batch execution Structured programming Product SW.
UI Style and Usability, User Experience Niteen Borge.
Quality Evaluation methodologies for e-Learning systems (in the frame of the EC Project UNITE) Tatiana Rikure Researcher, Riga Technical University (RTU),
Cognitive Science Overview Cognitive Apprenticeship Theory.
Incorporating Self Regulated Learning Techniques into Learning by Teaching Environments Biswas, G., Leelawong, K. and Belynne, K. The twenty sixth Annual.
1 Evaluating the Quality of the e-Learning Experience in Higher Education Anne Jelfs and Keir Thorpe, Institute of Educational Technology (IET), The Open.
Design Process … and some design inspiration. Course ReCap To make you notice interfaces, good and bad – You’ll never look at doors the same way again.
COMM89 Knowledge-Based Systems Engineering Lecture 8 Life-cycles and Methodologies
Assessment Formats Charlotte Kotopoulous Regis University EDEL_450 Assessment of Learning.
Introduction The design, development and maintenance of concurrent software are difficult tasks. Truly effective support methodologies have yet to be developed.
Using Emotional Coping Strategies in Intelligent Tutoring Systems Soumaya Chaffar & Claude Frasson Departement d’Informatique et de Recherche Opérationnelle.
Fall 2002CS/PSY Predictive Evaluation (Evaluation Without Users) Gathering data about usability of a design by a specified group of users for a particular.
User Interface Evaluation Introduction Lecture #15.
Design Evaluation Overview Introduction Model for Interface Design Evaluation Types of Evaluation –Conceptual Design –Usability –Learning Outcome.
Learning Adaptation Slavi Stoyanov. LA Position Conceptual Design –Operational definition of LD (what is it? Why is it? whom is it for? what do we adapt.
1 Design and evaluation methods: Objectives n Design life cycle: HF input and neglect n Levels of system design: Going beyond the interface n Sources of.
HCC 831 User Interface Design and Evaluation. What is Usability?
SIE 515 Design Evaluation Lecture 7.
Human-Computer Interaction
Selection of Instructional Methods and Media
Model based design.
CIS 210 Systems Analysis and Development
Lesson 1 Foundations of measurement in Psychology
Analysis of Software Usability Evaluation Methods
Usability Techniques Lecture 13.
User Interface Design and Evaluation
Comp 15 - Usability & Human Factors
Proper functionality Good human computer interface Easy to maintain
Evaluation.
HCI Evaluation Techniques
CSM18 Usability Engineering
SYSTEM TESTING AND DEPLOYMENT
Presentation transcript:

1 Reference Model for Evaluating Intelligent Tutoring Systems Esma Aimeur, Claude Frasson Laboratoire HERON Informatique et recherche opérationnelle Université de Montréal

2 Laboratoire HERON - TICE 2000 Introduction Goal of an ITS (Intelligent Tutoring System): produce the behaviour of an intelligent (competent) human tutor who can adapt his teaching to the learning rythm of the learner Ability to model and reason about domain knowledge, human thinking, learning processes, and teaching process building an ITS needs also to evaluate the system lack of evaluation methodology

3 Laboratoire HERON - TICE 2000 Architecture of an ITS Student

4 Laboratoire HERON - TICE 2000 Evaluation of ITS Evaluation can be formative or summative Summative: after the design process Formative: during the design (define and refine goals and methods) Evaluation techniques used in –ITS –Software engineering

5 Laboratoire HERON - TICE 2000 Evaluation of some ITS SHERLOCK : experimental evaluation VCR Tutor : 4 versions compared Formative qualitative evaluation –iterative design –formative methods qualitative : characteristics of a situation quantitative : finding causes and consequences –case studies

6 Laboratoire HERON - TICE 2000 Lester’s evaluation On animated pedagogical agents (Design a Plant) --- botanical and physiology 100 middle school students Important educational benefits : improved problem solving Better than less expressive animated agents

7 Laboratoire HERON - TICE 2000 Littman- Soloway What is the educational impact on students ? What is the relationship between the architecture of an ITS and its behavior ? External evaluation of the student model (Proust) Internal evaluation : architecture, how ITS respond to input values

8 Laboratoire HERON - TICE 2000 Van Lehn Teaching metacognitive skills to implement and evaluate an ITS (SE- Coach guide self-explanation) Empirical evaluation Fundamental questions –How the design tools are used efficiently ? –How the learner improve his knowledge ?

9 Laboratoire HERON - TICE 2000 Software engineering techniques Product : Boehm’model –software doing what the user want it to do –use resources correctly –easy to learn –well designed, code –easily tested and maintained

10 Laboratoire HERON - TICE 2000 HCI techniques Evaluating design –cognitive walthrough (how easy a system is to learn) –heuristic evaluation (visibility, consistency, flexibility, helps for the user) –model-based evaluation (GOMS) predict user performance

11 Laboratoire HERON - TICE 2000 Evaluationg Implementation Empirical and experimental methods Observational methods (user completes a set of tasks) Query techniques –interviews of the users about their experience –questionnaires : questions fixed in advance

12 Laboratoire HERON - TICE 2000 Learner model Layers : – cognitive –affective –inferential Learning time : speed of knowledge acquisition Tracability : of learner’s actions

13 Laboratoire HERON - TICE 2000 Learner model Levels of knowledge –Based on works of Gagné (1985) –7 levels of knowledge –4 principal levels Novice Beginner Intermediate Expert

14 Laboratoire HERON - TICE 2000  One-on-one  Co-learner  Learning Companion  Learning by Disturbing Learning strategies

15 Laboratoire HERON - TICE 2000 Learning strategies Diversity Adaptability (switching to different strategies) Modification Memorization Feedback Reduce cognitive load

16 Laboratoire HERON - TICE 2000 Curriculum Generalisation Consistency Knowledge articulation Reusability Navigability Maintenance

17 Laboratoire HERON - TICE 2000 Interface Intuitive : learner should understand easily the functions to apply Interactivity : allowing the learner to be active in his learning environment Matching : using words, phrase and concepts

18 Laboratoire HERON - TICE 2000 General Productivity rate : the most important factor. Time spent to produce one hour of ITS based course Learning outcomes : performance obtained by the learner after the session Ease of use of the tools

19 Laboratoire HERON - TICE 2000 Conclusion Evaluation process is complex but needs to be realistic Develop first, evaluate later Evaluation criteria as a guideline for estimation

20 Laboratoire HERON - TICE 2000 Merci de votre attention

21 Laboratoire HERON - TICE 2000 Stratégies d ’apprentissage TuteurCo-apprenant CompagnonApprentissage par explication