1. Layered Evaluation of Adaptive Systems Dr. Stephan Weibelzahl National College of Ireland Dublin http://www.weibelzahl.de/ http://www.weibelzahl.de/ Dr Alexandros Paramythis Johannes Kepler University Linz http://www.fim.uni-linz.ac.at/staff/paramythis/ http://www.fim.uni-linz.ac.at/staff/paramythis/

2. Programmiersprache C++Winter 2005Operator overloading (2) (2) Example System HTML-Tutor: An Adaptive Learning System Introduction to HTML and Publishing on the Web

3. Programmiersprache C++Winter 2005Operator overloading (3) (3) Pre-Tests optional pretest

4. Programmiersprache C++Winter 2005Operator overloading (4) (4) Exercises

5. Programmiersprache C++Winter 2005Operator overloading (5) (5) Exercise Feedback

6. Programmiersprache C++Winter 2005Operator overloading (6) (6) adaptive link annotation Adaptation Strategy

7. Programmiersprache C++Winter 2005Operator overloading (7) (7) Adaptation Strategy adaptive curriculum sequencing

8. Programmiersprache C++Winter 2005Operator overloading (8) (8) How can I evaluate my system? How can we evaluate adaptivity? How can we find out what’s wrong? How can we find ways to improve it? Compare adaptive version to non-adaptive version? »What could we learn from that? »What can we not learn from that?

9. Programmiersprache C++Winter 2005Operator overloading (9) (9) Layered evaluation Basic premises of layered evaluation »Don’t treat adaptation as a “monolithic” / singular process (at least not only as such) »Rather, “break it down” into its constituents (“layers”), and »Evaluate each of them separately where necessary and feasible

10. Programmiersprache C++Winter 2005Operator overloading (10) (10) How can I evaluate my system? Questions addressed in this lecture »Evaluation layer(s): What to evaluate? »Criteria: What are possible measures of success? (see also a previous lecture) Covered in the next lecture »What evaluation methods and data collection techniques ar appropriate for our goals?

11. Programmiersprache C++Winter 2005Operator overloading (11) (11) Layered Evaluation collect input data interpret data model the current state of the world decide upon adaptation applyadaptation non- interactive “sensors” interactive “front-end” “static” models application model task model system model... “dynamic” models user model context model interaction history... adaptivetheory Paramythis & Weibelzahl, 2005

12. Programmiersprache C++Winter 2005Operator overloading (12) (12) Collection of Input Data Adaptive system observes user behaviour and context, e.g., click stream, input, sensor data, etc. Questions »Does the data collection work? »Is the user behaviour registered accurately? Criteria »Reliability (consistency of data) »Accuracy »Latency collect input data interpret data model the current state of the world decide upon adaptation applyadaptation

13. Programmiersprache C++Winter 2005Operator overloading (13) (13) Collection of Input Data Examples of questions to be answered »Eye-tracking for task detection: Does the user actually look at the part of the screen that the eye- tracker indicates? »HTML-Tutor: Are test items reliable? »Movie recommender: Are ratings of movies consistent per user? Would a user rate the movie in the same way again after one week? collect input data interpret data model the current state of the world decide upon adaptation applyadaptation

14. Programmiersprache C++Winter 2005Operator overloading (14) (14) Interpretation of the Collected Data Adaptive system interprets the recorded behaviour, giving meaning to raw data »Sometimes trivial (click on “next” button means, user wants to proceed to next page) »However, interpretation is possibly based on assumptions and might require inference Question »Are the users doing what the system thinks they are doing? Possible criteria »Validity collect input data interpret data model the current state of the world decide upon adaptation applyadaptation

15. Programmiersprache C++Winter 2005Operator overloading (15) (15) Interpretation of the Collected Data Examples of questions to be answered »HTML-Tutor: Is the content of a page actually “known” when Learner visited the page Learner answered test- items correctly »Movie recommender: Does a user actually like a movie when giving a positive rating? collect input data interpret data model the current state of the world decide upon adaptation applyadaptation

16. Programmiersprache C++Winter 2005Operator overloading (16) (16) Example Study with HTML-Tutor Learners use system Learners complete post-test Comparison of model (“visited”, “known”) and real data collect input data interpret data model the current state of the world decide upon adaptation applyadaptation

17. Programmiersprache C++Winter 2005Operator overloading (17) (17) Modelling of the Current State of the World Based on observations the system infers the current state of the world, e.g., user model, context model »Usually this is the AI component of the system (Bayesian network, rules, etc) Questions »Does the model reflect the real world? »Is the world modelled in an appropriate way? Possible criteria »Primarily: Validity »Secondary: Comprehensiveness, Redundancy, Precision, Sensitivity, Scrutability collect input data interpret data model the current state of the world decide upon adaptation applyadaptation

18. Programmiersprache C++Winter 2005Operator overloading (18) (18) Modelling of the Current State of the World Examples of questions to be answered »HTML-Tutor: Are pages that are inferred to be “known” (e.g., prerequisites of more advanced concepts) actually known? »Movie Recommender: Do users like a movie that got high ratings from somebody with similar preferences? collect input data interpret data model the current state of the world decide upon adaptation applyadaptation

19. Programmiersprache C++Winter 2005Operator overloading (19) (19) Example Study with HTML-Tutor Learners learn concepts in class Learners use system Learners complete post-test Comparison of model (“inferred”) and real data

20. Programmiersprache C++Winter 2005Operator overloading (20) (20) Decide upon Adaptation Adaptive System decides which adaptation theory/strategy to apply given the current user model Questions »Is it necessary to intervene? »Did the system select a good and appropriate adaptation strategy? Possible criteria »Necessity »Appropriateness »Subjective acceptance collect input data interpret data model the current state of the world decide upon adaptation applyadaptation

21. Programmiersprache C++Winter 2005Operator overloading (21) (21) Decide upon Adaptation Examples of questions to be answered »HTML-Tutor: The learner model seems to indicate that the learner acquired sufficient knowledge about the current chapter. Shall we recommend to proceed to the next chapter? Shall we annotate the current chapter as “known”? »Movie recommender: Shall we recommend a certain movie (push) or wait till the user asks for a recommendation (pull)? collect input data interpret data model the current state of the world decide upon adaptation applyadaptation

22. Programmiersprache C++Winter 2005Operator overloading (22) (22) Example Study Learners use system under different conditions »With and without annotation »With and without sequencing Results »No effect on number of pages visited, overall impression or perceived successful adaptation »Annotation increases number of pages visited per minute How could this study be improved to better fit the layer? collect input data interpret data model the current state of the world decide upon adaptation applyadaptation

23. Programmiersprache C++Winter 2005Operator overloading (23) (23) Applying Adaptation Decisions The adaptation decision can be applied in different ways (e.g., different colours, layouts, formulations) Questions »Is the concrete instantiation of the adaptation decision working? »Do users understand what it means? Do they like it? Possible criteria »Usability »Obtrusiveness »Acceptance »Timeliness »User control collect input data interpret data model the current state of the world decide upon adaptation applyadaptation

24. Programmiersprache C++Winter 2005Operator overloading (24) (24) Applying Adaptation Decisions Examples of questions to be answered »HTML-Tutor: Is a red bullet a good way to indicate a “not recommended” page? “Continue with the next suggested page”? »Movie Recommender: Shall we provide the full list of recommended movies? Only one movie plus “more” button? “Based on your ratings we believe that you might like the following movies…”? collect input data interpret data model the current state of the world decide upon adaptation applyadaptation

25. Programmiersprache C++Winter 2005Operator overloading (25) (25) Evaluating Adaptation as a Whole The big picture »Looking at the system as a whole: Does it work? Questions »Does the system achieve its goals? »Does it improve interaction? »Do users like the system? Possible criteria »Effectiveness »Efficiency »Usability »System specific criteria collect input data interpret data model the current state of the world decide upon adaptation applyadaptation

26. Programmiersprache C++Winter 2005Operator overloading (26) (26) System-specific criteria The layered approach described, and the proposed generic criteria, are not by themselves sufficient to assess whether adaptivity “meets its goals” A complete evaluation design then must include system-specific criteria that are »Specific to what gets adapted, how, and with what goal »Do not neglect the intrinsic characteristics of the application domain(s) These do not need to be addressed in isolation »But they tend to address adaptivity as a whole, so one has to be careful in how they are combined with the assessment of other criteria

27. Programmiersprache C++Winter 2005Operator overloading (27) (27) Evaluating Adaptation as a Whole Examples of questions to be answered »HTML-Tutor: Does adaptation to prior-knowledge save time? »Movie Recommender: Do users find movies they like and would they have found these movies otherwise? collect input data interpret data model the current state of the world decide upon adaptation applyadaptation

28. Programmiersprache C++Winter 2005Operator overloading (28) (28) Field Study What’s the impact of offering an adaptive prior-knowledge test in an on-line course? (Weibelzahl & Weber, 2002) 140 users learned with the HTML-Tutor »optional pre-test for 3 chapters »final knowledge test at the end of the course »criteria: duration, knowledge »statistical analysis: MANOVA and ANOVA Photo © BrowserBob, 2007BrowserBob

29. Programmiersprache C++Winter 2005Operator overloading (29) (29) Results no pre-test presented pre-test presented No differences in knowledge Completed course much quicker

30. Programmiersprache C++Winter 2005Operator overloading (30) (30) Layered Evaluation – Complete example Where is the user looking on the screen What concept is the user reading about What concepts has the user learned eye-tracker Educational AHS “static” models model of concepts and relation to pages “dynamic” models learner (overlay) model adaptivetheory Guide user to exercises for the learned concepts Add icons in front of links to the exercises

31. Programmiersprache C++Winter 2005Operator overloading (31) (31) Layered Evaluation Summary Main premises »Break adaptation process down into its constituents (“layers”) »Evaluate each of them separately where necessary and feasible »Meant to provide guidance rather than prescribing a certain way of evaluation Benefits »Offers guidance for possible studies (“separation of concerns”) »Helps to identify problems and wrong assumptions »Guides development process (formative evaluation)

32. Programmiersprache C++Winter 2005Operator overloading (32) (32) Reading List [Layered Evaluation] Paramythis, A. & Weibelzahl, S. (2005). A Decomposition Model for the Layered Evaluation of Interactive Adaptive Systems. In Ardissono, L., Brna, P., & Mitrovic, A. (Eds.), Proceedings of the 10th International Conference on User Modeling (UM2005), Edinburgh, Scotland, UK, July 24-29 (pp. 438-442) (Lecture Notes in Computer Science LNAI 3538, Springer Verlag). Berlin: Springer. [Layered evaluation] Weibelzahl, S. (2001). Evaluation of adaptive systems. In M. Bauer, P. Gmytrasiewicz & J. Vassileva (Eds.), User Modeling 2001: Proceedings of the Eighth International Conference, UM2001. (pp. 292-294) (Lecture Notes in Computer Science LNAI 2109; © Springer-Verlag). Berlin: Springer. [Layered evaluation] Paramythis, A., Totter, A., & Stephanidis, C. (2001). A modular approach to the evaluation of Adaptive User Interfaces. In S. Weibelzahl, D. Chin & G. Weber (Eds.), Proceedings of the Workshop on Empirical Evaluations of Adaptive Systems, held in the context of the 8th International Conference on User Modeling (UM'2001), 13-17 July, Sonthofen, Germany (pp.9-24). Freiburg: Pedagogical University of Freiburg.

33. Programmiersprache C++Winter 2005Operator overloading (33) (33) Additional resources Evaluation of Adaptive Systems Hub http://www.easy-hub.org/ »Previous workshops »Guidelines »Literature references