1. : Flexible Open Learner Modeling Sergey Sosnovsky, PAWS@SIS@PITT

2. References Susan Bull,, UK. Mabbott, A. & Bull, S. (2004). Alternative Views on Knowledge: Presentation of Open Learner Models, ITS2004, 689-698. Mabbott, A. & Bull, S. (2006). Student Preferences for Editing, Persuading and Negotiating the Open Learner Model, ITS2006, 481-490. Kerly, A. & Bull, S. (2006). The Potential for Chatbots in Negotiated Learner Modelling, ITS2006, 443-452.

3. Outline Open User Model Flexi-OLM: –Viewing LM –Editing LM –Persuading LM –Negotiating LM –Multiple LM Presentations –Evaluation Demo

4. What: Visualization of the learner model, providing a learner with a mechanism to explore it, sometimes, negotiate it. Why: When a learner is engaged in the analysis of the learner model he is reflecting upon his domain knowledge and experience re- calling and re-considering ideas of which he is aware. Open Learner Modeling

5. Flexi-OLM Models student understanding of basic C programming Uses color coding for telling students about the concept knowledge levels: –While – limited understanding –Pale yellow – somewhat limited –Yellow/green – moderate –Bright green – excellent –Red – misconception –Grey – insufficient data Large topics include smaller concepts. Clicking on a topic in the model brings more detailed concept-wise information about this topic understanding. Knowledge are assessed with the help of short questions After playing with the system: –Questions correspond to only one concept –No knowledge Inference between concepts –Very simple modeling formula (seems like average with linear thresholds for knowledge levels)

6. Editing LM Flexi-OLM allows direct editing of LM Possible scenarios for this will be: –new learner wishes to inform the system about topics she already understands; –the learner grasps a concept outside the system and wants LM to reflect this; –the learner correctly guesses a series of answers => LM has a higher knowledge level than she believes she has.

7. Scrutinizing and Persuading LM Less direct method: –A learner registers a disagreement with the LM and propose a change –Flexi-OLM explains its believes by presenting the evidence supporting these beliefs –If the learner still wishes to proceed, she has the opportunity to ‘persuade’ the LM by answering a series of test questions. Possible Scenarios: –A learner believes her knowledge may be different than the system asserts, but lacks the confidence to edit it unchallenged, –A learner seeks the satisfaction of proving the system wrong

8. Negotiating LM Flexi-OLM supports conversation-based negotiation of LM: –A learner is chatting with the system (as he thinks) –Flexi-OLM maintains separate believe models for LM and for a learner –It is ensured that the same dialogue moves are available to both parties => Each party: has full control over their own beliefs, can challenge the other’s belief, can seek justification for the other’s belief (on the LM side justification is based on the past learner’s answers), may request justification before changing their own beliefs, may ultimately decide to leave their belief unchanged. –If the difference between LM’s and Student’s beliefs is: 1 level –The LM accepts the learner’s suggestion 2 levels – A compromise is offered (of changing both beliefs by one level) 3 levels – The systems seeks a justification (the learner will be asked to answer a question) [possible hack] – gradual change of the LM belief by one level

9. Negotiating LM (cont.) The “Wizard-of-Oz” Paradigm Human experimenter takes the role of the chatbot – “Wizard” The “Wizard” follows a protocol designed to ensure: –that responses to students remained consistent between users, and –that the ‘chatbot’ was believable to users. To enact the protocol, the Wizard was provided with some 350 pre-authored ‘chatbot’ negotiation initiations and responses to user inputs. Typical conversation:

10. LM Presentations in Flexi-OLM hierarchy, a logical grouping of related concepts; lectures, where topics are organized the same as in the related lecture course; concept map showing relationships between the topics; prerequisites, showing possible sequences for studying topics; index, an alphabetical list; ranked, where topics are listed in order of proficiency; textual summary.

11. Experiment 1 (2004) Two questions: –Is it beneficial for students to have a choice over presentation of open LM, or it causes information overflow? –Is there any strong preference for a particular LM view among individuals and if so, can it be predicted on the basis of learning style? 23 undergraduate students Experiment flow: –Pre-test (control flow in C) to populated LM –Browsing session, where students can choose among 4 different presentations

12. Experiment 1: Results

13. Experiment 2 (2006) Main question : –What are the students preferences concerning editing, negotiating and persuading LM? 8 third-year undergraduate students Experiment flow: –Initial testing to populate LM –1-hour LM exploring session (edit & persuade) –20 minutes of negotiating with LM Editing LM (full student control) Negotiating LM Persuading LM (full system control)

14. Experiment 2: Results 34 11 23 36 20 Sum

15. Experiment 3 (2006) The goal: –To explore the feasibility of using a chat-based interface in an OLM system 11 final year undergraduates Experiment flow: –Self-assessment of student knowledge of each concept, providing the initial user’s beliefs to the system. –Interacting with the system – to populate LM and provide it with its beliefs about student knowledge –Then students were shown how to use the chatbot and asked to interact with it for at least 20 minutes. –Post-experiment questionnaire

16. Experiment 3(2006)

17. Demo http://olm.eee.bham.ac.uk/flexi-olm/login.php

18. Thank You for Your Questions