ADAPTIVE ASSESSMENT IN WEB-BASED LEARNING By Dunwei Wen (1), Sabine Graf (2), Chung Hsien Lan (3), Terry Anderson (1), Kinshuk (1), Ken Dickson(1) (1) Athabasca University, Alberta, Canada (2) Women's Postgraduate College for Internet Technologies, Vienna University of Technology, Austria (3) Nanya Institute of Technology, Jhongli, Taiwan Presenter: Dunwei Wen IEEE ICME International Conference on Multimedia & Expo, July 2-5, 2006, Beijing

IEEE International Conference on Multimedia & Expo, Beijing, OUTLINE I. Am I Ready – An Adaptive Readiness Self-assessment System  1. Introduction  2. Knowledge Modeling 2.1. User Model 2.2. Counseling Model 2.3. Process Model 2.4. Assessment Model  3. System Structure And Realization  4. Conclusion of Am I Ready II. Performance Self-Assessment III. Peer Assessment

IEEE International Conference on Multimedia & Expo, Beijing, I.AM I READY for Athabasca Unv. A DISTANCE EDUCATION READINESS SELF-ASSESSMENT SYSTEM Features  Is an adaptive online readiness self-assessment system  Helps prospective students understand their requirements readiness  Provides information of distance learning at Athabasca University. Based on integrated knowledge models  dynamically adjusts the contents of the self-assessment according to the interaction between the knowledge models the user’s responses.

IEEE International Conference on Multimedia & Expo, Beijing, Introduction Online readiness self-assessments are widely used in universities that provide distance education services. Most current readiness self-assessment tools are online questionnaires. The shortcoming of those systems:  The same question set for different users  The same question sequence for every user  Less remedial information to help users

IEEE International Conference on Multimedia & Expo, Beijing, BASIC IDEAS - features User-oriented  Questions are filtered for different users Dynamic and adaptive  Next questions rely on users’ previous answers Remedial  Instant feedback and remedial assessment information

IEEE International Conference on Multimedia & Expo, Beijing, BASIC IDEAS – methods Technologies to model counselling knowledge  User Model  Counselling Model  Process Model  Assessment Model. Operators adopted according to the above models of a user:  Enabling some questions  Disabling some questions  Sorting questions by priorities of questions  Checking contradictory answers  Showing real time information

IEEE International Conference on Multimedia & Expo, Beijing, Knowledge Modeling And Architecture

IEEE International Conference on Multimedia & Expo, Beijing, USER MODEL (1) STATIC MODEL Acting as the first assessment filter: Highest level of education Employment status Financial capability Computer skills Disability Full or partial assessment

IEEE International Conference on Multimedia & Expo, Beijing, STATIC USER MODEL

IEEE International Conference on Multimedia & Expo, Beijing, (2) DYNAMIC MODEL Records of a user’s response history are dynamically modified in real time as the assessment proceeds. The answer to each question The records of questions that are disabled, enabled, or changed to higher priority

IEEE International Conference on Multimedia & Expo, Beijing, COUNSELLING MODEL Modeling questions  Dividing counselling information into fields and their sub-fields  Assigning questions to each sub-field  Setting up relations between questions Authoring instant information  Instant information (including descriptions, links to Web pages with multimedia resources etc) can be assigned to each choice of a question. They will show whenever you make a choice. Pre-defined answer types Yes and No Yes, No and Not Clear Grades Multi-Checks

IEEE International Conference on Multimedia & Expo, Beijing, Questions & Instant Information

IEEE International Conference on Multimedia & Expo, Beijing, FOUR KINDS OF RELATIONS Enable Disable Plus Contradictory Examples: (Enable (answer(i) of question(j), ( question list ) (sub-field list) ) ) (Contradict (answer(i) of question(j), (list of answers of some questions)))

IEEE International Conference on Multimedia & Expo, Beijing, PROCESS MODEL Modeling the behavior that controls the counseling process Actions:  Utilizing user model and counseling model  Changing question priority in real time  Sorting questions by priority  Filtering questions

IEEE International Conference on Multimedia & Expo, Beijing, NEXT PAGE

IEEE International Conference on Multimedia & Expo, Beijing, ASSESSMENT Provides assessment information in response to user’s answers Two kinds of assessment information:  Question related  Question-group related Example: (Assessment (i)  (answer(i 1 ) of question(j 1 ), (answer(i 2 ) of question(j 2 ), … ) )

IEEE International Conference on Multimedia & Expo, Beijing, ASSESSMENT

IEEE International Conference on Multimedia & Expo, Beijing, SYSTEM STRUCTURE

IEEE International Conference on Multimedia & Expo, Beijing, Process Interaction

IEEE International Conference on Multimedia & Expo, Beijing, REALIZATION Putting the knowledge into PostgreSQL database (21 tables)  including rules, relations, facts and text information Determining the question sequences  Reasoning by the support of SQL Interacting with users  Java/JSP based HTML Webpage Dialog

IEEE International Conference on Multimedia & Expo, Beijing, CONCLUSION Am I Ready  User oriented & user specific self-assessment  Dynamic & adaptive self-assessment process  Real time guidance and remedial information to improve the readiness of students for distance education  More effective and more natural self-assessment process  Flexible design for more general use

IEEE International Conference on Multimedia & Expo, Beijing, II. Performance Self-Assessment How can performance self-assessment help to address the different learning styles of students in learning management systems?  For providing adaptive courses  For improving adaptivity by gathering additional information about the students

IEEE International Conference on Multimedia & Expo, Beijing, Motivation Learning Management Systems (LMS) are often and successfully used in e-education but provide little or in the most cases no adaptivity Learners have different needs Considering learning styles makes learning easier and increases the learning progress

IEEE International Conference on Multimedia & Expo, Beijing, Felder-Silverman Learning Style Model Richard M. Felder and Linda K. Silverman, 1988 Each learner has a preference on each of the four dimensions:  Active – Reflective learning by doing – learning by thinking things through learning by discussing & group work – work alone  Sensing – Intuitive concrete material – abstract material more practical – more innovative and creative patient and careful – not patient and careful with details standard procedures – challenges  Visual – Verbal learning from pictures – learning from words  Sequential – Global learn in linear steps – learn in large leaps good in using partial knowledge – need “big picture” interested in details – interested in the overview

IEEE International Conference on Multimedia & Expo, Beijing, Adaptive Features for Self-Assessments Courses can be adapted with respect to the order and number of included learning objects Regarding self-assessments we consider:  Theoretical tests  Practical exercises Adaptation features for theoretical tests  Presentation at the beginning of each chapter  Presentation at the end of each chapter  Presentation at the end of the course Adaptation features for practical exercises  Number of presented exercises  Presentation at the beginning of each chapter  Presentation at the end of each chapter

IEEE International Conference on Multimedia & Expo, Beijing, Adaptation with respect to learning styles Active learning style:  Theoretical tests and exercises at the beginning and end of each chapter (due to the preference for active learning)  High number of exercises Reflective learning style:  First present learning material, then self-assessments  students have time to reflect about the material  Low number of exercises Sensing learning style:  First present learning material, then self-assessments  students can use the learned material for performing the self-assessments  High number of exercises (due to the preference for problem solving) Intuitive learning styles:  Present self-assessments first  provide learners with challenges  Low number of exercises

IEEE International Conference on Multimedia & Expo, Beijing, Adaptation with respect to learning styles Sequential learning style:  Avoid theoretical tests at the end of the course (since sequential learners prefer to test their knowledge in short intervals)  Theoretical tests at the end of each chapter are more suitable Global learning style:  Avoid self-assessments at the beginning of the chapter (since global learners need the big picture for performing self-assessments)  Theoretical tests can be provided at the end of the course

IEEE International Conference on Multimedia & Expo, Beijing, Self-Assessments for Improving Adaptivity Providing adaptivity requires knowing the students’ learning styles Self-assessments can be used to gather additional information from learners This information can help to identify the learning styles of students more accurately and therefore enable the system to provide more suitable adaptivity Examples for patterns:  Time students spend on self-assessments  sensing/intuitive  Number of performed exercises  active/reflective  Performance on questions about multimedia content  visual/verbal  Performance on questions about details and overview  sequential/global

IEEE International Conference on Multimedia & Expo, Beijing, III. Peer Assessment Peer assessment is one form of group assessment The issue of fairness has to be concerned in group assessment The proposed methodology can aggregate students’ marks and consider individual learning styles to  Reduce personal bias  Enhance the accuracy of the assessment

IEEE International Conference on Multimedia & Expo, Beijing, Adaptive Peer Assessment The process of adaptive peer assessment  Detect reviewers’ learning styles Learning Styles: Active/reflective, sensing/intuitive, visual/verbal, and sequential/global  Consider assessment criteria and learning styles Assessment criteria: Creativity, Completeness, Execution, and Security  Aggregate all marks and sends the result to the original author Assessment feedback = w 1 x 1 +w 2 x 2 +…w i x i Assessment criteria Weight of learning styles

IEEE International Conference on Multimedia & Expo, Beijing, Relations between Assessment Criteria and Learning Styles Execution and Security  Active students tend to be experimentalists Completeness  Sensing students like solving problems by standard methods and are patient with detail Creativity  Intuitive students like innovation

IEEE International Conference on Multimedia & Expo, Beijing, FUTURE WORK A knowledge editor to enhance the speed and accuracy of knowledge modeling, modification and upgrade of AM I READY. Graphical statistical information representation of readiness in self- assessment report. Data mining for analyzing the question-answer pairs and their relations, and expanding the knowledge base. A formal knowledge description and a SQL-based reasoning method. Distance education readiness self-assessment is only a rudimentary step to general dynamic and adaptive counseling systems. More multimedia resources.

IEEE International Conference on Multimedia & Expo, Beijing, THANK YOU! Questions?