Giving More Adaptation Flexibility to Authors of Adaptive Assessments Symeon Retalis University of Piraeus Department of Technology Education and Digital Systems

AthenaQTI2 Need for Adaptation One can find a great variety of learners in an on-line course or assessment: both in terms of their knowledge, but also their learning style, preferences, etc. The learning experience becomes more fruitful for the learner when these differences are accounted for, thus offering a tailored experience.

AthenaQTI3 What do they offer? They try to estimate the learner’s knowledge on the topics of the learning material. Based on the learner’s performance, they increase or decrease the knowledge level on the specific topic. They follow some built-in rules to determine the learner’s knowledge level and also to offer the corresponding material. The rules always remain the same. Feedback and help to the learner.

AthenaQTI4 Assessment Tools Some of the well-known commercial authoring tools include: Unit-Exam Questionmark Perception CourseBuilder JavaScript QuizMaker Quiz Rocket Test Generator Pro None of the above tools supports adaptation. Systems that support adaptation include: InterBook SIETTE AHA! NetCoach ActiveMath However, apart from SIETTE, none of the above systems offers assessment authoring.

AthenaQTI5 What is needed? More flexibility to the author of the assessment, i.e. the instructor. Each assessment has different characteristics: target audience, goals, difficulty level, etc. These characteristics cannot be ignored and handled by the same standard adaptation rules.

AthenaQTI6 AthenaQTI is a web-based adaptive assessment authoring tool. It conforms to the IMS_QTI specification, a fact that gives it the advantage of interoperability It gives the author the ability to form his/her own adaptation rules. It allows the author to create, save and edit adaptive assessments on the web, without the installation of any software. It uses a user model, which also conforms to international standards.

AthenaQTI7 Standards The tool conforms to the IMS_QTI (Question & Test Interoperability) specification. IMS_QTI provides the basic structure for the assessment (representation of questions, results, etc.). It also enables the interoperability and portability of the assessments among either web-based assessment systems or even Learning Management Systems

AthenaQTI8 User Model Necessary for adaptation Includes:  demographic data  learner’s prior knowledge  learner’s education level and area of expertise  learner’s demonstrated knowledge level on the topics assessed.  history of performance

AthenaQTI9 Tool’s Operations An author can: Create and save an assessment Create and add rules to an assessment Edit an assessment A learner can: Take an assessment

AthenaQTI10 Assessment Creation The author gives details about the assessment, also selecting its topic from a given vocabulary (currently CS related)

AthenaQTI11 Section Creation The author adds sections to the assessment and also specifies the section’s details, as well as its thematic topic. The author is directed to the sub-topics of the assessment’s topic.

AthenaQTI12 Types of items created True/False Multiple choice Fill in the blanks Multiple image choice Image hot-spot

AthenaQTI13 Rule creation The author defines a number of conditions that will be checked at a ‘trigger point’ (which s/he also defines) and the action that will be taken if they are satisfied.

AthenaQTI14 Assessment execution Every learner is recognized and s/he receives the appropriate assessment. Her/his user model is constantly updated during the assessment. Feedback is given after every question answered. Every new learner/user has to create a new account. During this process the new user gives demographic data and an estimation of her/his knowledge on different topics.

AthenaQTI15 Comparison to SIETTE SIETTE is the most relevant tool to which AthenaQTI can be compared. In SIETTE, question selection is based on a function that estimates the probability of a correct answer to a particular question, ultimately leading to an estimation of the student’s level of knowledge. The question (amongst the pool of the questions that have not been posed yet) with the highest probability will be posed. AthenaQTI does not use functions to estimate any parameter; rather the author is given the flexibility to express his/her didactical philosophy and methods through the creation or appropriate rules. Furthermore, our tool supports a wider range of question types than SIETTE, which seems to handle mainly multiple-choice. The most important advancement of AthenaQTI lies in the fact that it fully conforms to the IMS_QTI standard, making it very powerful since interoperability is currently a vital issue.

AthenaQTI16 Evaluation The tool was given for testing to 11 people who where involved in education (either students or instructors). They were subsequently given questionnaires to complete General findings: Good navigation Nice, simple interface Straightforward meaning of forms/fields Slight difficulty in understanding immediately the rule creation More explanatory directions would be useful.