Multiple answer questionnaires can help students in self-evaluating their preparation to the exams. Fabrizio Fontana and Matteo Martini Università degli Studi G. Marconi, Italy

DL growth In the last decade, we assisted to a large increase of university and professional courses delivered online This is due to the advantages of DL in terms of time, availability, etc. But also to the always increasing quality of these courses pushed up from WEB evolution and collaborative learning

An important problem: University Drop-Out Rate

Drop-out rate Students drop-out rate is too high especially for such countries/courses Average value for OCSE countries around 30% (too high) Italy and US have a drop-out rate greater than 50% (real unsolved problem)

Drop-out rate: Italy Dropout rate reach 55% considering all courses and cycles Almost 50% dropout in first year! Data from Italian ministry for three years first cycle bachelor graduate Only 23% graduate in 3 years Drop-out Almost 6% is still studying after 9 years! Still studying

What can contrast this trend? From a survey conducted at Bari University*: Better quality of learning objects Specified solution for workers-student Greater availability of teachers Personalized tutoring *http://www.dip-statistica.uniba.it/html/annali/2010/ANNALI9-19.pdf Suggestions from students of a Traditional University

University first year From Italian statistics, about 50% of students drop out university during first year Students can be afraid when passing from secondary school to university i.e. when passing from a continuous teacher control to an autonomous study

University first year Before starting, our students must learn how to study, i.e.: Develop a personal study-method Organize their work in complete autonomy Surely, gain a self-government is part of students growth but if not properly helped our students can drop out university.

First year drop-out Starting from these assumptions we want to realize a technological based self assessment activity to: Permit self evaluation during study Drive students to a better organized work Permit self evaluation before final exam Indirectly measure course quality and single arguments clearness

Assessment requirements Initial requirements can be summarized as: Activity with different questions every time Possibility to choose arguments Self-evaluation by students Direct teacher control

Proposed assessment Our proposal? An old, well known and clear approach: Multiple Answer Questionnaire … but technologically evolved!

Proposed assessment Adopted features: 15 questions with 4 possible answers, only 1 correct Test is considered “passed” if at least 11 answers are correct (>70%)

Assessment requirements Let’s go through Initial requirements … Activity with different questions every time Possibility to choose arguments Self-evaluation by students Direct teacher control

Assessment requirements Let’s go through Initial requirements … Activity with different questions every time Possibility to choose arguments Self-evaluation by students Direct teacher control

Max time per question This value has been chosen taking into account that some questions are not definitions but simple exercises requiring thinking and ability with formulas and theorems. Moreover, this time has been fixed to give to students the possibility to meditate on questions avoiding stress due to time but also to do not give the possibility to consult course materials.

Assessment requirements Let’s go through Initial requirements … Activity with different questions every time Possibility to choose arguments Self-evaluation by students Direct teacher control

Self-Evaluation When completed the questionnaire, a detailed report is showed on screen with: Details of corrects-wrongs and eventually right answers Possibility to send report to student private mail address (to meditate) and to teacher To push students in contacting teachers, 3 thresholds for the feedback have been identified: >70% correct, Test passed Between 30% and 70%, Test not passed with the possibility to send report to teacher Below 30% a pop-up appears strongly suggesting to send report and contacting teacher ** ** Statistically, answering randomly, 25% of questions could be correct.

Assessment requirements Let’s go through Initial requirements … Activity with different questions every time Possibility to choose arguments Self-evaluation by students Direct teacher control

Teacher control Teacher task is to contact students to explain errors, explain arguments not fully understood and/or suggest additional readings This creates a direct and efficient contact between teachers and students with a personalized tutoring during study

Quality implication This questionnaire represents: An help to students improving study efficiency and then pass-rate An help to teachers to measure single arguments quality

Quality implication Storing the entire set of completed questionnaires, important consideration can be done on both students community and course quality. Analyzing reports, instructors can evaluate to which extent learning materials are unfriendly to the students community and eventually decide to insert new learning objects.

Case Study: General Physics Right answer (%) Questions Sub-sample of 50 questions over 500

Case Study: General Physics Average Right answers 10% Average Right answers 50% Right answer (%) Questions Sub-sample of 50 questions over 500 This group is relative to a single arguments For specialist, 2.5 sigma far from the average Average 46% +/- 16%

Case Study: General Physics Even if we still have a small data sample of completed questionnaires (less than 1 academic year, 500 tests), we identified a hard understandable argument (Entropy tbp) Then … A new didactical unit has been already prepared and inserted into LMS to better explain this argument During next weeks we will re-analyze plot to measure students response.

Future development To help instructors not skilled or friendly with computer technology and/or statistical analysis, a specific web interface has been created. System hosted on a dedicated server and each teacher has his own credentials to access the server.

Future development A user friendly page permits to: Create different tests Indicate one or more correct questions Tag every question with an ID for arguments selection Customize the max time per question Statistically analyze single student response or the results of the entire students community

Conclusion Starting from the simple idea of a multiple answer questionnaire we develop a powerful technological learning tool Didactical utility of this product has been demonstrated To allow the use of this architecture in every course, a web interface has been realized simply permitting tests creation and students/course analysis