1. A comparison of online and paper-based test results Sanja Čandrlić, sanjac@inf.uniri.hr sanjac@inf.uniri.hr Martina Ašenbrener Katić, masenbrener@inf.uniri.hrmasenbrener@inf.uniri.hr

2. Presentation outline Introduction Research and testing environment Research results Conclusion Future work 214th Workshop SE Education and Reverse EngineeringSinaia

3. Introduction Comparative research of paper-based testing vs. online testing ICT more and more used, also in education –Many advantages –Knowledge assesment as an important part of educational process The aim of the research: –Can online tests replace traditional paper based tests? –What types of questions should online test contain? Online test 314th Workshop SE Education and Reverse EngineeringSinaia

4. Online testing Reasons to move to online testing? –Automated correction -> more students, less time, the computer program evaluates the tests (for some questions) Objective & subjective question types –True/false –Multiple choices –Connections –Essay –Short answer 414th Workshop SE Education and Reverse EngineeringSinaia

5. Question types 514th Workshop SE Education and Reverse EngineeringSinaia

6. Research environment - a customized Moodle solution, adapted to the needs of UniRi 1231 tests included in this research 3 courses in the field of information systems design: –Data Modeling, –Process Modeling –Information Systems 614th Workshop SE Education and Reverse EngineeringSinaia

7. Research environment Online testing since 2009 –first online tests included a certain percentage of subjective questions –Data Modeling, Process Modeling Online testing since 2011 – intention to ease the correction –online tests with only objective questions –Information Systems course –the number of students and teachers' teaching load increased Can online tests consisting exclusively of objective questions completely replace paper-based tests? 714th Workshop SE Education and Reverse EngineeringSinaia

8. Test properties and participants 15 – 25 questions from different categories in online tests and paper- based tests Time limit to solve the tests, both for online and paper-based tests Shuffled questions in online tests Online tests set up to be solved in a ‘safe window’ Single major and Double major program of informatics 8 14th Workshop SE Education and Reverse EngineeringSinaia

9. Tests set up 9 14th Workshop SE Education and Reverse EngineeringSinaia

10. Questions and points set up 10 14th Workshop SE Education and Reverse EngineeringSinaia 50% points 70% 30% 100%

11. Main hypothesis Descriptive statistical analysis: High variance coefficient for paper-based test results – arithmetic mean cannot be considered as representative central tendency indicator Low value of quartile deviation coefficient – median is a better indicator of central tendency H: There is no significant difference in median values of the results achieved during MudRi online tests and traditional paper-based tests 1114th Workshop SE Education and Reverse EngineeringSinaia

12. Test results 1214th Workshop SE Education and Reverse EngineeringSinaia 1.Box-plot diagrams for Data Modeling course (left online tests, right paper-based tests) Box-plot diagram for DM course (left online, right paper- based) Box-plot diagram for PM course (left online, right paper- based) Box-plot diagram for IS course (left online, right paper- based) 1.Box-plot diagrams for Information Systems course (left online tests, right paper-based tests)

13. Research results Normality analysis indicated the use of nonparametric statistic methods for hypothesis testing Mann-Whitney test: Data Modeling course – no significant statistic difference in median values (p=0.2401) between online and paper-based tests. Process Modeling course – no significant statistic difference in median values (p= 0.3777) between online and paper-based tests. Information Systems course – the difference is extremely significant (p=0.0066). 1314th Workshop SE Education and Reverse EngineeringSinaia

14. Conclusion For DM & PM the H is confirmed For IS the H is rejected IS – better results -> lower levels of knowledge, based on recognition Paper-based tests: –higher max points -> subjective factor, partially correct, “hidden knowledge” –min points = 0 - > students give up 1414th Workshop SE Education and Reverse EngineeringSinaia

15. Conclusion (2) Transition from paper-based to online tests: Essay-> objective q. To explain, to describe,... 30 % of subjective q., manual correction Advantages of online tests: –automated correction, –easy to use (after forming the question bank) –immediate test results for the students –students prefer to use the keyboard –comparative overview of test results (for the teacher) Advantages of paper-based tests: –easy to prepare –„drawing” questions 1514th Workshop SE Education and Reverse EngineeringSinaia

16. Future work Efforts to further improve methods for online assessment Revising the question bank Analysis of students’ answers Too easy questions (100% correct) Too hard questions (potentially unclearly stated, incomprehensible to the students) Plan to update the question bank with sets of different types of questions for formative assessment – to test their knowledge during preparation for the tests 1614th Workshop SE Education and Reverse EngineeringSinaia

17. Thank you for your attention! Questions? 1714th Workshop SE Education and Reverse EngineeringSinaia