Cheryl Ng Ling Hui Hee Jee Mei, Ph.D Universiti Teknologi Malaysia Teacher Practices in Scoring Multiple-Choice Items, Interpreting Test Scores & Performing Item Analysis Cheryl Ng Ling Hui Hee Jee Mei, Ph.D Universiti Teknologi Malaysia Introduction – M.Ed (Measurement and Evaluation) final semester; carried out as part of my master’s project

Introduction Multiple choice items are widely used to assess student learning. They feature prominently in public examinations and likewise in teacher-made-tests. Research Project – to develop a tool to assist teachers in scoring multiple choice items, interpreting test scores and performing item analysis. Thus, this preliminary needs analysis study was conducted…. Easy to score objectively – more items can be included in a single test – better coverage of the scope of study However raw score alone does not give us much information of educational value

Aim and Purposes of the Study Objectives: Identify teachers’ current practice and the challenges they face in scoring multiple-choice items, interpreting test scores and performing item analysis; and Ascertain the prevalence of OMR scanners and item analysis software in Malaysian secondary schools. Aim: Findings from this preliminary study serve to inform subsequent development of a software to aid teachers in these three aspects.

Methodology Research Design: Survey Participants: 130 secondary school teachers serving in public schools across Malaysia. Instrument: Questionnaire with 3 sections: Section A: Demographic Profile – gender, location and type of school, teaching experience Section B: Teacher Practices – scoring OMR answer sheets, interpreting test scores, performing item analysis Section C: OMR Scanner and Item Analysis Software – availability in school, willingness to invest in one etc.

Methodology Precedures: Questionnaires were disseminated to various schools through the help of colleagues and friends. An online version of the questionnaire was created using a free online survey tool and the link was sent to potential respondents. The collected data were analysed using descriptive statistics.

Respondents’ Demographic Profile Number of respondents, N = 130 teachers Number of schools represented = 74 schools Teaching experience 0-33 years

Teachers’ Practices : Scoring Teachers’ estimates on the amount of time (in seconds) needed to score one OMR answer sheet (assuming that there are 40 items), count the number of correct answers and convert the raw score to percentages :- Table 1 Means and Standard Deviations for Teachers’ Practices in Scoring Mean Standard Deviation Scoring OMR answer sheet (N=130) 51.52 37.69 Counting the number of correct answers (N=127) 27.29 26.43 Converting the raw score to percentage (N=114) 22.40 25.87 Total 101.21 As the number of items differ across subjects and levels, the respondents were told to assume that there were 40 items. Scoring + Counting + Converting = 101.21 seconds; 40 students = 1hr 8 mins / class

Teachers’ Practices : Scoring Teachers have to spend a substantial amount of time to process OMR answer sheets when they do it manually. Based on the mean of teachers’ estimates, 1 OMR answer sheet takes around 101.21 seconds (i.e. 1.69 minutes) to mark. 5 classes x 40 students = 200 OMR answer sheets 200 sheets x 1.69 minutes = 5 hours 38 minutes

Teachers’ Practices : Scoring They would need to spend even more time if they were to do the following (commented by individual teachers): “Check if two answers were marked for one question” (3s) “Make a note of items answered wrongly by students” (120s) “Recheck for errors in marking” (20s) “Key in the data” (60s)

School Requirements to Interpret Test Scores The respondents were asked to indicate on a five-point Likert scale (every time, often, occasionally, seldom and never) how frequently they were required by their schools to interpret test scores. The result shows that school requirements vary from school to school. Mean = 3.09, S.D. = 1.406

Teachers’ Practices – Interpreting Test Scores Every time Often Occasionally Seldom Never (5) (4) (3) (2) (1) Mean S.D. Determine class average 25 20 24 28 33 2.82 1.462 19% 15% 18% 22% 25% Determine standard deviation 5 23 69 1.84 1.091 4% 21% 53% Convert raw scores to standard scores 15 8 12 30 65 2.06 1.374 12% 6% 9% 23% 50% Only a third of the respondents (34%) consistently determine the class mean. Almost half of them (47%) seldom or never do so. Almost three quarters of the respondents indicated that they seldom or never determine the standard deviation (74%) and convert raw scores to standard scores (73%).

Challenges in Interpreting Test Scores “I struggle with time constraints.” (106 respondents, 81.54%) “I do not know how to do it.” (44 respondents, 33.85%) Other comments: “Too much paper work.” “Do not do it as I find the result irrelevant.” “Do not face any challenges (not required to do it / SU Peperiksaan does it for teachers)” Time constraint is the greatest challenge that hinders teachers from interpreting test scores. (106 respondents , 81.54%) Another one third of the respondents reported that they did not know how to interpret test scores. (44 respondents, 33.85%)

School Requirements to Perform Item Analysis School requirements to perform item analysis vary from school to school. Comparatively, more schools require teachers to perform item analysis. Mean = 3.44, S.D. = 1.341

Teachers’ Practices – Performing Item Analysis Every time Often Occasionally Seldom Never (5) (4) (3) (2) (1) Mean S.D. Determine the item difficulty of each item 19 41 30 24 16 3.18 1.256 15% 32% 23% 19% 12% Determine the discrimination index of each item 6 32 36 40 2.32 1.169 5% 24% 28% 31% 70% of the respondents determine the item difficulty (every time 15%; often 32%; occasionally 23%). More than half (59%) seldom or never determine the discrimination index; only 5% did so for every test.

Challenges in Performing Item Analysis “I struggle with time constraints.” (115 respondents, 88.46%) “I do not know how to do it.” (42 respondents, 32.31%) Other comments: “The information from item analysis is considered irrelevant by students.” “Do not do it as I find the result irrelevant.” “Do not face any challenges (not required to do it)” Similarly, time constraint is the greatest challenge that hinders teachers from performing item analysis. (115 respondents , 88.46%) Another one third of the respondents reported that they did not know how to do it. (42 respondents, 32.31%)

Prevalence of OMR Scanners & Item Analysis Software Out of the 74 schools represented in this study, only 13 schools (18%) have invested in an OMR scanner and 16 schools (22%) provide teachers with item analysis software.

Teachers’ Willingness to Invest in the Tool

Teachers’ Willingness to Invest in the Tool Among those who are willing to invest (N=84), the respondents gave a wide range of answers when asked how much they are willing to pay for such a tool. Their answers range from RM5 (min) to RM3000 (max). The most common response was RM100 (mode, 22 respondents). The median was slightly higher at RM150. The mean was RM337.65, with a standard deviation of RM512.97. Max = RM3000 Mean = RM337.65 Median = RM150 Mode = RM100 Min = RM5

Discussion Teachers have to spend a substantial amount of time to score OMR answer sheets, interpret test scores and perform item analysis if they do it manually. Time constraint is the greatest challenge that hinder teachers from interpreting test scores and performing item analysis. Therefore, a tool to aid teachers in scoring OMR answer sheets would greatly reduce teachers’ burden and allow them more time to focus on enhancing test items and improving their teaching practice. However, the tool should be made available to teachers at an affordable price range (RM100-RM350).

Discussion Interpreting test scores and performing item analysis are crucial aspects of the assessment process. Interpreting test scores will give insights to the teachers as to their students’ performance in the test. It also helps to highlight areas of weakness that need to be addressed. Information from the item analysis is crucial for improving the item quality. From the item analysis results, teachers can decide whether to retain, modify or discard test items. Functional and modified items could be stored for future use. (Gronlund and Linn, 1990; Reynolds, Livingston and Willson, 2010) .

Discussion Requirements to interpret test scores and perform item analysis vary from school to school. If teachers neglect to interpret test scores and perform item analysis, then the value of the assessment process is greatly reduced. Suggestion: Teachers should be required to interpret test scores and perform item analysis as part of their professional practice. However, they should be released from the burden of clerical work (i.e. manually scoring OMR answer sheets and keying in the data) which could be done effectively with the assistance of an OMR processing and item analysis tool.

Discussion In conclusion, there is a need to develop an affordable and teacher-friendly tool to help teachers to process OMR answer sheets and provide them with information on students’ performance in the test as well as the quality of the items.

Universiti Teknologi Malaysia ~Thank you~ Cheryl Ng Ling Hui Universiti Teknologi Malaysia cherlinghui@gmail.com http://intoherworldofteaching.wordpress.com