Best Practices in Quality Test Administration Lessons learned from studies conducted during the rollout of PARCC

Official purposes of the TA study Teachers and Administrators are prepared to administer PARCC as intended Administrators have access to necessary training resources and can find them Technology improves and facilitates the assessment experience for all Students respond to online assessments as intended Massachusetts Department of Elementary and Secondary Education

Why seek a third party study of test administration policy PARCC in 2011: 20+ different states with 20 different perspectives In Massachusetts: Our state test drive of PARCC (2014-2016) Our standard for adoption: PARCC must be as good as or better than MCAS Strong institutional support and opposition for the common core and PARCC casts doubt on state and main contractor findings Relatively poor readiness for online testing in our state Differing views about young students’ facility with keyboards and technology Massachusetts Department of Elementary and Secondary Education Massachusetts Department of Elementary and Secondary Education

What did we need to learn in our state? Are our schools and students ready for online testing? Did testing glitches affect most students? Did the students still prefer online testing despite the novel difficulties? Was PARCC measuring the content taught in our classrooms? Did time limits affect testing? MCAS tests are NOT timed, PARCC tests are. Is an online test as good or better than a paper test? Massachusetts Department of Elementary and Secondary Education

What did we actually learn?: As a consortium (2015 operational) Training materials and online tutorials need to be available much earlier 2/3rds did not consult tutorials prior to phase 1 of testing; 1/3rd did not consult tutorials prior to phase 2 Preparing for PARCC was more disruptive and time consuming than other assessments Most (71% Math; 78% ELA) said all but a few of the questions were about topics they learned in school A majority (51%) said the material was more difficult than their schoolwork in Math; 38% in ELA Massachusetts Department of Elementary and Secondary Education

What did we actually learn?: As a state By all survey measures Massachusetts schools trailed other states in readiness for computer-based testing 81% of MA test administrators had never given a computer-based test before the PARCC field test of 2014; down to 68% in 2015 48% said their training did not prepare them to solve basic problems related to technology during the 2014 field test; same in 2015 Massachusetts Department of Elementary and Secondary Education

What did we learn (continued) During the field test just 10% of the online test administrators reported “no technology-related problems occurred”; 8% in 2015 Most of the problems encountered were resolved quickly and easily. Online tutorials for training TAs were generally described as effective, but one third of test administrators did not complete a tutorial. Massachusetts Department of Elementary and Secondary Education

What did MA students say about taking PARCC field tests online? The majority of students expressed a preference for the online tests over paper-and-pencil tests (74% ELA, 56% Math) All or most test questions asked students about things they had learned in school this year (87% in ELA, 70% in math) 28% reported that the ELA test was more difficult than their school work; 61% of students reported that the math test was more difficult 61% of students said the math test was harder than math school work; 28% said the ELA test was harder than ELA school work. Conversely, only 5% of students said the math test was easier than their school work, compared to 17% of students for ELA. 70% of students said most or all math questions asked about things they had learned this year, compared to 87% in ELA. 64% of students took one or more practice math tests on computer; 62% for ELA Problems on the math test: The computer stopped working: 6% The computer worked slowly: 12% I had a hard time dragging or moving things on the screen: 6% I had a hard time making changes to my answers: 22% 83% (Math) to 94% (ELA) of students responded that they had enough time Problems on the ELA test: The computer stopped working: 7% The computer worked slowly: 10% I had a hard time dragging or moving things on the screen: 10% I had a hard time making changes to my answers: 4% Massachusetts Department of Elementary and Secondary Education Massachusetts Department of Elementary and Secondary Education

What did students say about preparing for PARCC online? 2 out of 3 students took practice test(s) on a computer or tablet to get ready Approx. 87% use a computer or tablet at least once a week at home Approx. 60% use a computer or tablet at least once a week in school Massachusetts Department of Elementary and Secondary Education

What did students say about taking the PARCC PBA field test online? How hard it is to type answers depends on the subject area: ELA: 87% found it easy to type answers Math: 58% found it easy to type answers; 41% found it hard Many students experienced a technology related problem ELA: 31% reported a problem during testing Math: 46% (22% had trouble changing answers) Massachusetts Department of Elementary and Secondary Education

What did teachers say about taking PARCC online? 81% had never before administered a computer based test Most students were able to complete the test without needing the additional time allotted 29% said none of the students in their class needed additional time 44% said less than 1/3 needed additional time 28% said about half to the majority needed the additional time 48% said the training did not prepare them to solve basic problems related to technology; 46% agreed that the training was adequate Massachusetts Department of Elementary and Secondary Education

What other test administration studies were needed to support PARCC? Comparability research for states like Louisiana using some, but not all PARCC content. Research into test validity Research to support standard setting Cognitive Labs Accommodation policies Massachusetts Department of Elementary and Secondary Education

What did we do with this information? We doubled down on our efforts to train schools and teachers for online testing. The majority of MA districts continue to test with paper, but all are committed to online testing by 2019 Massachusetts’ board of education voted to develop a hybrid, next generation test With greater local control over accommodation and timing policies Remaining in PARCC with a commitment to online testing Massachusetts Department of Elementary and Secondary Education