1. Best Practices in Quality Test Administration
Lessons learned from studies conducted during the rollout of PARCC

2. Official purposes of the TA study
Teachers and Administrators are prepared to administer PARCC as intended
Administrators have access to necessary training resources and can find them
Technology improves and facilitates the assessment experience for all
Students respond to online assessments as intended
Massachusetts Department of Elementary and Secondary Education

3. Why seek a third party study of test administration policy
PARCC in 2011: 20+ different states with 20 different perspectives
In Massachusetts:
Our state test drive of PARCC ( )
Our standard for adoption: PARCC must be as good as or better than MCAS
Strong institutional support and opposition for the common core and PARCC casts doubt on state and main contractor findings
Relatively poor readiness for online testing in our state
Differing views about young students’ facility with keyboards and technology
Massachusetts Department of Elementary and Secondary Education
Massachusetts Department of Elementary and Secondary Education

4. What did we need to learn in our state?
Are our schools and students ready for online testing?
Did testing glitches affect most students?
Did the students still prefer online testing despite the novel difficulties?
Was PARCC measuring the content taught in our classrooms?
Did time limits affect testing?
MCAS tests are NOT timed, PARCC tests are.
Is an online test as good or better than a paper test?
Massachusetts Department of Elementary and Secondary Education

5. What did we actually learn?: As a consortium (2015 operational)
Training materials and online tutorials need to be available much earlier
2/3rds did not consult tutorials prior to phase 1 of testing; 1/3rd did not consult tutorials prior to phase 2
Preparing for PARCC was more disruptive and time consuming than other assessments
Most (71% Math; 78% ELA) said all but a few of the questions were about topics they learned in school
A majority (51%) said the material was more difficult than their schoolwork in Math; 38% in ELA
Massachusetts Department of Elementary and Secondary Education

6. What did we actually learn?: As a state
By all survey measures Massachusetts schools trailed other states in readiness for computer-based testing
81% of MA test administrators had never given a computer-based test before the PARCC field test of 2014; down to 68% in 2015
48% said their training did not prepare them to solve basic problems related to technology during the 2014 field test; same in 2015
Massachusetts Department of Elementary and Secondary Education

7. What did we learn (continued)
During the field test just 10% of the online test administrators reported “no technology-related problems occurred”; 8% in 2015
Most of the problems encountered were resolved quickly and easily.
Online tutorials for training TAs were generally described as effective, but one third of test administrators did not complete a tutorial.
Massachusetts Department of Elementary and Secondary Education

8. What did MA students say about taking PARCC field tests online?
The majority of students expressed a preference for the online tests over paper-and-pencil tests (74% ELA, 56% Math)
All or most test questions asked students about things they had learned in school this year (87% in ELA, 70% in math)
28% reported that the ELA test was more difficult than their school work; 61% of students reported that the math test was more difficult
61% of students said the math test was harder than math school work; 28% said the ELA test was harder than ELA school work. Conversely, only 5% of students said the math test was easier than their school work, compared to 17% of students for ELA.
70% of students said most or all math questions asked about things they had learned this year, compared to 87% in ELA.
64% of students took one or more practice math tests on computer; 62% for ELA
Problems on the math test:
The computer stopped working: 6%
The computer worked slowly: 12%
I had a hard time dragging or moving things on the screen: 6%
I had a hard time making changes to my answers: 22%
83% (Math) to 94% (ELA) of students responded that they had enough time
Problems on the ELA test: The computer stopped working: 7%
The computer worked slowly: 10%
I had a hard time dragging or moving things on the screen: 10%
I had a hard time making changes to my answers: 4%
Massachusetts Department of Elementary and Secondary Education
Massachusetts Department of Elementary and Secondary Education

9. What did students say about preparing for PARCC online?
2 out of 3 students took practice test(s) on a computer or tablet to get ready
Approx. 87% use a computer or tablet at least once a week at home
Approx. 60% use a computer or tablet at least once a week in school
Massachusetts Department of Elementary and Secondary Education

10. What did students say about taking the PARCC PBA field test online?
How hard it is to type answers depends on the subject area:
ELA: 87% found it easy to type answers
Math: 58% found it easy to type answers; % found it hard
Many students experienced a technology related problem
ELA: 31% reported a problem during testing
Math: 46% (22% had trouble changing answers)
Massachusetts Department of Elementary and Secondary Education

11. What did teachers say about taking PARCC online?
81% had never before administered a computer based test
Most students were able to complete the test without needing the additional time allotted
29% said none of the students in their class needed additional time
44% said less than 1/3 needed additional time
28% said about half to the majority needed the additional time
48% said the training did not prepare them to solve basic problems related to technology; 46% agreed that the training was adequate
Massachusetts Department of Elementary and Secondary Education

12. What other test administration studies were needed to support PARCC?
Comparability research for states like Louisiana using some, but not all PARCC content.
Research into test validity
Research to support standard setting
Cognitive Labs
Accommodation policies
Massachusetts Department of Elementary and Secondary Education

13. What did we do with this information?
We doubled down on our efforts to train schools and teachers for online testing.
The majority of MA districts continue to test with paper, but all are committed to online testing by 2019
Massachusetts’ board of education voted to develop a hybrid, next generation test
With greater local control over accommodation and timing policies
Remaining in PARCC with a commitment to online testing
Massachusetts Department of Elementary and Secondary Education