Educational Testing Service 7/31/2019 Examining the Role of Students in the Formative Assessment Process: The Relationship between Teacher Beliefs and Practices and Student Beliefs Christine J. Lyon Elia Mavronikolas Jilliam Joe Educational Testing Service Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

Formative Assessment “a process in which information about learning is evoked and then used to modify the teaching and learning activities in which teachers and students are engaged” (Black, Harrison, Lee, & Wiliam, 2003)

Formative Assessment “a process in which information about learning is evoked and then used to modify the teaching and learning activities in which teachers and students are engaged” (Black, Harrison, Lee, & Wiliam, 2003)

Classroom Contract “unspoken agreement between teachers and students about who is responsible for learning, whose voice is valued, and how the pace of work is determined” (Lyon, Wylie, and Mavronikolas, 2011) 7/31/2019 Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

Intersecting Constructs The classroom contract must be consistent with the core expectations, beliefs, and practices of formative assessment . All students can achieve. Students can function as active learners. Students need explicit supports, routines, and opportunities to engage in behaviors consistent with those beliefs. 7/31/2019 Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

Research Questions and Hypotheses Student Beliefs and Expectations Student Attitudes towards Mathematics and Science Teacher Beliefs and Expectations Moderate Teacher Practices High 7/31/2019 Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

Sample and Intervention Four participating districts 202 teachers completed a two-year professional development program Introductory workshop Teacher learning community meetings More mathematics teachers (57%) than science teachers (43%)

Teacher Measures Assessment Practices Survey (APS): Measures the frequency and quality of teachers’ formative assessment practice

Teacher Measures Classroom Contract Beliefs and Attitudes Survey (CCBAS): Includes 10 likert-scale questions related to the classroom contract. Discussing correct and incorrect answers with students deepens their understanding. If students are given sample student work they will copy the samples. 7/31/2019 Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

Student Measures Student Attitudes Survey: Measures students’ attitudes toward learning, perceptions of control over learning, and other social-emotional variables. Attitudes Experiential Attitudes: “Studying math makes me nervous” Instrumental Attitudes: “I can be successful in life without knowing math” Classroom contract: “The teacher takes time to explain important ideas so that everyone understands” “At the start of the lesson, I understand what we are going to learn” 7/31/2019 Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

Analysis Plan Hierarchical Linear Modeling Accounts for students nested within classrooms/teachers Three models Unconditional (intercept-only) Model Random Coefficients Model with Level-1 Predictor Student Pre-test score Random Coefficients Model with Level-2 Predictors Teacher APS score CCBAS score Students Classrooms/Teachers Intervention T1 S1 S2 … Sn T2 7/31/2019 Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

Results Model Description Mathematics Science AE AI CC Yes No N/A Unconditional: Provides a baseline for more complex comparisons Is there a support for a Multi-level Model (if no, use simple regression)? Yes No Random Coefficients Model: Includes pre-test as a level-1 predictor Is pretest score a significant predictor of post-test score? Is there support for level-2 predictors? Random Coefficients Model with level-2 predictors: Includes CCBAS and APS as level 2 predictors Does teacher score on the APS or CCBAS significantly predict student post-test score after controlling for pre-test? N/A After controlling for the pretest, signficiant variation was still observed between classrooms. Level 2 predictors are trying to explain the differences between classrooms and investigates teacher beliefs and practices as two potential sources. 7/31/2019 Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

Results Model Description Mathematics Science AE AI CC Yes No N/A Unconditional: Provides a baseline for more complex comparisons Is there a support for a Multi-level Model (if no, use simple regression)? Yes No Random Coefficients Model: Includes pre-test as a level-1 predictor Is pretest score a significant predictor of post-test score? Is there support for level-2 predictors? Random Coefficients Model with level-2 predictors: Includes CCBAS and APS as level 2 predictors Does teacher score on the APS or CCBAS significantly predict student post-test score after controlling for pre-test? N/A After controlling for the pretest, signficiant variation was still observed between classrooms. Level 2 predictors are trying to explain the differences between classrooms and investigates teacher beliefs and practices as two potential sources. 7/31/2019 Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

Results Model Description Mathematics Science AE AI CC Yes No N/A Unconditional: Provides a baseline for more complex comparisons Is there a support for a Multi-level Model (if no, use simple regression)? Yes No Random Coefficients Model: Includes pre-test as a level-1 predictor Is pretest score a significant predictor of post-test score? Is there support for level-2 predictors? Random Coefficients Model with level-2 predictors: Includes CCBAS and APS as level 2 predictors Does teacher score on the APS or CCBAS significantly predict student post-test score after controlling for pre-test? N/A After controlling for the pretest, signficiant variation was still observed between classrooms. Level 2 predictors are trying to explain the differences between classrooms and investigates teacher beliefs and practices as two potential sources. 7/31/2019 Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

Overview of Findings Student attitudes were less consistent with the underlying theory of formative assessment at the end of the first year of the intervention. There was significant classroom-to-classroom variation in student responses. Student pre-test scores were a significant predictor of post-test scores. However, teacher beliefs and practices did not predict student attitudes towards mathematics and science or student attitudes towards the classroom contract after controlling for pre-test score. 7/31/2019 Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

Potential Explanations High school students have been in the educational context for between 9-12 years and attitudes and expectations may be set. It may take longer than a year to change them. Teachers were in their first year of the intervention and were trying out new things throughout the year. Even though practice changed significantly it might not have been consistent enough or long enough to impact student attitudes. Teacher self report may not actually accurately represent practice. 7/31/2019 Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

Next Steps More validity evidence for the measurement of behaviors and attitudes is needed. Additional studies to understand the source of classroom-to-classroom variation on attitudes is needed. 7/31/2019 Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

7/31/2019 Contact Information Christine Lyon clyon@ets.org Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

Appendix Model specifications and findings for all models and content areas. 7/31/2019 Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

Descriptive statistics for mathematics Shows changes in student attitudes from pre-test to post-test Student attitudes decreased across all three scales at post-test Scale Mean (Std.dev) Mean Difference T (p value) Internal Consistency Reliability Pre Post AE 19.51 (5.069) 19.24 (5.183) -0.27 1.392 (.164) .818 .851 AI 23.79 (3.928) 22.86 (4.177) -0.93 .605 (<.001) .827 .836 CC 27.89 (5.313) 27.56 (5.391) -0.33 1.315 (.189) .763 .781 The decrease for instrumental attitudes was significant 7/31/2019 Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

Descriptive statistics for mathematics Measures the extent to which items within a scale are consistently measuring a construct together Descriptive statistics for mathematics Scale Mean (Std.dev) Mean Difference T (p value) Internal Consistency Reliability Pre Post AE 19.51 (5.069) 19.24 (5.183) -0.27 1.392 (.164) .818 .851 AI 23.79 (3.928) 22.86 (4.177) -0.93 .605 (<.001) .827 .836 CC 27.89 (5.313) 27.56 (5.391) -0.33 1.315 (.189) .763 .781 Supports the use of sub-scores as opposed to total scores This supports the results of the factor analysis and the use of sub-scores as opposed to Consistency of .700 or above is considered good for group comparisons 7/31/2019 Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

Descriptive statistics for science Student attitudes decreased across all three scales at post-test Scale Mean (Std.dev) Mean Difference T (p value) Internal Consistency Reliability Pre Post AE 20.31 (4.476) 20.00 (4.498) -0.31 1.649 (.100) .809 .802 AI 21.15 (3.955) 20.73 (4.293) -0.42 2.486 (<.05) .804 .851 CC 31.05 (5.628) 30.49 (6.181) -0.56 2.086 (<.05) .768 The decrease for instrumental attitudes and classroom contract was significant Internal consistency reliability was above .70 for all three scales supporting the use of sub-scores 7/31/2019 Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

Mathematics Results 7/31/2019 Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

Mathematics Results Unconditional (intercept-only) Model Provides a baseline for comparison with more complex models Addresses the extent of classroom-to-classroom variation Addresses the extent of student-to-student variation within a teacher 7/31/2019 Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

Mathematics Results: Unconditional Model The average post-test score for a student’s classroom Mathematics Results: Unconditional Model The total proportion of the variance among classrooms was 0.13 or 87% of the variance was among students within classrooms Scale Fixed Effect Coefficient se AIC ICC (among teachers) AE Average classroom mean 18.1962 0.3794 3067.9 .13 Random Effect Variance Component Classroom mean, u0j 3.4964 Level-1 effect, rij 22.835   AI 22.8332 0.2529 2885.1 .06 1.0257 16.4776 CC 27.4861 0.2897 3148 .03 0.8759 28.1908 The amount of teacher-to-teacher variation in average student post-test scores Degree to which nesting is observed in the data. ICCs between 0.02-0.18 provide a rationale for multi-level models with nesting 7/31/2019 Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

Mathematics Results: Unconditional Model Scale Fixed Effect Coefficient se AIC ICC (among teachers) AE Average classroom mean 18.1962 0.3794 3067.9 .13 Random Effect Variance Component Classroom mean, u0j 3.4964 Level-1 effect, rij 22.835   AI 22.8332 0.2529 2885.1 .06 1.0257 16.4776 CC 27.4861 0.2897 3148 .03 0.8759 28.1908 Based on the unconditional model we claim that teacher-to-teacher variation in student posttest scores for AE and AI is significantly different from 0 indicated that the random coefficients model is warranted. For CC the teacher-to teacher variation is not significantly different from 0 and suggests that a simple regression can be fitted for all classrooms. 7/31/2019 Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

Mathematics Results Random Coefficients Model Includes a level-1 predictor: pre-test score Explores if there is significant classroom-to-classroom variation in post-test scores after controlling for pre-test scores 7/31/2019 Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

Mathematics Results: Random Coefficients Model Less than .05 indicates that pre-test score is a significant predictor of post-test score Scale Fixed Effect Coefficient se T p value AIC AE Average classroom mean 19.15 0.2901 66 <.0001 2832.1 Pre-test 0.61 0.03449 17.54 Random Effect Variance Component Classroom mean, u0j 1.95 0.729 Level-1 effect, rij 14.2928 0.933   AI 22.86 0.1495 152.9 2676.5 0.63 0.0381 16.55 11.3362 0.7134 CC Average post-test score 27.56 .21548 127.90 .44 .04059 10.94 .1917 True for all three scales True for all three scales True for all three scales For all three models, pretest is a significant predictor of post test score. 7/31/2019 Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

Mathematics Results: Random Coefficients Model Indicates the amount of variance in average post-test scores that remains among classrooms after controlling for the pre-test scores Scale Fixed Effect Coefficient se T p value AIC AE Average classroom mean 19.15 0.2901 66 <.0001 2832.1 Pre-test 0.61 0.03449 17.54 Random Effect Variance Component Classroom mean, u0j 1.95 0.729 Level-1 effect, rij 14.2928 0.933   AI 22.86 0.1495 152.9 2676.5 0.63 0.0381 16.55 11.3362 0.7134 CC Average post-test score 27.56 .21548 127.90 .44 .04059 10.94 .1917 Variation still remained supporting the random co-efficients model with level-2 predictors Classroom-to-classroom variance was reduced to near 0 after controlling for pre-test For all three models, pretest is a significant predictor of post test score. 7/31/2019 Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

Mathematics Results Random Coefficients Model with level-2 predictors Includes two level-2 classroom predictors Teacher APS score Teacher CCBAS score Explores the amount of variance explained by these teacher attributes after controlling for student pre-test score 7/31/2019 Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

Mathematics Results: Level 2 predictors Fixed Effect Coefficient se t p value AIC Average classroom mean 17.36 2.615 6.64 <.0001 2824.1 Pre-test 0.60 0.035 17.42 APS 0.02 0.034 0.57 0.5723 CCBAS 0.04 0.078 0.47 0.6378 Random Effect Variance Component Pre-test, u1j 2.00 0.788 Level-1 effect, rij 14.36 0.942   Indicates the amount of variance in average post-test scores that is accounted for by each predictor The amount of variance predicted by APS and CCBAS is very small 7/31/2019 Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

Mathematics Results: Level 2 predictors Indicates whether the predictor is a significant predictor of classroom-to-classroom variation Fixed Effect Coefficient se t p value AIC Average classroom mean 17.36 2.615 6.64 <.0001 2824.1 Pretest 0.60 0.035 17.42 APS 0.02 0.034 0.57 0.5723 CCBAS 0.04 0.078 0.47 0.6378 Random Effect Variance Component Pre-test, u1j 2.00 0.788 Level-1 effect, rij 14.36 0.942   Neither APS or CCBAS is a significant predictor 7/31/2019 Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

Science Results 7/31/2019 Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

Science Results: Unconditional Model Scale Fixed Effect Coefficient se AIC ICC AE Average classroom mean 19.9238 .28774 2737.7 .06 Random Effect Variance Component Classroom mean, u0j 1.1568 Level-1 effect, rij 19.1132   AI 20.6447 .2491 2698.7 .03 .6449 17.8141 CC 30.4601 .3926 3035.1 2.1210 36.1192 7/31/2019 Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

Science Results: Random Coefficients Model Scale Fixed Effect Coefficient se t p value AIC AE Average classroom mean 20.00 .2181 33 <.0001 2550 Pre-test .5826 .03776 15.43 Random Effect Variance Component Classroom mean, u0j .5460 .3742 Level-1 effect, rij 12.8140 .8650   CC 30.49 .2776 109.85 2895.9 .5699 .04353 13.09 .04917 .5500 27.2729 1.8301 AI Average post-test score 20.73 .15738 131.72 .6614 .03984 16.60 .3711 7/31/2019 Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

Science Results: Level 2 predictors Scale Fixed Effect Coefficient se t p value AIC AE Average classroom mean 18.8565 2.0126 9.37 <.0001 2559.1 Pre-test 0.5809 0.038 15.28 APS 0.006 0.024 0.25 0.8027 CCBAS 0.028 0.061 0.46 0.6490 Random Effect Variance Component Pre-test, u1j 0.6237 0.4165 Level-1 effect, rij 12.8178 0.8659   CC 28.5933 2.3510 12.16 2902.8 0.5704 0.043 13.12 -0.0318 -1.11 0.2764 0.0840 0.071 1.18 0.2462 0.3245 0.5835 27.4046 1.8481 7/31/2019 Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.