1. Educational Testing Service
7/31/2019
Examining the Role of Students in the Formative Assessment Process: The Relationship between Teacher Beliefs and Practices and Student Beliefs
Christine J. Lyon
Elia Mavronikolas
Jilliam Joe
Educational Testing Service
Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

2. Formative Assessment
“a process in which information about learning is evoked and then used to modify the teaching and learning activities in which teachers and students are engaged” (Black, Harrison, Lee, & Wiliam, 2003)

3. Formative Assessment
“a process in which information about learning is evoked and then used to modify the teaching and learning activities in which teachers and students are engaged” (Black, Harrison, Lee, & Wiliam, 2003)

4. Classroom Contract
“unspoken agreement between teachers and students about who is responsible for learning, whose voice is valued, and how the pace of work is determined” (Lyon, Wylie, and Mavronikolas, 2011)
7/31/2019
Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

5. Intersecting Constructs
The classroom contract must be consistent with the core expectations, beliefs, and practices of formative assessment .
All students can achieve.
Students can function as active learners.
Students need explicit supports, routines, and opportunities to engage in behaviors consistent with those beliefs.
7/31/2019
Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

6. Research Questions and Hypotheses
Student Beliefs and Expectations
Student Attitudes towards Mathematics and Science
Teacher Beliefs and Expectations
Moderate
Teacher Practices
High
7/31/2019
Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

7. Sample and Intervention
Four participating districts
202 teachers completed a two-year professional development program
Introductory workshop
Teacher learning community meetings
More mathematics teachers (57%) than science teachers (43%)

8. Teacher Measures
Assessment Practices Survey (APS): Measures the frequency and quality of teachers’ formative assessment practice

9. Teacher Measures
Classroom Contract Beliefs and Attitudes Survey (CCBAS): Includes 10 likert-scale questions related to the classroom contract.
Discussing correct and incorrect answers with students deepens their understanding.
If students are given sample student work they will copy the samples.
7/31/2019
Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

10. Student Measures
Student Attitudes Survey: Measures students’ attitudes toward learning, perceptions of control over learning, and other social-emotional variables.
Attitudes
Experiential Attitudes: “Studying math makes me nervous”
Instrumental Attitudes: “I can be successful in life without knowing math”
Classroom contract:
“The teacher takes time to explain important ideas so that everyone understands”
“At the start of the lesson, I understand what we are going to learn”
7/31/2019
Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

11. Analysis Plan Hierarchical Linear Modeling
Accounts for students nested within classrooms/teachers
Three models
Unconditional (intercept-only) Model
Random Coefficients Model with Level-1 Predictor
Student Pre-test score
Random Coefficients Model with Level-2 Predictors
Teacher APS score
CCBAS score
Students
Classrooms/Teachers
Intervention T1 S1 S2 … Sn T2
7/31/2019
Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

12. Results Model Description Mathematics Science AE AI CC Yes No N/A
Unconditional: Provides a baseline for more complex comparisons
Is there a support for a Multi-level Model (if no, use simple regression)?
Yes No
Random Coefficients Model: Includes pre-test as a level-1 predictor
Is pretest score a significant predictor of post-test score?
Is there support for level-2 predictors?
Random Coefficients Model with level-2 predictors: Includes CCBAS and APS as level 2 predictors
Does teacher score on the APS or CCBAS significantly predict student post-test score after controlling for pre-test?
N/A
After controlling for the pretest, signficiant variation was still observed between classrooms. Level 2 predictors are trying to explain the differences between classrooms and investigates teacher beliefs and practices as two potential sources.
7/31/2019
Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

13. Results Model Description Mathematics Science AE AI CC Yes No N/A
Unconditional: Provides a baseline for more complex comparisons
Is there a support for a Multi-level Model (if no, use simple regression)?
Yes No
Random Coefficients Model: Includes pre-test as a level-1 predictor
Is pretest score a significant predictor of post-test score?
Is there support for level-2 predictors?
Random Coefficients Model with level-2 predictors: Includes CCBAS and APS as level 2 predictors
Does teacher score on the APS or CCBAS significantly predict student post-test score after controlling for pre-test?
N/A
After controlling for the pretest, signficiant variation was still observed between classrooms. Level 2 predictors are trying to explain the differences between classrooms and investigates teacher beliefs and practices as two potential sources.
7/31/2019
Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

14. Results Model Description Mathematics Science AE AI CC Yes No N/A
Unconditional: Provides a baseline for more complex comparisons
Is there a support for a Multi-level Model (if no, use simple regression)?
Yes No
Random Coefficients Model: Includes pre-test as a level-1 predictor
Is pretest score a significant predictor of post-test score?
Is there support for level-2 predictors?
Random Coefficients Model with level-2 predictors: Includes CCBAS and APS as level 2 predictors
Does teacher score on the APS or CCBAS significantly predict student post-test score after controlling for pre-test?
N/A
After controlling for the pretest, signficiant variation was still observed between classrooms. Level 2 predictors are trying to explain the differences between classrooms and investigates teacher beliefs and practices as two potential sources.
7/31/2019
Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

15. Overview of Findings
Student attitudes were less consistent with the underlying theory of formative assessment at the end of the first year of the intervention.
There was significant classroom-to-classroom variation in student responses.
Student pre-test scores were a significant predictor of post-test scores.
However, teacher beliefs and practices did not predict student attitudes towards mathematics and science or student attitudes towards the classroom contract after controlling for pre-test score.
7/31/2019
Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

16. Potential Explanations
High school students have been in the educational context for between 9-12 years and attitudes and expectations may be set. It may take longer than a year to change them.
Teachers were in their first year of the intervention and were trying out new things throughout the year. Even though practice changed significantly it might not have been consistent enough or long enough to impact student attitudes.
Teacher self report may not actually accurately represent practice.
7/31/2019
Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

17. Next Steps
More validity evidence for the measurement of behaviors and attitudes is needed.
Additional studies to understand the source of classroom-to-classroom variation on attitudes is needed.
7/31/2019
Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

18. 7/31/2019
Contact Information Christine Lyon
Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

19. Appendix
Model specifications and findings for all models and content areas.
7/31/2019
Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

20. Descriptive statistics for mathematics
Shows changes in student attitudes from pre-test to post-test
Student attitudes decreased across all three scales at post-test
Scale
Mean (Std.dev)
Mean Difference T
(p value)
Internal Consistency Reliability
Pre
Post AE
19.51 (5.069)
19.24 (5.183)
-0.27
1.392 (.164)
.818
.851 AI
23.79 (3.928)
22.86 (4.177)
-0.93
.605 (<.001)
.827
.836 CC
27.89 (5.313)
27.56 (5.391)
-0.33
1.315 (.189)
.763
.781
The decrease for instrumental attitudes was significant
7/31/2019
Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

21. Descriptive statistics for mathematics
Measures the extent to which items within a scale are consistently measuring a construct together
Descriptive statistics for mathematics
Scale
Mean (Std.dev)
Mean Difference T
(p value)
Internal Consistency Reliability
Pre
Post AE
19.51 (5.069)
19.24 (5.183)
-0.27
1.392 (.164)
.818
.851 AI
23.79 (3.928)
22.86 (4.177)
-0.93
.605 (<.001)
.827
.836 CC
27.89 (5.313)
27.56 (5.391)
-0.33
1.315 (.189)
.763
.781
Supports the use of sub-scores as opposed to total scores
This supports the results of the factor analysis and the use of sub-scores as opposed to
Consistency of .700 or above is considered good for group comparisons
7/31/2019
Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

22. Descriptive statistics for science
Student attitudes decreased across all three scales at post-test
Scale
Mean (Std.dev)
Mean Difference T
(p value)
Internal Consistency Reliability
Pre
Post AE
20.31 (4.476)
20.00 (4.498)
-0.31
1.649 (.100)
.809
.802 AI
21.15 (3.955)
20.73 (4.293)
-0.42
2.486 (<.05)
.804
.851 CC
31.05 (5.628)
30.49 (6.181)
-0.56
2.086 (<.05)
.768
The decrease for instrumental attitudes and classroom contract was significant
Internal consistency reliability was above .70 for all three scales supporting the use of sub-scores
7/31/2019
Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

23. Mathematics Results 7/31/2019
Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

24. Mathematics Results Unconditional (intercept-only) Model
Provides a baseline for comparison with more complex models
Addresses the extent of classroom-to-classroom variation
Addresses the extent of student-to-student variation within a teacher
7/31/2019
Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

25. Mathematics Results: Unconditional Model
The average post-test score for a student’s classroom
Mathematics Results: Unconditional Model
The total proportion of the variance among classrooms was 0.13 or 87% of the variance was among students within classrooms
Scale
Fixed Effect
Coefficient se
AIC
ICC (among teachers) AE
Average classroom mean
0.3794
3067.9
.13
Random Effect
Variance Component
Classroom mean, u0j
3.4964
Level-1 effect, rij
22.835 AI
0.2529
2885.1
.06
1.0257 CC
0.2897
3148
.03
0.8759
The amount of teacher-to-teacher variation in average student post-test scores
Degree to which nesting is observed in the data. ICCs between provide a rationale for multi-level models with nesting
7/31/2019
Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

26. Mathematics Results: Unconditional Model
Scale
Fixed Effect
Coefficient se
AIC
ICC (among teachers) AE
Average classroom mean
0.3794
3067.9
.13
Random Effect
Variance Component
Classroom mean, u0j
3.4964
Level-1 effect, rij
22.835 AI
0.2529
2885.1
.06
1.0257 CC
0.2897
3148
.03
0.8759
Based on the unconditional model we claim that teacher-to-teacher variation in student posttest scores for AE and AI is significantly different from 0 indicated that the random coefficients model is warranted. For CC the teacher-to teacher variation is not significantly different from 0 and suggests that a simple regression can be fitted for all classrooms.
7/31/2019
Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

27. Mathematics Results Random Coefficients Model
Includes a level-1 predictor: pre-test score
Explores if there is significant classroom-to-classroom variation in post-test scores after controlling for pre-test scores
7/31/2019
Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

28. Mathematics Results: Random Coefficients Model
Less than .05 indicates that pre-test score is a significant predictor of post-test score
Scale
Fixed Effect
Coefficient se T
p value
AIC AE
Average classroom mean
19.15
0.2901 66
<.0001
2832.1
Pre-test
0.61
17.54
Random Effect
Variance Component
Classroom mean, u0j
1.95
0.729
Level-1 effect, rij
0.933 AI
22.86
0.1495
152.9
2676.5
0.63
0.0381
16.55
0.7134 CC
Average post-test score
27.56
.21548
127.90
.44
.04059
10.94
.1917
True for all three scales
True for all three scales
True for all three scales
For all three models, pretest is a significant predictor of post test score.
7/31/2019
Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

29. Mathematics Results: Random Coefficients Model
Indicates the amount of variance in average post-test scores that remains among classrooms after controlling for the pre-test scores
Scale
Fixed Effect
Coefficient se T
p value
AIC AE
Average classroom mean
19.15
0.2901 66
<.0001
2832.1
Pre-test
0.61
17.54
Random Effect
Variance Component
Classroom mean, u0j
1.95
0.729
Level-1 effect, rij
0.933 AI
22.86
0.1495
152.9
2676.5
0.63
0.0381
16.55
0.7134 CC
Average post-test score
27.56
.21548
127.90
.44
.04059
10.94
.1917
Variation still remained supporting the random co-efficients model with level-2 predictors
Classroom-to-classroom variance was reduced to near 0 after controlling for pre-test
For all three models, pretest is a significant predictor of post test score.
7/31/2019
Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

30. Mathematics Results Random Coefficients Model with level-2 predictors
Includes two level-2 classroom predictors
Teacher APS score
Teacher CCBAS score
Explores the amount of variance explained by these teacher attributes after controlling for student pre-test score
7/31/2019
Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

31. Mathematics Results: Level 2 predictors
Fixed Effect
Coefficient se t
p value
AIC
Average classroom mean
17.36
2.615
6.64
<.0001
2824.1
Pre-test
0.60
0.035
17.42
APS
0.02
0.034
0.57
0.5723
CCBAS
0.04
0.078
0.47
0.6378
Random Effect
Variance Component
Pre-test, u1j
2.00
0.788
Level-1 effect, rij
14.36
0.942
Indicates the amount of variance in average post-test scores that is accounted for by each predictor
The amount of variance predicted by APS and CCBAS is very small
7/31/2019
Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

32. Mathematics Results: Level 2 predictors
Indicates whether the predictor is a significant predictor of classroom-to-classroom variation
Fixed Effect
Coefficient se t
p value
AIC
Average classroom mean
17.36
2.615
6.64
<.0001
2824.1
Pretest
0.60
0.035
17.42
APS
0.02
0.034
0.57
0.5723
CCBAS
0.04
0.078
0.47
0.6378
Random Effect
Variance Component
Pre-test, u1j
2.00
0.788
Level-1 effect, rij
14.36
0.942
Neither APS or CCBAS is a significant predictor
7/31/2019
Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

33. Science Results
7/31/2019
Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

34. Science Results: Unconditional Model
Scale
Fixed Effect
Coefficient se
AIC
ICC AE
Average classroom mean
.28774
2737.7
.06
Random Effect
Variance Component
Classroom mean, u0j
1.1568
Level-1 effect, rij AI
.2491
2698.7
.03
.6449 CC
.3926
3035.1
2.1210
7/31/2019
Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

35. Science Results: Random Coefficients Model
Scale
Fixed Effect
Coefficient se t
p value
AIC AE
Average classroom mean
20.00
.2181 33
<.0001
2550
Pre-test
.5826
.03776
15.43
Random Effect
Variance Component
Classroom mean, u0j
.5460
.3742
Level-1 effect, rij
.8650 CC
30.49
.2776
109.85
2895.9
.5699
.04353
13.09
.04917
.5500
1.8301 AI
Average post-test score
20.73
.15738
131.72
.6614
.03984
16.60
.3711
7/31/2019
Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.

36. Science Results: Level 2 predictors
Scale
Fixed Effect
Coefficient se t
p value
AIC AE
Average classroom mean
2.0126
9.37
<.0001
2559.1
Pre-test
0.5809
0.038
15.28
APS
0.006
0.024
0.25
0.8027
CCBAS
0.028
0.061
0.46
0.6490
Random Effect
Variance Component
Pre-test, u1j
0.6237
0.4165
Level-1 effect, rij
0.8659 CC
2.3510
12.16
2902.8
0.5704
0.043
13.12
-1.11
0.2764
0.0840
0.071
1.18
0.2462
0.3245
0.5835
1.8481
7/31/2019
Confidential and Proprietary. Copyright © 2014 by Educational Testing Service. All rights reserved.