1. Paul Stemmer stemmerp@mi.gov NAEP State Coordinator Michigan Department of Education National Conference on Student Assessment, Minneapolis, Minnesota June 2012 Paul Stemmer stemmerp@mi.gov NAEP State Coordinator Michigan Department of Education National Conference on Student Assessment, Minneapolis, Minnesota June 2012 Describing NAEP Background Question Data: A Cautionary Tale

2. NAEP Background Question Data Analysis NAEP Grade 4 Reading Trends Comparison 2

3. NAEP Background Question Data Analysis NAEP Grade 4 Reading Trends Comparison (Enlarged) 3

4. NAEP Background Question Data Analysis What is different about these comparisons? 4

5. NAEP Background Question Data Analysis What is different about these comparisons? (enlarged) 5 Avg SS 202 214 219 222 * 229 237 238 * 217 219 221 * 215 225 224 Nat. Pub. MA MI OH Avg SS 220 237 219 224

6. NAEP Background Question Data Analysis Life is Good! 6

7. NAEP Background Question Data Analysis NAEP Trends in Grade 4 Reading Average Scaled Scores

8. NAEP Background Question Data Analysis Wait a Minute, What Happened? 8

9. NAEP Background Question Data Analysis Wait a Minute, What Happened? (enlarged) 9 Avg. SS * 217 219 222 * * 191 190

10. NAEP Background Question Data Analysis What explains this? What about other TUDAs? 10

11. NAEP Background Question Data Analysis What about OHIO and Cleveland? 11

12. NAEP Background Question Data Analysis What about MA and Boston? 12

13. NAEP Background Question Data Analysis Why the discrepancies?  Two Possibilities  A. If the data is in error?  Differences in respondent question interpretation – definitions?  Need to improve the question  Social Desirability  B. If data is “true,” perhaps more interesting?  Internal School Variables  External School Factors 13

14. NAEP Background Question Data Analysis Checking Social Desirability 09 to 11 Comparison 14

15. NAEP Background Question Data Analysis Why the discrepancies?  If data is “true,” perhaps more interesting? What are the possible intervening factors?  Internal School Variables  Professional development  Rigor  Instructional Methods/Impact  Professional knowhow vs. execution  Administration, management and follow through  Resources and resource utilization  External School Factors  Chronic Absenteeism  Concentration of Poverty  Health Factors  Cultural differences 15

16. NAEP Background Question Data Analysis Conclusions?  Descriptive data can be helpful and informative  Have to keep reminding ourselves it is not causative/predictive  Are we jumping to solutions without fully understanding how best to solve these problems of large urban school districts?  Do we infer too much from snapshot assessment results about the quality of teaching and administration?  Does arguing otherwise sound like excuses?  Aren’t we all guilty of trying to connect the dots?  Trying to simplify the story  Making inferences beyond what the data is telling us. 16

17. NAEP Background Question Data Analysis Finally  Prediction is very hard, especially about the future. - Yogi Berra 17

18. NAEP Background Question Data Analysis Contact: Paul Stemmer, Ph.D., NAEP State Coordinator Michigan Department of Education Office of Educational Assessment and Accountability PO Box 30008 Lansing, MI 48909 (517)241-2360 stemmerp@mi.gov For More Information: