Delivering on the Promise of Data in Education Winter Forum February 2013 Jack Buckley Commissioner National Center for Education Statistics

A Really Simple Theory of the Tradeoff Between Quality and Accountability Accountability Pressure Data Quality What we wish were true How it is

Accountability Pressure Data Quality The “Chowky Dar” Region: Increased Accountability Pressure Improves Quality

Chowky Dar? "The government are very keen on amassing statistics. They collect them, add them, raise them to the nth power, take the cube root and prepare wonderful diagrams. But you must never forget that every one of these figures comes in the first instance from the chowky dar (village watchman in India), who just puts down what he damn pleases.” –Josiah Stamp No accountability = no incentive for accurate reporting Some accountability (even “soft”) can yield improvements in quality through reduction of “noise” Example: Last year’s U.S. News and World Report high school rankings and the quality of school-level CCD data

Accountability Pressure Data Quality The High-Stakes Region: Too Much Pressure Can Distort the Data

High-Stakes Here the threat is not usually noise—it’s intentional misreporting. As the stakes get higher, a percentage of individuals involved in the system will subvert it. Example: the intentional cheating by some Atlanta Public Schools teachers and administrators on the Georgia state assessment. Some ways to reduce accountability pressure are sampling (NAEP versus state tests) or confidentiality through aggregate reporting, but not an option for every application.

Accountability Pressure Data Quality Can We Get to the Middle? Only if We Control How the Data Are Used. Good Luck with that.

Accountability Pressure Data Quality Q: If Policy Makers Set the level of Accountability Pressure, what Can We Do? A: Find Ways to Shift the Whole Curve Up

So How Can We Shift the Curve? Cooperation, training, technical assistance, sharing best practices. In other words, the Forum and related efforts. Improvements in collection technology (broadly defined) can help—SLDSs, better edit checks, smarter tools, automation, integrated systems, methods for detecting distortion, common data standards. This is the basic strategy behind NCES’s activities in administrative data: Build a strong community, invest in better systems, develop common standards, and improve our tools and technology. In short, if we can shift the curve, data quality can theoretically be improved at all levels of accountability.

How NCES is Shifting the Curve Today and Tomorrow CEDS V 3.0 end of January On time and almost under budget Continued development work on CEDS Align and Connect Tools Assisting OCR with redesign of the CRDC collection tool Working with EDFacts and state partners to integrate from CEDS to data groups Maintaining and improving our Forum, MIS/Summer Data, SLDS conferences to work with the field Working with our partners to reexamine and improve the back end of collection systems like CCD, CRDC, EDFacts, and IPEDS.

But Data Quality Is not the Only Challenge to Usefulness Some uses of the data are obvious and benefit obviously from quality improvement—simple descriptive statistics, monitoring and early warning systems, feedback reports. But other uses—prediction, causal inference, evaluation—can only be improved so much through shifting the curve. Even with perfectly measured data, there are many threats to valid inference. Large scale IT systems, shiny technology, “Big Data,” aren’t substitutes for principles of careful scientific research design. There are no easy answers to difficult questions. The surest way to end the new era of data in education is through overpromising and large-scale failure. “Those who ignore Statistics are condemned to reinvent it.” –Brad Efron.