Education, Evidence, and Policy Cecilia Elena Rouse, Ph.D. Member of the Council of Economic Advisers June 8, 2009

The Council of Economic Advisers was established by Congress in the Employment Act of 1946: “There is hereby created in the Executive Office of the President a Council of Economic Advisers (hereinafter called the “Council”). The Council shall be composed of three members who shall be appointed … to appraise programs and activities of the Government …and to formulate and recommend national economic policy to promote employment, production, and purchasing power under free competitive enterprise.”

It shall be the duty and function of the Council -- to assist and advise the President in the preparation of the Economic Report; to gather timely and authoritative information concerning economic developments and economic trends….; to appraise the various programs and activities of the Federal Government….; to develop and recommend to the President national economic policies to foster and promote free competitive enterprise, to avoid economic fluctuations or to diminish the effects thereof, and to maintain employment, production, and purchasing power; to make and furnish such studies, reports thereon, and recommendations….

Educators “Scientific Learning's family of programs and products are for anyone who wants to improve language, reading and overall communication skills. From children who struggle with basic language skills or attention problems to adults who want to improve reading comprehension and organizational skills, Scientific Learning's programs and products have something for everyone.”

4 schools in a large urban school district in the Northeast Eligibility based on low reading score on Fall 2001 State Assessment (Bottom 20%) 513 students in grades 3-6 Random assignment within grade and school Administered CELF-3 and Reading Edge Test; SFA Assessments Rouse, Krueger, and Markman Evaluation of Fast ForWord

CELF-3 Test: Gains Among FFW Participants N = 43 Effect size=0.3σ p-value=0.008

Reading Edge Test: Gains Among FFW Participants N = 244 Effect size=0.7σ p-value=0.000

But, what would have happened had the children not participated in FFW (i.e., what’s the counterfactual)?

CELF-3 Test: Gains Among FFW Participants and Controls N = 89 “Net” Effect size=0.04σ p-value=0.73

SLC’s Reading Edge Test: Gains Among FFW Participants and Controls N = 485 “Net” Effect size=0.13σ p-value=0.098

Post FFW SFA Assessment: FFW Participants vs. Experimental Controls N = 197 FFW Participants; 176 Experimental Controls Effect size=0.05σ p-value=0.583

Post FFW SFA Assessment: FFW Participants vs. Non-experimental and Experimental Controls Experimental effect size=0.05σ (p-value=0.583) Non-experimental effect size=-0.3σ (p-value=0.000) N = 197 FFW Participants; 3850 Non-experimental Comparisons; 176 Experimental Controls

Non-experimental vs. Experimental Estimated Effects of FFW on Gain in SFA Assessments Non-experimental: N=4048 “Net” Effect size=0.04σ p-value=0.11 Experimental: N=373 “Net” Effect size=0.01σ p-value=0.60

Example of Results that Led to Push for “College for All”

Chicago’s “College Prep for All” Instituted in 1997 Requires all students to take: 1.Four years of English + Three years of Math, science, and social science 2.Particular classes: Algebra I, geometry, Algebra II (math); survey literature, American literature, European literature, world literature (English); biology, earth science, and chemistry or physics (science); world studies, U.S. history, and one elective (social science)

First Results from Chicago’s “College Prep for All” (Mathematics)

Limits of most “rigorous” studies for policy: Heterogenous treatment effects; Size of likely impacts are difficult to interpret/translate; Difficulty or ease of bringing to scale; Typically doesn’t illuminate the mechanism by which a particular program/policy works; Longer-run impacts are usually difficult because of cost (especially time).

So Where Does this Leave Us? Current evidence-base for most education programs/reforms is extremely thin. Need more high-quality impact evaluations to help guide everyday decisions. It’ll take cooperation from all to improve the quality of education research on impacts.

What Is PART? Standardize performance assessments across government Not used enough in decision-making Reform will emphasize cross-agency program goals and transparency

Define Program Activity Define How PART Processes Unfold Inform Decisions President’s Budget Appropriations, Authorization, Oversight Congressional Action Share Best Practices Executive Branch Build Public Trust ExpectMore Periodic Reassessment Assessment (OMB & Agency)‏ Improvemen t Plans Created OMB Internal Consistency Check Formal Agency Appeals Process