WA Student Achievement Initiative: Preliminary Findings from Points Analysis and Field Research Davis Jenkins Community College Research Center Teachers College, Columbia University Presentation to WA CTC BAC and WARP Councils July 14, 2010 Seattle

Washington State’s Student Achievement Initiative Goal is to increase academic achievement at the state’s 34 community and technical colleges, so that all students go farther, faster Funding awards are based on how well colleges perform relative to themselves Initial awards (fall 2009) based on the change in total points at each college across six metrics between the baseline year (AY2007) and AY2009

Features Getting National Attention Achievement measures recommended by taskforce and advisory group and informed by SBCTC and CCRC research Measures encompass all mission areas, including adult basic skills and developmental education Performance measures designed to be within colleges’ control Colleges receive quarterly performance data to help decide where to invest resources to accelerate student achievement

CCRC-IHELP SAI Policy Study 3-year study funded by the Gates Foundation Research questions: Is the initiative shaping college practices related to student retention and success? What is the relative influence/value of the performance funding vs. the measures and data? Have students moved farther, faster? Has the mix of students who reach the “tipping point” changed in a way the reflects the demographics of the state? What distinguishes colleges that perform well according to the achievement point framework from those that do less well? Is the SAI policy sustainable in WA and what aspects of the policy might be adopted by other states? (becomes part of base budget)

CCRC-IHELP SAI Policy Study Research methods Builds on CCRC study during 2007-08 “learning year” Quantitative analysis of student progression and college performance using SAI data Interviews at sample of 17 colleges (spring 2010, and spring 2012) Interviews with policy makers in WA and other states Deliverables Briefing for Gates and WA policy makers (Sept. 2010) Reports on quantitative analysis of achievement points (1 report per year) Guides for state policy makers and college leaders (Dec. 2010 and Dec. 2012) (becomes part of base budget)

Year 1 (2010) Research Activities 1) Quantitative analysis of achievement point changes during the initial funding period 2) Interviews on-site and by telephone with administrators, faculty, and staff at 17 colleges previously studied during 2007-08 learning year 3) Telephone interviews with policy makers in WA, OH, NC and TX

Initial Points Analysis From baseline to AY2009: Total points Change in total points Points per individual metric Points per student (college-level) Points per student (student-level)

Total Points and Change in Points At baseline, average points total was 8,684; by 2009 it had risen to 10,365 Points growth of 1,681 points, or 19%. Points growth (19%) exceeded enrollment growth (12%) Two-thirds of points growth is attributable to more students; one-third to more points per student -- may indicate greater effectiveness

Change in Points

Influence of College Size Enrollment and total points are strongly correlated: bigger colleges produce more points Enrollment and change in total points are also positively correlated: bigger colleges gain more points A college with 1,000 more students gained on average 52 more points per year (total gain is ~840 points per year)

Total Points per College (AY2009) and FTE Enrollment

Total Points per College (AY2009) and Total Expenditures

Change in Points per College and FTE Enrollment

Growing Influence of Basic Skills and College Readiness Most points produced in basic skills and college readiness By 2009, basic skills and college readiness were making an even larger contribution to total points These two metrics accounted for 1,056 of the 1,681 extra points gained per college Five colleges lost points in quantitative reasoning; this metric may be diminishing in importance Student-level analysis indicates that the basic skills metric is the most powerful in encouraging enrollment from traditionally under-served groups

Source of Change in Points 2007-09

Heterogeneity of Effects No simple model predicts likely points accumulation for any given student Student characteristics – such as age, full-time status, race or prior education – do not strongly and clearly influence total points and the change in points There are some significant influences on points across the six metrics, but these wash out in the aggregate Overall, there is little evidence that colleges serving more at-risk are being penalized by the SAI awards method

Change in Points per College Effects No correlation between change in points and: % on financial aid, % prior education=dropout % age under 20, % age over 65 % fulltime, % male, spending per FTE % Asian, % Black, % Hispanic, % other race Positive correlation between change in points and: Enrollment  Student socio-economic status (affects basic skills points)

Spring 2010 Field Research Preliminary Findings Views on Achievement Point framework Generally liked because it focuses attention on student progression, provides common vocabulary for retention efforts; yet awareness on most campuses is limited Concerns re: Achievement Point measures Disagreement re: emphasis on basic skills/college prep Concern that larger colleges garner more points Rewards enrollment growth, but does it reward improved student persistence and completion? How to give colleges credit for students who “swirl” Colleges having trouble replicating SAI results their with their data

2010 Field Research Draft Findings Views re: performance funding Viewed more negatively than the points framework; many dislike the “skim” due to lack of new funds Some correlation between attitude and ranking Agreement among many that performance funding is here to stay in WA and will expand Consensus that money talks, but current SAI funding level is not enough to change behavior Concern however that more money could skew college priorities, promote unhealthy competition Yet, few suggestions for a better model

2010 Field Research Draft Findings Use of SAI data for improvement Use varies – influenced by IR staff attitudes re: the data Use limited by early data concerns, uncertainty on how to use data to identify specific areas for improvement and “best practices” SAI has caused some colleges to look more closely at their own data on student progression and success Views on future of SAI Uncertainty about source, stability of future funding What happens when enrollment growth subsides?

Further Research Planned Further quantitative analysis (2011- 2012): Will early patterns be reinforced such that the points distribution becomes more uneven? Do alternative measures of institutional performance yield different funding allocations? “Highest momentum” -- Are students going farther and progressing faster? Field research comparing high- and low-performing colleges (spring 2012)

For more information: Please visit us on the web at http://ccrc.tc.columbia.edu, where you can download presentations, reports, CCRC Briefs, and sign-up for news announcements. Community College Research Center Institute on Education and the Economy, Teachers College, Columbia University 525 West 120th Street, Box 174, New York, NY 10027 E-mail: ccrc@columbia.edu Telephone: 212.678.3091