Using Evaluation for Continuous Program Improvement: A Conversation with the Accelerating Opportunity Evaluators Lauren Eyster and Maureen Conway National Conference on Integrated Skills Pathways – Seattle, WA May 8, :00 p.m. – 5:15 p.m.

Topics for Today’s Session Refresher on Evaluation Activities and Timelines Evaluation Updates Discussion of Implementation Evaluation Activities Discussion of Use of Data for Performance Monitoring and Improvement Q&A with the Evaluation Team

Goals of the AO Evaluation To generate evidence for state and federal policymakers, college administrators, funders, and other stakeholders about: 1)The process of implementing integrated college and career pathway designs and taking these designs to scale 2)Their impact for ABE and ESL students in college and in the labor market 3)Their cost effectiveness and financial sustainability

AO Evaluation Design Impact on AO Students Cost Effectiveness and Financial Sustainability Implementation and Scalability

Current Evaluation Activities Completed evaluation visits to all four states In discussions with each state to complete data agreement and submission schedule by end of May Will send analysis plans for state review by Monday, May 14 Will begin scheduling next quarterly call and site visits next week Preparing instruments for site visit and survey in June and July

Overall Evaluation Schedule Evaluation ActivityDates Initial Evaluation Planning Visits (One-Day Visits) March and April 2012 First Full Site Visit (Three-Day Visits) September-October 2012 First Community College SurveyNovember 2012 Second Full Site Visit (Three-Day Visits) April-May 2014 Second Community College SurveyOctober 2014 Ongoing Activities - Quarterly Check-in Calls - Semi-Annual Data Submissions March, June, September, and December during the grant period Summer and Winter

Implementation Analysis Questions How did states and community colleges establish integrated pathway designs? What shifts in the culture (attitudes and behavior) occurred within states and colleges concerning ABE/ESL students? How did states and community colleges move toward scale and the sustainability of pathways through policy changes, innovative financing, the use of data to support continuous improvement, and other means?

Site Visits to States and Community Colleges Twice during the grant period (Fall 2012 and Spring 2014) Three-day visits with two evaluation team members Two community colleges will be visited during each site visit Selection of the colleges will be determined with the state team

Community College Surveys Used to collect systematic data on implementation, policy, program costs, and sustainability across ALL community colleges Fielded once after the end of the first year of the grant (November 2012) and once at the end of the grant period (October 2014) Web-based survey but will be available in paper if requested Will work with the states in fielding the survey – reaching the right contacts at each college and following up with non-responders

Quarterly Check-in Calls To keep updated on changes to AO program design and implementation To provide states with an opportunity to ask questions about the evaluation activities Will occur March, June, September, and December each year during the grant period

What We Learned from Our Visits and Calls This Spring Recruitment is mainly from the student body but will be expanded in the future. Two common challenges have been the 12-credit standard and the loss of AtB. Colleges are excited about the integrated pathway model and report positive responses from students and faculty. Resources for coordinating support services is challenging but there are examples of promising strategies in colleges.

Implementation Discussion Questions What are some of the economic factors your state and colleges are currently facing that might affect your pathway programs or policy implementation? Which partners are likely to be more involved in the pathway programs in this first year than the subsequent years? What opportunities might we have to interview students and observe class time? Are there any changes in state policy that we should be aware of?

Evaluation Data Student record data submissions – Analyzed twice (fall/winter 2013 and spring/summer 2015) Community college survey data – Collected an analyzed twice (fall 2012 and fall 2014) Can provide analysis files to state team once data are cleaned and coded Plan to send draft sections of reports to states and colleges to ensure accuracy/clarity of findings

AO Evaluation Reports ReportsDue Date Formative first year implementation report (to JFF and leadership team) February 2013 Interim report (to JFF and leadership team)May 2013 Formative report (for public release)February 2014 Interim report (to JFF and leadership team)May 2014 Final Report (for public release)August 2015

Use of Data for Performance Improvement What types of internal analysis are you currently planning to do? Are there analyses that you would like to do but there may not be enough time or resources? How will internal analysis and evaluation reports be used for program improvement and decision-making?

Potential Analysis – Loss Points Pathways in a College Percent Retained to Second Term Percent Retained to Third Term Percent Retained to Fourth Term Pathway #1# students enrolled in second term/# of students enrolled in first term # students enrolled in third term/# of students enrolled in second term # students enrolled in fourth term/# of students enrolled in third term Pathway #2 Pathway #3

Potential Analysis – Participant Characteristics PathwaysPercent ABE by NRS Level Percent ESL by NRS Level Recruitment Source GenderAge Pathway #1% of AO students Breakdown by NRS level % Level 4 % Level 5 % Level 6 % of AO students Breakdown by NRS level % Level 5 % Level 6 % from student body % from One- Stop % from non- college ABE/ESL program % FemaleAverage age Pathway #2 Pathway #3

Open Time for Evaluation Q&A