Using Evaluation for Continuous Program Improvement: A Conversation with the Accelerating Opportunity Evaluators Lauren Eyster and Maureen Conway National.

Slides:



Advertisements
Similar presentations
Session Objectives Begin to understand the goals, purpose and rationale for Program Reviews Learn about the components of implementing Program Reviews.
Advertisements

Cedarville University Accreditation Self-Study Plan Presented by Dr. Thomas Mach.
QEP BASICS.
MADELEINE GABRIEL & JULIE DAS 9 FEBRUARY PURPOSE OF THIS SESSION Introductions Purpose of our input today Our role – we are evaluating the programme,
WIOA Timeline and Action Plan for States
Making Opportunity Affordable Grant
WA State I-BEST Program Planning and Approval
Sustainability Planning Pat Simmons Missouri Department of Health and Senior Services.
OVERVIEW OF ClASS METHODS and ACTIVITIES. Session Objectives By the end of the session, participants will be able to: Describe ClASS team composition.
Accelerating Opportunity Arkansas Bidders Webinar March 7, 2013.
College-wide Meeting October 28,2011. Success is what counts. 2 Achieving the Dream is a national initiative to help community colleges provide the best.
Accelerating Opportunity Evaluation Planning the Evaluation with the Accelerating Opportunity States February 10, :30 a.m. – 1:00 p.m.
ExCEL Academic Liaison Meeting Supporting Quality After School Programs for Our Students Tuesday, September 23, 2014.
Completion and Attrition in AGEP and non-AGEP Institutions National Science Foundation Grant # CGS Webinar January 17, 2012 Robert Sowell Jeff Allum.
Neil Naftzger Principal Researcher Washington 21st CCLC Evaluation February 2015 Copyright © 20XX American Institutes for Research. All rights reserved.
AB 86 Adult Education Regional Planning. What is AB86 ? $25 Million Statewide for Planning AB 86, Section 76, Article 3 The purpose is develop regional.
Orientation to the Accreditation Internal Evaluation (Self-Study) Flex Activity March 1, 2012 Lassen Community College.
4th Annual Innovation Challenge Kick-Off and Overview Fall
Principals’ Council Meetings May  Given feedback from multiple stakeholders and after much deliberation, PDE has made the determination to classify.
ABLE State Update Jeff Gove, State ABLE Director.
Assessment Leader Training General Education Student Learning Assessment GEO Training Series: 2 of 5 Spring 2012 February 13, 2012.
Interim Joint Committee on Education June 11, 2012.
Oregon Reading First Orientation Holiday Inn Portland Airport November 12, 2002 Oregon Department of Education.
Leadership Role Workshop: Work-Based Learning Friday, June 26 th 2:30 – 4pm 90 minutes.
AN INVITATION TO LEAD: United Way Partnerships Discussion of a New Way to Work Together. October 2012.
Academic Assessment Task Force Report August 15, 2013 Lori Escallier, Co-Chair Keith Sheppard, Co-Chair Chuck Taber, Co-Chair.
Strong Schools, Strong Communities Strategic Plan Implementation Process and Roles Saint Paul Public Schools has designed the following process and roles.
Assistant Principal Meeting August 28, :00am to 12:00pm.
Completion and Attrition in AGEP and non-AGEP Institutions Technical Workshop CGS Annual Meeting December 10, 2011 Robert Sowell Jeff Allum Nathan Bell.
Honors Level Course Implementation Webinar Honors Rubric and Portfolio Review Process October 7, 2013.
AHCCCS Update Meeting – Systems Update September 2015.
FAFSA Collective Impact Initiative Professional Learning Community January 13, 2015.
Commission on Colleges of the Southern Association of Colleges and Schools (SACS) Reaffirmation of Accreditation.
Legal Services Advisory Committee (LSAC) Grant Applicant Workshop February 11, 2013.
Time Management From Proposal to Dissertation Defense.
American Community Survey (ACS) Program Review Webinar March 6, 2012.
Creating Pathways for Education, Career and Life Success Webinar: Developing a Pathways Plan January 18, 2013 Facilitated by Jeff Fantine, Consultant.
SLC Capstone Meeting: Irvington High School June 2, 2008.
AdvancED TM External Review Exit Report Polk Pre-Collegiate Academy April 16– 17, 2014.
CIRTL Network Data Collection 3/2/2013. Institutional Portrait: Purpose Consistency with the TAR principle Accountability: – Helps us all monitor Network.
SACS and The Accreditation Process Faculty Convocation Southern University Monday, January 12, 2009 Presented By Emma Bradford Perry Dean of Libraries.
Summary of Sessions. SESSION 1 Knowing Yourself/Personal Leadership Wednesday, July 25, :00am-4:00pm, PDC Presenter: Dr. Larry Coble Topics:
Welcome Katherine Cox Print Download Help Exit View Notification from TA Choose Document : Proposal Report Proposal Approved Proposals/Reports.
Reform Model for Change Board of Education presentation by Superintendent: Dr. Kimberly Tooley.
Evaluation of the Indiana ECCS Initiative. State Context Previous Early Childhood System Initiatives –Step Ahead –Building Bright Beginnings SPRANS Grant.
© 2007 SRI International CPATH Principal Investigators Meeting: Program Evaluation Update March 26, 2010 By the Center for Education Policy Dr. Raymond.
Creating an Online Professional Development Academy for STEM Teachers 2015 UTeach Annual Conference.
Certification Learning Network February 2,
Orientation and Summer Institutes Implementer’s Forum October 2005 Susan Barrett PBIS Maryland.
September 2013 DO NOW! On a note card write down One success of your CLT today One challenge your CLT today faced today One way the Professional Learning.
School Accreditation School Improvement Planning.
SPC Advisory Committee Training - TAC Fall 2015 Institutional Research President’s Office 1 Abridged from the SPC Advisory Committee Training on October.
SPC Advisory Committee Training Fall 2015 Institutional Research President’s Office SPC 10/9/20151.
Statewide Evaluation Cohort 7 Overview of Evaluation March 23, 2010 Mikala L. Rahn, Ph.D.
District Collaboration Grant Requirements November 17, 2015.
Welcome to Cityspan. Today’s Goals 1.Educate you on the Chicago OST Project, which is the Wallace funded initiative to improve the quality of out-of school-time.
Accreditation Self-Study Progress Update Presentation to the SCCCD Board of Trustees Madera Center October 5, 2010 Tony Cantu, Fresno City College Marilyn.
Fed Challenge. College Fed Challenge is an academic competition. Teams of students research and analyze economic conditions and then present their analysis.
[Presentation location] [Presentation date] (Confirm ABT logo) Building Bridges and Bonds (B3): An introduction.
New Haven, A City of Great Schools MOVING FROM COMPLIANCE TO COHERENCE IN EVALUATION AND DEVELOPMENT: THE IMPACT OF THE E3 PROGRAM NEW HAVEN PUBLIC SCHOOLS.
Welcome Katherine Cox Print Download Help Exit View Notification from TA Choose Document : Proposal Report Proposal Approved Proposals/Reports.
HR Research Mary O’Regan. HR Research Mary O’Regan Laura McSweeney Tracy Eagles.
MA DSAC Collaboration Institute for Special and General Education Leaders Session 2: November 15, 2011 Focus on Clear and Collaborative Relationships.
Teacher Evaluation Timeline
Local Workforce Innovation Area Realignment Statewide Update 9/12/18
Foothill College Accreditation Self-Study Update
Working Families Success Network in Community College (WFSNCC) Initiative Nineteen community colleges in four states working to implement a strategy to.
The Heart of Student Success
Building Evidence The YCC Evaluation
Fed Challenge.
Presentation transcript:

Using Evaluation for Continuous Program Improvement: A Conversation with the Accelerating Opportunity Evaluators Lauren Eyster and Maureen Conway National Conference on Integrated Skills Pathways – Seattle, WA May 8, :00 p.m. – 5:15 p.m.

Topics for Today’s Session Refresher on Evaluation Activities and Timelines Evaluation Updates Discussion of Implementation Evaluation Activities Discussion of Use of Data for Performance Monitoring and Improvement Q&A with the Evaluation Team

Goals of the AO Evaluation To generate evidence for state and federal policymakers, college administrators, funders, and other stakeholders about: 1)The process of implementing integrated college and career pathway designs and taking these designs to scale 2)Their impact for ABE and ESL students in college and in the labor market 3)Their cost effectiveness and financial sustainability

AO Evaluation Design Impact on AO Students Cost Effectiveness and Financial Sustainability Implementation and Scalability

Current Evaluation Activities Completed evaluation visits to all four states In discussions with each state to complete data agreement and submission schedule by end of May Will send analysis plans for state review by Monday, May 14 Will begin scheduling next quarterly call and site visits next week Preparing instruments for site visit and survey in June and July

Overall Evaluation Schedule Evaluation ActivityDates Initial Evaluation Planning Visits (One-Day Visits) March and April 2012 First Full Site Visit (Three-Day Visits) September-October 2012 First Community College SurveyNovember 2012 Second Full Site Visit (Three-Day Visits) April-May 2014 Second Community College SurveyOctober 2014 Ongoing Activities - Quarterly Check-in Calls - Semi-Annual Data Submissions March, June, September, and December during the grant period Summer and Winter

Implementation Analysis Questions How did states and community colleges establish integrated pathway designs? What shifts in the culture (attitudes and behavior) occurred within states and colleges concerning ABE/ESL students? How did states and community colleges move toward scale and the sustainability of pathways through policy changes, innovative financing, the use of data to support continuous improvement, and other means?

Site Visits to States and Community Colleges Twice during the grant period (Fall 2012 and Spring 2014) Three-day visits with two evaluation team members Two community colleges will be visited during each site visit Selection of the colleges will be determined with the state team

Community College Surveys Used to collect systematic data on implementation, policy, program costs, and sustainability across ALL community colleges Fielded once after the end of the first year of the grant (November 2012) and once at the end of the grant period (October 2014) Web-based survey but will be available in paper if requested Will work with the states in fielding the survey – reaching the right contacts at each college and following up with non-responders

Quarterly Check-in Calls To keep updated on changes to AO program design and implementation To provide states with an opportunity to ask questions about the evaluation activities Will occur March, June, September, and December each year during the grant period

What We Learned from Our Visits and Calls This Spring Recruitment is mainly from the student body but will be expanded in the future. Two common challenges have been the 12-credit standard and the loss of AtB. Colleges are excited about the integrated pathway model and report positive responses from students and faculty. Resources for coordinating support services is challenging but there are examples of promising strategies in colleges.

Implementation Discussion Questions What are some of the economic factors your state and colleges are currently facing that might affect your pathway programs or policy implementation? Which partners are likely to be more involved in the pathway programs in this first year than the subsequent years? What opportunities might we have to interview students and observe class time? Are there any changes in state policy that we should be aware of?

Evaluation Data Student record data submissions – Analyzed twice (fall/winter 2013 and spring/summer 2015) Community college survey data – Collected an analyzed twice (fall 2012 and fall 2014) Can provide analysis files to state team once data are cleaned and coded Plan to send draft sections of reports to states and colleges to ensure accuracy/clarity of findings

AO Evaluation Reports ReportsDue Date Formative first year implementation report (to JFF and leadership team) February 2013 Interim report (to JFF and leadership team)May 2013 Formative report (for public release)February 2014 Interim report (to JFF and leadership team)May 2014 Final Report (for public release)August 2015

Use of Data for Performance Improvement What types of internal analysis are you currently planning to do? Are there analyses that you would like to do but there may not be enough time or resources? How will internal analysis and evaluation reports be used for program improvement and decision-making?

Potential Analysis – Loss Points Pathways in a College Percent Retained to Second Term Percent Retained to Third Term Percent Retained to Fourth Term Pathway #1# students enrolled in second term/# of students enrolled in first term # students enrolled in third term/# of students enrolled in second term # students enrolled in fourth term/# of students enrolled in third term Pathway #2 Pathway #3

Potential Analysis – Participant Characteristics PathwaysPercent ABE by NRS Level Percent ESL by NRS Level Recruitment Source GenderAge Pathway #1% of AO students Breakdown by NRS level % Level 4 % Level 5 % Level 6 % of AO students Breakdown by NRS level % Level 5 % Level 6 % from student body % from One- Stop % from non- college ABE/ESL program % FemaleAverage age Pathway #2 Pathway #3

Open Time for Evaluation Q&A