The Formative & Summative Evaluation Process TS & EOC SAEOPP Priority 1 Training PY 2009

Presenter Elizabeth C. Retamozo, Executive Director NOSOTROS Education Center Phone:

Presentation Materials PowerPoint Slide Handouts Additional Documents & Materials located at: –Go to Resources Page –Register Your Address –Click the Presentations & Links Tab –Click SAEOPP Priority 1 Training 3

4 Session Summary This session will focus on providing Talent Search staff with an overview of: Program Evaluation Types of Program Evaluations Methods & Types of Data Collection Quantitative & Qualitative Data Formative & Summative Methods

5 Evaluation Evaluation - Systematic method for collecting, analyzing, and using information to answer basic questions about your program. Helps to identify effective & ineffective services, practices, & approaches. Informs & assists program staff to make changes to improve program effectiveness.

6 Evaluation Plan Evaluation Plan - A written & comprehensive plan to determine the effectiveness of the program in meeting stated objectives & goals. Includes what you plan to do, how you plan to do it, who will do it, and when it will be done. Provides a mechanism for evaluating & implementing programmatic changes to improve program effectiveness Relates your project to GPRA & the TRIO Programs Performance Indicators

7 The GPRA Connection 1)Government Performance and Results Act of 1993 (GPRA) 2)DOE Response to GPRA Requirements: Goals 1-6 3)TRIO Programs Performance Indicator: Goal 5 4)TRIO Programs’ Specific Program Measures:  Upward Bound, Talent Search, Educational Opportunity Centers  Student Support Services & Ronald E. McNair 5)Prior Experience Criteria & Mandatory Objectives – APR 6)Project Evaluation Plan 7)Activities & Services Specific to Each Objective

8 Types of Evaluation Plans  Goals-Based Evaluation  Outcome-Based Evaluation  Process-Based Evaluation

9 Evaluation Plan “Buzz” Words  Qualitative Values  Quantitative Values  Formative Methods  Summative Methods

10 Qualitative Values Measurement of “feelings”, impressions, & perception  Ways to describe effectiveness that are measurable, but may not aggregate in traditional, quantifiable methods  Information that is difficult to measure, count, or express in numerical terms

11 Collection of Qualitative Data Interviews & Focus Groups Used to fully understand someone's impressions or experiences, or learn more about their answers to questionnaires Observations Used to gather accurate information about how a program actually operates, particularly about processes Questionnaires, Surveys, Checklists Used to quickly and/or easily get info in a non-threatening way

12 Quantitative Values Information that can be expressed in numerical terms, counted, or compared on a scale  Promotion to Next Grade Level Rates  Graduation From High School Rates  # of College Admissions Forms Completed  # of Financial Aid Applications Completed  College Enrollment Rates

13 Collection of Quantitative Data Report Cards Transcripts Test Scores (NCLB, PSAT, SAT, ACT) National Student Clearinghouse Reports

14 Formative Methods Continuous (daily, weekly, monthly, quarterly, semester) assessment of services/activities & progress towards achieving objectives Allows for changes in activities during the year so final objectives/outcomes will be met (changes in delivery of services)

15 Formative Methods Weekly/monthly student service reports  # of continuing student with 0 services  # students with less than 2 services  # of students with equal to or more than # services  # students that have received a specific services Student evaluation of workshops & field trips Grading period report cards (6/9 weeks)

16 Progress Towards Achieving Goals & Objectives During the Year - Formative Methods: Project staff document all services provided to students as they occur on Individual or Group Service Logs Project staff enter all data on service logs into the project database on a weekly basis Project database generates monthly/quarterly contact reports documenting which students have been served, the types of services provided, & the frequency of services provided.

17 Progress Towards Achieving Goals & Objectives During the Year - Formative Methods: Project Director tracks progress towards the accomplishment of all objectives via reports (quantitative) generated by the project database on a weekly/ monthly basis Participants rate specific group activities and workshops to rate their level of satisfaction with the quality of services on evaluation forms (qualitative).

18 Summative Methods End of year assessment that measures the results or outcomes of a project.  Concerned with a program's overall effectiveness  Quantitative data is analyzed for a given period  Qualitative evaluation supplements the quantitative data

19 Achieving Goals & Objectives at End of the Year - Summative Methods: Project database combines all monthly reports to produce a report documenting & confirming the achievement of project objectives at the end of the year (quantitative). Students complete end-of-year surveys (qualitative) to measure level of satisfaction with the overall quality of project services.

20 Achieving Objectives at the End of the Project Period - Summative Methods: Target school/agency/project personnel complete end of year evaluation forms (qualitative) to measure level of satisfaction with the overall quality of project services. Project Director reviews evaluation results to implement any corrective measures to eliminate potential problems by appropriately modifying & improving activities & services for the next program.

21 TRIO Evaluation Plans Goals Based Evaluation (Internal & Formative) Evaluates the extent to which projects are meeting predetermined goals or objectives. INTERNAL GOALS Outcomes Based Evaluation (Internal & Summative) Looks at impacts/benefits/changes to your students as a result of your project’s efforts during and/or after their participation in your project. PE OBJECTIVES Process Based Evaluation (External & Summative) Geared to fully understanding how a program works -- how does it produce that results that it does.

22 Goals Based Evaluation 1)What project goals were established? 2)What is the status of the project’s progress towards achieving the goals? 3)Will the goals be achieved according to the timelines specified in the program implementation or operations plan? 4)Do personnel have adequate resources (supplies, facilities, training, etc)?

23 Goals Based Evaluation 5)How should priorities be changed to more focus on achieving the goals? 6)How should timelines be changed? 7)How should goals be established in the future?

24 Sample TS Internal Goals 1)Identification, Recruitment, & Selection of Participants: To Identify, recruit, & select new students to have a total of 600 participants with 67% classified as both low-income & potential first generation college students within 90 days of the beginning of the program year.

25 Sample TS Internal Goals 2)Development of Needs Assessment & Individual Service Plans: 100% of all participants will have their needs assessed and an Annual Individual Service Plan developed within 30 days of their initial selection or renewal into the project each program year.

26 Sample TS Internal Goals 3)Delivery of Promotion Services To 9th - 11th Participants: 70% of all 9th - 11th grade participants will receive a minimum of three (3) college, career, and financial aid exploration services each semester for a total of six (6) services by the end of the program year.

27 Sample TS Internal Goals 4)Delivery of Individual College Admissions & Financial Aid Application Assistance to all College Ready Participants: 80% of all college ready participants will receive a minimum of five (5) individual advising sessions each semester for a total of ten (10) sessions by the end of the program year.

28 Sample TS Internal Goals 5)Increased Awareness & Knowledge of College, Financial Aid, & Career Information: 80% of participants will demonstrate an increase in knowledge of college, financial aid & career information as demonstrated by a comparison pre/post tests by the end of the program year.

29 Sample TS Internal Goals 6)Perception & Level of Satisfaction with Project Services: 85% of all participants and target school personnel surveyed will rate the project as "Excellent" on the Project Evaluation Form by the end of each project year.

30 Outcome Based Evaluation TS Goals Based Evaluation Plan

31 Sample EOC Internal Goals 1)Identification, Recruitment, & Selection of Clients: To Identify, recruit, & select new students to have a total of 1000 clients with 67% classified as both low-income & potential first-generation college students within 180 days of the beginning of the program year.

32 Sample EOC Internal Goals 2)Development of Needs Assessment & Individual Service Plans: 100% of all clients will have their needs assessed and an Individual Service Plan developed within 30 days of their initial selection into the project each program year.

33 Sample EOC Internal Goals 3)Delivery of GED/Continuing Education Services to Non-College Ready Clients: 100% of all Non-College Clients will receive a minimum of three (3) GED/Continuing Education services (individual advising, referrals, information, etc.) by the end of the program year.

34 Sample EOC Internal Goals 4)Delivery of Individual College Admissions & Financial Aid Application Assistance to all College Ready Clients: 100% of all college ready clients will receive a minimum of three (3) individual advising sessions by the end of the program year.

35 Sample EOC Internal Goals 5)Perception & Level of Satisfaction with Project Services: 85% of all Clients and target agency personnel surveyed will rate the project as "Excellent" on the Project Evaluation Form by the end of each project year.

36 Outcome Based Evaluation EOC Goals Based Evaluation Plan

37 Outcome Based Evaluation 1)Identify Major Outcomes (Mandatory Objectives) 2)Specify Observable Measures & Indicators 3)Identify what Info is Needed to Show these Indicators

38 Outcome Based Evaluation 4)Decide How this Info Can be Realistically & Efficiently Gathered 5)Assign Staff Responsibility & State Timelines 6)Analyze & Report Findings

39 Outcome Based Evaluation TS Outcome Based Evaluation Plan EOC Outcome Based Evaluation Plan

40 External Evaluation Recommended in last grant proposal competitions Must be conducted if included in grant proposals Allowable cost for the project May be paid by the grantee agency/institution May be provided free of charge – in-kind/donation

41 External Evaluation May be conducted by person working/employed by agency/institution BUT not for the project Must have experience conducting evaluations for educational opportunity programs & TRIO like programs Must state what type of evaluation will be conducted by the evaluator.

42 External Evaluation Process-Based Evaluation Process-based evaluations are geared to fully understanding how a program works -- how does it produce that results that it does. Useful if programs are long-standing and have changed over the years. Useful for accurately portraying to outside parties how a program truly operates.

43 Process-Based Evaluation Examines the extent to which a program is operating as intended by assessing ongoing program operations & whether the targeted population is being served. Helps program staff identify needed interventions and/or change program components to improve service delivery.

44 Process Based Evaluation 1)How does the project identify, recruit, & select students? 2)How does the project decide what specific services & activities to provide to its students? 3)How does the project monitor students’ academic progress?

45 Process Based Evaluation 4)How does the project deliver services to its students? 5)How does the project document services provided to student? 6)What is the project’s goals based evaluation plan? 7)What is the project’s outcome based evaluation plan? 8)How does the project plan for the next program year?

46 Process Based Evaluation Process Based Evaluation Plan

47 Basic Guide to Program Evaluation Written by Carter McNamara, MBA, PhD, Authenticity Consulting, LLC. Copyright Carter McNamara, MBA, PhD, Authenticity Consulting, LLC Adapted from the Field Guide to Nonprofit Program Design, Marketing and Evaluation.Field Guide to Nonprofit Program Design, Marketing and Evaluation Online Guide: A Basic Guide to Program Evaluation.pdf Bibliography & Resources

48 Basic Guide to Goals-Based Evaluation for Nonprofit Organizations with Very Limited Resources Written by Carter McNamara, MBA, PhD, Authenticity Consulting, LLC. Copyright Carter McNamara, MBA, PhD, Authenticity Consulting, LLC Online Guide: Basic Guide to Outcomes-Based Evaluation for Nonprofit Organizations with Very Limited Resources.mhtBasic Guide to Outcomes-Based Evaluation for Nonprofit Organizations with Very Limited Resources.mht Bibliography & Resources

49 US Department of Health & Human Services Administration for Children & Families - Office of Planning, Research & Development The Program Manager's Guide to Evaluation mguide/pmguide_toc.html This guide explains program evaluation -what it is, how to understand it, and how to do it. It answers your questions about evaluation and explains how to use evaluation to improve programs and benefit students, families, and staff. Bibliography & Resources

50 Questions & Answers