PROGRAM EVALUATION Carol Davis

Purpose of Evaluation The purpose of evaluation is to understand whether or not the program is achieving the intended outcomes. Helps identify what works and does not work.

Today’s Evaluation Discussion 1. PEEC Outcomes (Indigenous Evaluation) 2. Evaluation Plan 3. Formative Evaluation 4. Progress Evaluation 5. Summative Evaluation 6. Write Report

PEEC: Indigenous Evaluation Framework

Indigenous Evaluation Consultants American Indian Spiritual Leaders, Cultural Leaders and Educators assisted in the preparation of this guide. It’s main function is for evaluation of Indian Education programs.

Four Core Values Emerged 1. Tribes are a people of place 1. Tribes are a people of place 2. Tribes recognize their gifts 2. Tribes recognize their gifts 3. Tribes Honor family 3. Tribes Honor family 4. Tribes respect sovereignty 4. Tribes respect sovereignty

Indigenous Evaluation Framing Principles Tribal Colleges reflect tribal core values. Tribal Colleges reflect tribal core values. When we create programs, we are creating a set of understandings. When we create programs, we are creating a set of understandings. These understandings are our hypothesis of what we believe will happen. These understandings are our hypothesis of what we believe will happen. Our evaluations need to tell our story. Our evaluations need to tell our story.

Guiding Principles  An Indigenous framework can incorporate broadly held values while also remaining flexible and responsive to local traditions and cultures.  Responsive evaluation uses practices and methods from the field of evaluation that fit our needs and conditions.

Culture “[It] is culture that provides the tools for organizing and understanding our worlds in communicable ways.” “[It] is culture that provides the tools for organizing and understanding our worlds in communicable ways.” - Jerome Bruner- The Culture of Education - Jerome Bruner- The Culture of Education

Traditional Knowledge “Every act, element, plant, animal, and natural process is considered to have a moving spirit with which humans continually communicate.” “Every act, element, plant, animal, and natural process is considered to have a moving spirit with which humans continually communicate.” -Dr. Gregory Cajete- -Dr. Gregory Cajete- Native Science: Natural Laws of Native Science: Natural Laws of Interdependence (2000) Interdependence (2000)

Evaluation Plan The purpose of evaluation planning is to assess the understanding by all stakeholders of… The purpose of evaluation planning is to assess the understanding by all stakeholders of…  Project Goals  Objectives  Strategies, and  Timelines

Evaluation Planning is Guide for Formative and Summative When you “plan to tell your story,” your formative and summative evaluation becomes more than numbers. When you “plan to tell your story,” your formative and summative evaluation becomes more than numbers. You collect information. You do surveys. You collect information over time. You analyze results. You collect information. You do surveys. You collect information over time. You analyze results.

Evaluator Role  To assure that the program evaluation will meet the needs of the organization, reservation, and/or community, require that the evaluator submit an evaluation proposal prior to beginning the project.

Intended Outcomes Outcomes create the desired change or impact we seek through this program. Outcomes create the desired change or impact we seek through this program. Stakeholders should brainstorm the intended outcomes and post them. That will assure that everyone is working toward the same outcomes. Stakeholders should brainstorm the intended outcomes and post them. That will assure that everyone is working toward the same outcomes.

Examining the Story  What do we want to know as the story unfolds?  Drafting questions/statements guide the process of tracking the story.

Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data

Objectives Objectives are measurable. Objectives are measurable.

Outputs Your outputs result from your activities. Your outputs result from your activities. Staff can assist in determining the outputs for each activity. Staff can assist in determining the outputs for each activity. A process for “documenting” is essential and necessary when writing reports. A process for “documenting” is essential and necessary when writing reports.

A Program is a story PEEC may want more than numbers. What is PEECs legacy? You may want to find out how the students rate their participation in your program or how faculty view your services. PEEC may want more than numbers. What is PEECs legacy? You may want to find out how the students rate their participation in your program or how faculty view your services.

Survey Monkey SurveyMonkey's online survey tool is easy and quick. Create many types of surveys using their methodologist- certified survey templates. SurveyMonkey's online survey tool is easy and quick. Create many types of surveys using their methodologist- certified survey templates.

Report Program Impact? VALUE ADDED: VALUE ADDED: e.g., Do you have a student who is the first to attend college in the family? e.g., Do you have a student who is the first to attend college in the family? Do you have a student who comes from a foster home who is going to become an engineer? Do you have a student who comes from a foster home who is going to become an engineer? Is this the first tribal member who aspires to become an engineer? Is this the first tribal member who aspires to become an engineer?

Quantitative/Qualitative  Both methods can contribute to telling the story. You decide which is appropriate.  Quantitative can build Breadth (These are your numbers, percentages, etc.).  Qualitative can build depth (These are your personal stories, focus groups, etc.).

Qualitative Design  It is important for American Indian tribes to include qualitative reporting methods.  Truth emerges from consensus among informed people.  Value framework gives meaning to numbers.

Formative Evaluation  Formative Evaluation is assessment of the ongoing projects and activities  Evaluation should be conducted for action- related reasons and the information should be used to decide a course of action.

Progress Evaluation  Determine whether or not the project is being conducted as planned within the established timelines.

Summative Evaluation  Summative Evaluation is conducted at or toward the end of the program. The purpose is to assess the projects outcomes to determine its success.

Use of Results of Summative Evaluation  Sustainability  Disseminate your model  Continue funding  Continue on probation, and/or  Discontinue

Writing the Report Writing the Report Gather and organize the documents, data, Survey Monkey results & other information Gather and organize the documents, data, Survey Monkey results & other information Review the report format, instructions, and due date. Review the report format, instructions, and due date. Create a checklist to make sure you include all of the information. Create a checklist to make sure you include all of the information. Answer basic question: who, what, when, where, why, how, how many. Answer basic question: who, what, when, where, why, how, how many.

Today’s Evaluation Discussion 1. PEEC Outcomes (Indigenous Evaluation) 2. Evaluation Plan 3. Formative Evaluation 4. Progress Evaluation 5. Summative Evaluation 6. Write Report

References  American Indian Higher Education Consortium Indigenous Evaluation Framework.  Boyer, Paul Should Expediency Always Trump Tradition? AIHEC/NSF Project develops indigenous evaluation methods. Tribal College Journal. Vol. 18 No. 2. Pp  LaFrance, J., & Nichols, R. (2010). Reframing Evaluation:Defining Indigenous Eval. Framwork. Canadian Journal of Eval. 23(2),  NSF.Gov/Evaluation. March 3,  March 3,

MIIGWECH (Thank You)