1 Evaluation is a Partnership Dr. Darla M. Cooper Associate Director Center for Student Success Basic Skills Coordinators Workshop September 17, 2009

2 Overview Importance of Evaluation Importance of Evaluation Balance between Reality and Rigor Balance between Reality and Rigor Direct and Indirect Measures Direct and Indirect Measures Relationship between Faculty and Researchers Relationship between Faculty and Researchers How Researchers Can Help Faculty How Researchers Can Help Faculty What Researchers Need from Faculty What Researchers Need from Faculty Lessons Learned Lessons Learned

3 Importance of Evaluation Demonstrates whether program/project is having desired impact on students Demonstrates whether program/project is having desired impact on students Identifies what is working and what needs improvement Identifies what is working and what needs improvement Measures effect of any changes made within program/project Measures effect of any changes made within program/project Enables ongoing internal and external sharing/reporting of evaluative results Enables ongoing internal and external sharing/reporting of evaluative results Helps justify continued support and funding Helps justify continued support and funding

4 Balance between Reality and Rigor First identify data already being collected First identify data already being collected Data collection should not place an undue burden on the program/project Data collection should not place an undue burden on the program/project Use direct measures whenever possible and reasonable Use direct measures whenever possible and reasonable Need to ensure that data being collected are actually measuring what you intended to assess Need to ensure that data being collected are actually measuring what you intended to assess Requires conversation between program/project leaders and researcher to achieve a suitable balance Requires conversation between program/project leaders and researcher to achieve a suitable balance

5 Direct and Indirect Measures Direct Measures Provide evidence of cognitive (knowledge) or behavioral (skills) learning that directly corresponds to specific intended learning outcomes Provide evidence of cognitive (knowledge) or behavioral (skills) learning that directly corresponds to specific intended learning outcomes Examples: exams, papers, grades, portfolios Examples: exams, papers, grades, portfolios Indirect Measures Assess whether learning has been meaningful by gathering information related to perceptions, opinions, experiences, and achievements Assess whether learning has been meaningful by gathering information related to perceptions, opinions, experiences, and achievements Examples: surveys, journals, graduation rates Examples: surveys, journals, graduation rates

6 Direct and Indirect Measures DirectIndirect Measures actual learning YesNo Gathering information More challenging More accessible Objectivity More objective More subjective Reflects factual information More facts More beliefs Insight into learning experience Less likely More likely

7 Relationship between Faculty and Researchers Researchers need to understand the program/project Researchers need to understand the program/project Project/program leaders need to understand demands on researcher Project/program leaders need to understand demands on researcher Develop Evaluation Plan together Develop Evaluation Plan together Researcher is seen as a member of the team Researcher is seen as a member of the team Ongoing relationship is key to ongoing success of the evaluation Ongoing relationship is key to ongoing success of the evaluation

8 How Researchers Can Help Faculty Provide options for assessment methods Provide options for assessment methods Share knowledge of data already available Share knowledge of data already available Facilitate accurate data interpretation Facilitate accurate data interpretation Listen Listen NOT dictate data to be used NOT dictate data to be used NOT advocate changing program/project to fit data NOT advocate changing program/project to fit data

9 What Researchers Need from Faculty To be invited and included in the conversation early To be invited and included in the conversation early Share details of the project/program (e.g., goals, data, interventions, intended outcomes) Share details of the project/program (e.g., goals, data, interventions, intended outcomes) Consider options provided by researcher and provide constructive feedback Consider options provided by researcher and provide constructive feedback To be kept involved and informed about ongoing progress and changes in the program/project To be kept involved and informed about ongoing progress and changes in the program/project Work with researcher to make the data meaningful and useful Work with researcher to make the data meaningful and useful Patience and understanding Patience and understanding

10 Exercise Common types of programs/projects: Learning Communities Learning Communities Tutoring Tutoring Supplemental Instruction Supplemental Instruction First-Year Experience First-Year Experience

11 Exercise Preparing to Meet with Researcher Have answers to the following questions: 1. Goal(s) – What effect do you intend the program/project to have? 2. Outcomes – What tangible results do you expect to see from students? 3. Intervention – What treatment(s) students will receive? Where and when are the points of contact with students? 4. Data – What data will demonstrate intended outcomes? What data are the program/project already collecting or planning to collect?

12 Exercise Preparing to Meet with Researcher Examples: 1. Goal(s) – Improve students’ success in basic skills courses through …? 2. Outcomes –Users have higher success rates than non-users; Increased rate of success to transfer over time 3. Intervention – one-on-one tutoring, phone calls during 1 st three weeks of term 4. Data – Course grades, test results, portfolio assessments, surveys, ?

13 Case Study SBCC’s Partnership for Student Success (What really happens when faculty and researchers get together)

14 Lessons Learned Start discussion with what you want to know about program/project Start discussion with what you want to know about program/project Decide on data collection BEFORE implementation, if possible Decide on data collection BEFORE implementation, if possible Be flexible, open and available Be flexible, open and available Be involved, invested and stay informed Be involved, invested and stay informed Work together as partners Work together as partners Data and research are your friends!!! Data and research are your friends!!!

15 Questions?

16 Thank You!!