Presents: Information for participants: Your microphone will be muted for the formal presentation. If your audio portion the presentation is not working, try running the Audio Set Up Wizard (under meeting). How to Engage: 1)Use the “Set Status” feature. 2)Type questions into the chat box. We will address them right away or as part of scheduled Q&A breaks. 3)Please send follow up questions to the Project Evaluator Network Working Group: 1 "6 Steps for STEP Evaluation” Presented by Dr. David Blair at St. Edward's University and Dr. Raymond McGhee at SRI International.

David Blair, St. Edwards University Raymond McGhee, SRI International 6 Steps to get started on your STEP Evaluation

Objectives Promote an exchange of ideas on relevant evaluation topics in the STEP CoP. Provide a general framework or roadmap for thinking about STEP evaluation tasks. Set the stage for continued one-on-one interactions between STEP project evaluators. DB-3

Overview Engage project leaders in evaluation design and development Document project achievements Describe challenges and adjustments made by the project Use your different advisory committees to guide and inform your evaluation planning Analyze data to document implementation, lessons learned and impact Share results in a strategic fashion with key stakeholders RM-4

Opening Question Who should participate in evaluation planning and why should they participate? 5

STEP 1 RM-6

Communicate early and regularly with leadership Document decisions and next steps Show how evaluation results can be leveraged to improve implementation Be persistent without being annoying PI/Project Leader Collaboration

Opening Question: What are the key research questions that your project wants to document? Why? 8

STEP 2 DB-9

Key Activities/Number of students impacted Data Collection Student Characteristics Surveys Focus Groups Journals Impact Data Other Document Achievements

Successes Experienced Increased GPA, retention, research interest, cohort building, student satisfaction Project activities integrated at institutions Off-shoots Different versions Other project impacts beyond intended goals Socialization, confidence Document Achievements (cont’d).

Partnership achievements Roles Changes as project proceeds Unexpected benefits Meetings/Groups Additional resources Document Achievements (cont’d.)

Opening Question: What can be learned from a project’s misfires or mistakes in implementation? 13

STEP 3 DB-14

Challenges faced Grant notification Recruitment of participants Funding for additional staff Finding an evaluator Students not taking advantage of resources Significant changes Document fully Describe challenges/adaptation/ assessment/sustainability

Intermediate measures and metrics to monitor progress Data Analysis Quantitative Qualitative Mixed Methods Describe challenges/adaptation/ assessment/sustainability (cont’d.)

Opening Question: How can your advisory committee(s) best contribute to your project evaluation? 17

STEP 4 RM-18

Advisory Board Involvement Internal – use them to help facilitate data collection and analysis efforts as well as interpreting the results of data analyses. This board may also help frame results for different stakeholders within the participating institutions. External – use them to help you identify key audiences for your evaluation results as well as the key messages you want to share based on your evaluation results. 19

Opening Question: Why is it important for an evaluator to adopt "methodological pluralism" with respect to collecting and analyzing project evaluation data? 20

STEP 5 DB-21

Lessons Learned/Overall Impact Summary of how well project is meeting project goals Tie back to original goals Work with your external evaluator Fully explain changes and corrections Actions taken to sustain project Additional funding Integrating within the university culture and budget 22

Opening Question: How can we better target our project evaluation findings so that they have the largest impact on institutional practice? 23

STEP 6 RM24

Dissemination of Results Rationale – share successes to key stakeholders to secure support for additional funding and institutionalization of project efforts. Rationale can be sharpened with external advisory input. 25

Dissemination of Results (cont’d.) Targets – with input from external advisory committee, identify multiple audiences who could use the results of your evaluation. Include targets at different levels of the institution as well as key external audiences (e.g., industry partners, sister institutions, 3 rd Year Review board.) Also determine what messages, metrics, or narratives speak to the concerns of these audiences. 26

Dissemination of Results (cont’d.) Methods – develop a dissemination plan that harnesses multiple modes of disseminating your evaluation’s results. Include print as well as electronic modes of sharing your work. 27

Dissemination of Results (cont’d.) Products – based on the audiences identified, develop several dissemination products tailored for each audience. Collaborate PI and external advisory panel’s recommendations to determine the high value products. 28

6 Steps to get You Started 1. Project leader collaboration 2. Document achievements 3. Describe challenges/adaptation/assessment/ sustainability 4. Advisory board involvement 5. Document implementation, lessons learned and impact 6. Dissemination of results 29

30 Conclusion/Questions Future materials and resources Suggested Checklist to help with plan preparation General project evaluation references