1. Getting Started With Your ATE Evaluation ATE PI Conference October 24, 2012 This material is based upon work supported by the National Science Foundation under Grant Nos. 0802245 0802245 and 1204683. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of NSF.

2. To promote the goals of the ATE program by - partnering with ATE projects and centers to strengthen the program's evaluation knowledge base - expand the use of exemplary evaluation practices - support the continuous improvement of technician education throughout the nation ’s Mission

3. - Understand the role evaluation should play in your ATE project - Have clear expectations for your evaluator - Know where to turn if you need more guidance about your ATE evaluation Session Objectives

4. Formative Evaluation Learn how the project could be improved as it is implemented Summative Evaluation Provide evidence of the quality and impact of your achievements Accountability Document what you did with the grant money Evaluation Purposes

5. The most important purpose of evaluation is not to prove, but to improve. ― Daniel Stufflebeam Formative Evaluation

6. Project staff… - make time for evaluation - are open to hearing negative findings - use feedback to inform decision making Formative Evaluation

7. Evaluator… - learns about the project, its context, and stakeholders - gathers data in sound, efficient ways - provides timely feedback - guides project staff in understanding and using results Formative Evaluation

8. What is the project’s quality? What are the project’s impacts? Summative Evaluation

9. What is the project’s quality? What are the project’s impacts? - changes in knowledge, skills, abilities, attitudes, performance, practices, or policies - show up at the individual, program, institution, organization, regional, or national level - occur in the short term, long term, or in between Summative Evaluation

10. Document… - What you did - Who you reached - Who you worked with - What you produced (outputs) Accountability

11. www.evalu-ate.org/about_us Create a project vita - Mission - Goals - Funding - Activities - Products - Staff - Students - Contributors & collaborators - Consultants

12. Formative Evaluation How is the project doing and how can it improve? Accountability What was done, who was served, who contributed, what are the outputs? Summative Evaluation What is the quality and impact of the project? Evaluation Purposes & Uses

13. Formative Evaluation How is the project doing and how can it improve? Accountability What was done, who was served, who contributed, what are the outputs? Summative Evaluation What is the quality and impact of the project? Evaluation Purposes & Uses - Feedback to internal stakeholders - Reflection and redirection

14. Formative Evaluation How is the project doing and how can it improve? Accountability What was done, who was served, who contributed, what are the outputs? - Summative report to key stakeholders - FastLane/annual report to NSF - ATE Impacts book - Annual ATE survey - New proposals - Promotion and dissemination Summative Evaluation What is the quality and impact of the project? Evaluation Purposes & Uses - Feedback to internal stakeholders - Reflection and redirection

15. Report Sections - Participants - Activities & findings - Publications & products - Contributions Annual Reports to NSF www.fastlane.nsf.gov

16. See also: EvaluATE’s summary of FastLane report components evalu-ate.org/resources Keyword: FastLane Annual Reports to NSF

17. For example, # of students and their demographic characteristics # of professional development participants # of materials developed www.evalu-ate.org/annual_survey Annual ATE Survey

18. Student impact data Workforce impact data www.atecenters.org www.ateprojectimpact.org ATE Projects/Centers Impact Book & Website

19. “ “ The Project Description must begin with the subsection on Results of Prior Support…. This subsection must contain specific outcomes and results including metrics to demonstrate the impact of the activities undertaken including evidence of the quality and effectiveness of the project's deliverables. —ATE Program Solicitation “ Results of Prior NSF Support

20. Be an Informed Consumer Joint Committee on Standards for Educational Evaluation The Program Evaluation Standards www.jcsee.org American Evaluation Association Guiding Principles for Evaluators www.eval.org Canadian Evaluation Society Competencies for Evaluation Practice www.evaluationcanada.ca

21. What should we expect to receive from our evaluator? ― Participant in the 2010 ATE PI Conference Getting Started Workshop

22. - Detailed evaluation plan - Opportunities to review and provide feedback on draft plans and materials - Interim reports - Annual reports - Knowledge and insights you could not have obtained on your own Evaluation Deliverables

23. EVAL REPORT EVALUATORPI PROGRAM OFFICER OTHER PROJECT STAKEHOLDERS, E.G.: partners, advisors, participants, host administrators Reporting

24. EVAL REPORT EVALUATORPI PROGRAM OFFICER OTHER PROJECT STAKEHOLDERS, E.G.: partners, advisors, participants, host administrators Reporting

25. EVAL REPORT EVALUATORPI PROGRAM OFFICER OTHER PROJECT STAKEHOLDERS, E.G.: partners, advisors, participants, host administrators Reporting

26. EVAL REPORT EVALUATORPI PROGRAM OFFICER OTHER PROJECT STAKEHOLDERS, E.G.: partners, advisors, participants, host administrators Reporting

27. EVAL REPORT Ways of reporting evaluation results to your program officer PI PROGRAM OFFICER Ask your program officer how he/she prefers the evaluation results to be conveyed. Complete original report Original executive summary Evaluators’ summary in annual report Reporting

28. EVAL REPORT Ways of reporting evaluation results to your program officer PI PROGRAM OFFICER Ask your program officer how he/she prefers the evaluation results to be conveyed. Complete original report Original executive summary Evaluators’ summary in annual report ASK! Reporting

29. Using EvaluATE www.evalu-ate.org

30. Newsletters Feature articles by experienced ATE PIs and evaluators Q & A Definitions & discussion of evaluation terms Resource highlights

31. Resource Library Search by - Evaluation task - ATE topic - Keyword - Author

32. Evaluator Directory Search by - location - discipline - project type - keyword

33. Annual Survey Survey form FAQs Reports

34. Webinars Upcoming events Materials from past events: - Recordings - Slides - Handouts

35. Next Webinar Register at www.evalu-ate.org/events

36. Conference Activities Concurrent with this session PI and evaluator workshop on evaluation Tonight Showcase #209-210 Tomorrow Afternoon Birds of a Feather Session: Beyond Satisfaction and Short-Term Self-Reports: Evaluating the Impact of Your ATE Grant

37. Thank You!

38. Advice from the Trenches Elaine’s Golden Rule: The one who has the gold rules! i.e., don’t disperse grant funds to contractors or partners until obligations have been met “ “ Elaine Craft ATE PI & Evaluator

39. Advice from the Trenches Set up regular data collection milestones or checkpoints so that data is being collected as part of the normal process of carrying out the grant and doesn't have to be ‘bolted’ onto the process of the grant after it is underway. “ “ Karl Kapp ATE Evaluator

40. Advice from the Trenches The evaluator is the expert in evaluation but you are the expert on your project. If you feel that the evalua- tion is not providing the information you need, then talk to your evaluator. Do it immediately. … You can change evaluators but first, communicate, communicate. communicate. “ “ Jane Ostrander ATE PI & National Visiting Committee chairperson