1. Developing Your Evaluation Skills

2. Methods of Evaluation

3. Data Collection Methods   Survey   Interview   Observation   Video, photograph   Case study   Test   Group technique   Environmental scan   Document/data review, analysis   Portfolio review   Testimonial   Expert or peer review   Simulation   Journal, log, diary   Unobtrusive measure

4. Surveys & Questionnaires

5. Surveys/Questionnaires   Advantages: – –Uniformity of data – –Efficiency of administration – –Anonymous – –Multiple administration – –Record obtained

6. Surveys/Questionnaires   Questions to ask: – –Is self-report adequate? – –Are there resources to develop and administer and summarize? – –Can I access respondents? – –Will I mail, e-mail, phone? – –Is target audience able/willing?

7. Surveys/Questionnaires   Design variables: – –Information needed – –Wording of questions – –Order of questions – –Structure of questions – –Demographic needs

8. Surveys/Questionnaires   To increase response rates: – –Pilot test – –Transmittal/introductory letter from “High Level Official” – –Mail advance postcard – –Use check marks mainly – –Polished professional copy – –Phone or e-mail reminder – –Use few open-ended questions – –Short, clear, meaningful questions

9. Case Studies

10.   Put a“human face” on impact data   Illustrate complex interactions   Show multiple reinforcers   Provide program re-design rationale   Share unexpected impacts

11. Case Studies Advantages:   In-depth understanding   Insights into feelings and casual relationships   Individualized Strengths:   Understand and improve program   Fuller understanding of influences on behavior   Plan for a quantitative study

12. Case Studies Disadvantages:   Subjective data which may not be representative   Incomplete as a “total picture” of program impact

13. Case Studies First Decision:   What is the unit of analysis? – –One family, individual, business – –One group (e.g. 4-H Club, Advisory Committee, etc.) – –A whole program (e.g. IPM, FSNEP, Shooting Sports, etc.)

14. Case Studies Next: All data collection is directed to obtain a complete picture of that unit:   Observations   Interviews   Focus groups   Questionnaires   Testimonials

15. Case Studies What case?   Typical example   Unique story: New audience, creative use, new opportunities   Emphasize empowerment of learner   Show barriers to change

16. Group Techniques of Evaluation Advantages   Can be rapid   Creative variety/movement   Use group reinforcement and commitment

17. Group Techniques of Evaluation Disadvantages   Limited information   May require public disclosure

18. Group Evaluation Techniques Disclosure Hand Raise Dot Technique Value Continuum Small Group Discussions Partner Commitments Card Sort

19. Group Evaluation Techniques Private Short Survey Short Interviews Call Back Volunteers Commitment plus Self-Addressed Post Card