Developing Your Evaluation Skills

Methods of Evaluation

Data Collection Methods   Survey   Interview   Observation   Video, photograph   Case study   Test   Group technique   Environmental scan   Document/data review, analysis   Portfolio review   Testimonial   Expert or peer review   Simulation   Journal, log, diary   Unobtrusive measure

Surveys & Questionnaires

Surveys/Questionnaires   Advantages: – –Uniformity of data – –Efficiency of administration – –Anonymous – –Multiple administration – –Record obtained

Surveys/Questionnaires   Questions to ask: – –Is self-report adequate? – –Are there resources to develop and administer and summarize? – –Can I access respondents? – –Will I mail, , phone? – –Is target audience able/willing?

Surveys/Questionnaires   Design variables: – –Information needed – –Wording of questions – –Order of questions – –Structure of questions – –Demographic needs

Surveys/Questionnaires   To increase response rates: – –Pilot test – –Transmittal/introductory letter from “High Level Official” – –Mail advance postcard – –Use check marks mainly – –Polished professional copy – –Phone or reminder – –Use few open-ended questions – –Short, clear, meaningful questions

Case Studies

  Put a“human face” on impact data   Illustrate complex interactions   Show multiple reinforcers   Provide program re-design rationale   Share unexpected impacts

Case Studies Advantages:   In-depth understanding   Insights into feelings and casual relationships   Individualized Strengths:   Understand and improve program   Fuller understanding of influences on behavior   Plan for a quantitative study

Case Studies Disadvantages:   Subjective data which may not be representative   Incomplete as a “total picture” of program impact

Case Studies First Decision:   What is the unit of analysis? – –One family, individual, business – –One group (e.g. 4-H Club, Advisory Committee, etc.) – –A whole program (e.g. IPM, FSNEP, Shooting Sports, etc.)

Case Studies Next: All data collection is directed to obtain a complete picture of that unit:   Observations   Interviews   Focus groups   Questionnaires   Testimonials

Case Studies What case?   Typical example   Unique story: New audience, creative use, new opportunities   Emphasize empowerment of learner   Show barriers to change

Group Techniques of Evaluation Advantages   Can be rapid   Creative variety/movement   Use group reinforcement and commitment

Group Techniques of Evaluation Disadvantages   Limited information   May require public disclosure

Group Evaluation Techniques Disclosure Hand Raise Dot Technique Value Continuum Small Group Discussions Partner Commitments Card Sort

Group Evaluation Techniques Private Short Survey Short Interviews Call Back Volunteers Commitment plus Self-Addressed Post Card