Evaluation Design Designing & Refining Questions Ben Silliman Youth Development Specialist NC 4-H

Who Knows What? Youth (over age 3) can report their –Feelings, interests, experiences, goals, opinions/ attitudes on knowledge and skills –(reliable observers may give opinions on these) General Rules for who answers what: –You cant know what you dont know (thus retrospective or observer ratings usually better) –No one knows your feelings, experiences, and attitudes/aspirations better than you (let respondents speak for themselves)

Who Knows What? Youth (over age 3) can complete valid, developmentally appropriate tests of –Knowledge, Skill, Practice (reliable observers may also give opinions on these) How to know what others know –Demonstration, description, or defense (authentic assessment) gives best evidence …esp. if judged by trained, reliable observers using the same criteria given to youth

Thinking About Questions Adjust for Differences in Self-Administered vs. Interviewer-Administered Questionnaires Self: make it simple (Eighth Grade), straightforward, and logical. Q1-> Q2 (complex skip patterns, crowded pages, complex instructions delay and discourage respondents) Interviewer: accommodate skip patterns, allow for diverse lines of questioning, interaction about ideas, objects, events (note interviewer skills)

Thinking About Questions Order questions for smooth flow –Be forthcoming about purpose and sponsor –Group topic or type of questions –Move from general to specific questions –Ask the easy questions first-build rapport –Ask more complex or thought-provoking questions near the middleuse rapport –Ask sensitive questions near the end-ditto

Thinking About Questions Phrase questions carefully to avoid bias Leading the respondent –What do you like about… vs –W hat, if anything, do you like… (not assuming respondents like something)

Thinking About Questions Keep audience focused Avoid multiple-option items –Did public speaking help you gain confidence and skill? Avoid complex phrases (Answer Never-Seldom-Often-Always) –This 4-H program did not have an appropriate level of discipline

Thinking About Questions Adjusting questionnaire length, remembering that –A relevant, purposeful study can motivate cooperation and thought –A clear and smooth-flowing routine can focus and energize respondents –-A respectful and gracious approach can affirm respondents, making them feel valued

Which Questions to Use Open-ended: no set response Is there anything else you would like to tell about your 4-H camping experience? –Communicates listening –Allows evaluator to see how respondents prioritize and organize information

Which Questions to Use Close-ended: set response choices For the 4-H project areas listed, rank (1 st, 2 nd, 3 rd ) each according to your interest: ___ Arts & Communication ___ Citizenship & Leadership ___ Family & Consumer Science ___ Health & Nutrition ___ Science & Technology

Styles of Close-ended Qs Likert (esp. for feelings, attitudes) How important are cash prizes in your decision to participate in 4-H events? Not at allVery Important 12345

Styles of Close-ended Qs Multiple Choice (best or all relevant options) What is the best thing to do when you plan not to use a gun for a while? (choose one) a)Cover it with a cloth b)Place it on top of a cabinet c)Remove ammunition d)Call law enforcement

Styles of Close-ended Qs Ordinal (ranking of options) Rank these topics in order of importance (1 st, 2 nd, 3 rd, etc) for volunteer training ___ Managing a 4-H club ___ Teaching using experiential method ___ Planning service activities ___ Conducting parties ___ Mastering program evaluation

Styles of Close-ended Qs Categorical (identifying by group) Tell us about yourself. I am aa) Girlb) Boy Numerical (identifying by number) Tell us about yourself. My age is___ years