Selecting Your Evaluation Tools Chapter Five

Introduction  Collecting information  Program considerations  Feasibility  Acceptability  Credibility

Feasibility  Where do new tools come from?  What to use as an outcome tool  Validation of a tool  Reliability of a tool  Personnel time to administer tools  Collection of data is key

Acceptability  From the participants perspective  Give good reasons for being in the evaluation  Respect confidentiality  What is the total time commitment?  Fill out a questionnaire  Personal or phone interview  Focus group of several people

Acceptability cont…  English as a second language  Literacy of participants  Age associated hearing and vision loss  Several sessions instead of one long one  Interview based vs. self completion  Use a simple Likert scale

Credibility  Justification of your program  Consult users of info before collecting  Purpose of data collection?  Rationale for tools?  Cost and time expected?  Type of info obtained?

Existing Outcome Measures  Use published measures  They have scoring and administration rules  Reliability is consistency or repeatability  Test re-test reliability  Validity is truthfulness of tool  Does it measure what it says it measures

Consistency in Collection  Standardized collection guidelines  Administer to each client the same way  Procedures should be standardized  Sample representativeness  Sample at different times  Get input from drop-outs

Familiarity  Quantitative data  Numbers representing data  Qualitative data  Answers to open ended questions  Performance measures may be used  Self-report questions may also be used

Familiarity cont…  Client satisfaction surveys  Ease of development  Reach large number of clients  Takes little time  Ease of presentation of findings  Problems with self report questionnaires does not preclude their use

Existing Records  Ease of use makes them attractive  Data is already in database  No client participation time or effort  Low cost of data collection  Table 5.1 page 67

Observation  Site visits  Observing exercise sessions  Observing member use of facilities  Case studies following one participant  Personal testimonials can be powerful  Functional performance measures

Questionnaires  Designed for self completion  Registration forms and satisfaction forms  Close ended questions  Primarily quantitative  Reaches highest number of people  Completion at site or take home  Administrative assistance is needed

Questionnaires cont…  Can use phone or mail  Absentee clients may be reached  Drop-outs may also be reached  Cost of mailing and return postage  surveys may be advantageous  All mail needs to be personalized

Interviews  May be unstructured  Difficult to interpret data  Often last several hours  Primarily qualitative  Read the questionnaire and record responses