1. Partnering with an Evaluator Presented by Joanne Kahn 1

2. Session goals  Understand how to hire and partner with a consultant to assist you with evaluation.  Set the stage for an effective evaluation. 2

3. 1 - Project definition 2- Vendor selection 3- Contract negotiation 4- Data collection 5- Reporting Steps in the partnership 3

4. Project Definition: what to include? Things you could do in-house: Library use statistics Point-of-use inquiry by staff Skills tests Feedback forms/surveys (but could get design help) Outside firm helpful for: Survey design Focus groups Interviews Observation Instructor assessments 4

5. Steps in the partnership Project definition Vendor selection Contract negotiation Data collection Reporting 5

6. Vendor selection Does your library have an existing relationship with a particular research firm? To find an evaluator, ask peers or check the American Evaluators Association website www.eval.org.www.eval.org Members commit to: Systematic inquiry Competence Integrity/honesty Respect for people Responsibilities for general and public welfare 6

7. Vendor selection Interview for: Compatibility with your goals Flexibility Experience with methods you want to use Staff who speak/write in needed languages Schedule availability Willing to work within your budget constraints A portfolio of similar work References — are they easy to work with? See handout: article from “Usable Knowledge.” www.usablellc.net/White_Papers/Hiring%20an%20Evaluation%20Consultant.pdf 7

8. Points to discuss with the evaluator  Timing and frequency  Who will do each task?  Which participants? How many?  Protecting the confidentiality of participants’ personal information  How will participants be recruited?  Incentives and perhaps transportation for participants 8

9. Evaluator’s perspective* *Special thanks to Juliette Mackin of NPC Research.  Focus on the story you want to tell stakeholders, and collect the data you need to tell it.  If you change staff (or delegate), make sure everyone knows the goals.  Plan how you will handle unexpected or negative feedback. The evaluator will take the approach that all information is useful. 9

10. Steps in the partnership Project definition Vendor selection Contract negotiation Data collection Reporting 10

11. Contract negotiation  Ask for a written proposal and fixed-price bid, and understand how the total was reached (hourly rates and line-item costs).  If bid exceeds budget, look for ways to reduce costs: Junior staff members may be less expensive than partners You may be able to reduce their time by:  Arranging the venue, incentives, and refreshments  Providing pre-screened participants with phone/email contact info  Bundling this project with another 11

12. Contract negotiation  Set a schedule that includes time for review and revisions.  Agree up front: What intermediate deliverables will be provided? Who needs to sign-off on the report? 12

13. Steps in the partnership Project definition Vendor selection Contract negotiation Data collection Reporting 13

14.  You may draft questions for the evaluator to review and include.  Have you made the process as participant-friendly as possible?  Measure before/after results.  Are the questions clear? For multiple choice questions, are the choices adequate?  Are there open-ended questions? Data collection — surveys 14

15.  How will you recruit/screen participants?  Have you made the logistics as participant-friendly as possible?  The evaluator will probably want you out of the room.  Review script for completeness, especially follow-on questions.  Will there be a transcript or recording? Data collection — focus groups 15

16. Steps in the partnership Project definition Vendor selection Contract negotiation Data collection Reporting 16

17. Reporting Respect the firm’s objectivity! But it’s fair to ask them to have a positive outlook. 17

18. Reporting: make it readable! o Executive summary conveys the main points o Outline is easy to follow o Visually attractive – photos!! o Do tables/graphs improve the report? o Are patron comments or anecdotes included? o Patterns — do data sources corroborate each other? o Conclusions should follow from the data 18