Partnering with an Evaluator Presented by Joanne Kahn 1

Session goals  Understand how to hire and partner with a consultant to assist you with evaluation.  Set the stage for an effective evaluation. 2

Project definition Vendor selection Contract negotiation Data collection Reporting Steps in the partnership 3

What will you contract out? Things you probably do in-house: Library use statistics Point-of-use inquiry by staff Skills tests Feedback forms/surveys (but could get design help) Outside firm helpful for: Survey design Focus groups Interviews Observation Instructor assessments 4

Steps in the partnership Project definition Vendor selection Contract negotiation Data collection Reporting 5

Vendor selection Does your library have an existing relationship with a particular research firm? To find an evaluator, ask peers or check the American Evaluators Association website Members commit to: Systematic inquiry Competence Integrity/honesty Respect for people Responsibilities for general and public welfare 6

Vendor selection Interview for: Compatibility with your goals Experience with methods you want to use Staff who speak/write in needed languages Schedule availability Willing to work within your budget constraints A portfolio of similar work References — are they easy to work with? See handout: article from “Usable Knowledge.” 7

Evaluator’s perspective* *Special thanks to Juliette Mackin of NPC Research.  Focus on the story you want to tell stakeholders, and collect the data you need to tell it.  If you change staff (or delegate), make sure everyone knows the goals.  Plan how you will handle unexpected or negative feedback. The evaluator will take the approach that all information is useful. 8

Points to discuss with the evaluator  Goals  Timing and frequency  Who will do each task?  Which participants? How many?  Protecting the confidentiality of participants’ personal information  How will participants be recruited?  Incentives and perhaps transportation for participants 9

Steps in the partnership Project definition Vendor selection Contract negotiation Data collection Reporting 10

Contract negotiation  Ask for a written proposal and fixed-price bid, and understand how the total was reached (hourly rates and line-item costs).  If bid exceeds budget, look for ways to reduce costs: Junior staff members may be less expensive than partners You may be able to reduce their time by:  Arranging the venue, incentives, and refreshments  Providing pre-screened participants with phone/ contact info  Bundling this project with another 11

Contract negotiation  Set a schedule that includes time for review and revisions.  Agree up front: What intermediate deliverables will be provided? Who needs to sign-off on the report? 12

Steps in the partnership Project definition Vendor selection Contract negotiation Data collection Reporting 13

 You may draft questions for the evaluator to review and include.  Have you made the process as participant-friendly as possible?  Measure before/after results.  Are the questions clear? For multiple choice questions, are the choices adequate?  Are there open-ended questions? Data collection — surveys 14

 How will you recruit/screen participants?  Have you made the logistics as participant-friendly as possible?  The evaluator will probably want you out of the room.  Review script for completeness, especially follow-on questions.  Will there be a transcript or recording? Data collection — focus groups 15

Steps in the partnership Project definition Vendor selection Contract negotiation Data collection Reporting 16

Reporting Respect the firm’s objectivity! But it’s fair to ask them to have a positive outlook. 17

Reporting: make it readable! o Does the executive summary convey the main points? o Is the outline easy to follow? o Is information made visually attractive? o Do tables/graphs improve the report? o Are there photos you can provide? o Does the report identify patterns — do data sources corroborate each other? o Do conclusions follow from the data? o Are patron comments or anecdotes included? 18