Evaluating Program Success Cherie McCraw Born to Read Institute November 2001

Plan for Evaluation Evaluate with Passion and Purpose n Decide what is appropriate n Decide what is feasible n Decide what is adequate n Dont reinvent the wheel

Traditional Project Measurement InputsActivitiesOutputs Resources - Funding - Staff - Volunteers - Equipment Services - Training - Tutoring - Mentoring - Installing Products - # of classes - # of students - # of books - # participants

We dont just look at countable things anymore. We look at the outcome(s) of our programs… how are people changed? how are people benefited?

Outcomes Measurement InputsActivitiesOutputsOutcomes Resources - Funding - Staff - Volunteers - Equipment Services - Training - Tutoring - Mentoring - Installing Products - # of classes - # of students - # of books - # participants Change in Client - Knowledge - Skills - Behavior - Condition

Pressure to Know More n From: –Funders - local, state and federal –Public - press, community How do we know the dollars make a difference? n From: –Boards –Management –Staff How do we best use our scarce resources?

Why Change? n The numbers dont answer So what? n Outcomes give us structure n Create a strong case for funds in a competitive market n Establish ourselves as players for community-wide outcomes n Just because we always have …is not a good reason to do something today

Outcomes Can: n Provide focus n Measure results n Take 3-5 years to implement

Outcomes Cant: n Tell you whether your program caused the outcome n Indicate why the level of outcome was achieved n Suggest actions to take to improve the outcome

Choosing Which Outcomes to Measure n Use the KISS method! n Look at the data you already collect n Choose outcomes that are clear and meaningful to the public n For similar projects or grants- identify common outcome(s)

Outcome Measurement n Resources –money –staff –volunteers –equipment & supplies n Constraints –laws –regulations –funders requirements InputsActivitiesOutputsOutcomes

Criteria for Inputs n Identify a resource used for the project n Identify a single element n Are concise, clear, and lingo-free n Are quantified if possible

Examples of BTR Inputs n Easy books n Childrens librarian n Partners: (ID each) n Even Start families n Deposit collections n 5 trained volunteers n 2 library computers n Printed materials from hospitals n 51 Laubach Literacy Action trained & certified volunteers n Library facilities n Matching funds n Grant funds n Office supplies n 1 County vehicle n BTR program materials

Outcomes Measurement n Services –training –education –counseling –mentoring InputsActivitiesOutputsOutcomes

Criteria for Activities n Are verbs n Are focused on the client not on the operations of the program n Show what the program does n Identify a single action

Examples of BTR Activities n Provide weekly toddler and infant storytimes in all library locations n Put packets together n Present parenting workshops n Conduct lapsits n Provide bags for parents n Train volunteers n Enhance collections with age appropriate books n Distribute packets through home visits n Enroll low income new parents/teens and their children n Provide referral service to the adult literacy tutoring program

Examples of BTR Activities n Develop brochure n Conduct BTR publicity campaign: radio, billboards, and posters n Develop a BTR Family Literacy web page with links n Conduct phone interviews, personal interviews and focus groups n Host meetings with partners n Implement a summer reading program n Create BTR newsletter and mailing list n Issue library cards to program participants

Outcomes Measurement n Products –classes taught –counseling sessions conducted –educational materials distributed –hours of service delivered –participants served InputsActivitiesOutputsOutcomes

Criteria for Outputs n Measure how many for the activities n Often are simply the activities quantified-50 computer classes taught n Measure a single activity n Do not measure a change for a client (that would be an outcome)

Examples of BTR Outputs n 120 pregnant new moms served n 8 lapsits at Health Department n 12 Expectant Mothers programs, 150 people attend n 5,000 packets n 50 new books added n 2 training sessions for project partners n # attending lapsits n # of parent programs given n # of bags distributed n # of volunteers n # web page hits n # of posters, billboards, radio PSAs n # parents/children attending storytimes

Outcomes Measurement n Benefits for People –new knowledge –increased skills –changed attitudes or values –modified behavior –improved condition –altered status InputsActivitiesOutputsOutcomes

Criteria for Outcomes n Are client-focused n Measure a single meaningful change in the client n Are within the scope of the project n Identify who achieves the outcome n Are concise, clear, and lingo-free n Are objective n Are specific

Examples of BTR Outcomes n Parents or caregivers read to their children n Parents set a goal for their own or their childs education n BTR staff gain insight into the information needs of the clients n Community organizations cooperate to provide services to break inter-generational cycle of illiteracy n Parents or caregivers use the library

We know what we want to achieve with our programs… BUT... How do we measure our success AND demonstrate to funders, boards and commissions that we were successful?

Your Evaluation Plan!

Outcomes Measurement: Evaluation Plan IndicatorsSources/Methods Data you collect to measure indicators of success will let you know you have achieved your outcome(s) and to what extent you have achieved them. InputsActivitiesOutputsOutcomes

Criteria for Indicators n Show how well a program is doing on an outcome n Show that the outcomes have been achieved- Number of parents who read daily to their children n Are stated as a number and/or a percentage n Each outcome has at least one indicator Measure at the individual, not group, level

Setting Targets n Most programs cannot establish targets in the first year of collecting data n Possible sources for targets: –Program baseline –National statistics –Best practices research

Examples of BTR Indicators n 40% program participants come to the library. n 50% increase in time parents or caregivers read to their children. n 150 library cards issued. n # of times library card is used per month. n # of positive responses to questionnaires. n # of parents sharing books and reading with their children.

Examples of BTR Indicators n # & % of teen program participants who use the library. n # & % of parents reading to their child on a regular basis. n # & % of parents attending workshops. n # & % of parents setting educational goals. n # & % of partners who say the partnership helped extend their services. n # of certificates given at end of program.

Criteria for Sources n Who provides the data n There should be at least one source for each indicator n Possible sources include: –Clients –Family members –Project staff –Existing records –Volunteer observers

Criteria for Methods n How the outcome will be measured n Provide specific data to measure an indicator n Possible methods include: –Review of project records –Questionnaire or survey –Interview –Rating by a trained observer

Examples of BTR Data Sources and Methods n Library staff/Count of coupons for diaper bags given at programs and redeemed at the library. n Parent interviews by library and Healthy Start staff. n Automated circulation system/track of usage of coded library cards. n Program administrator/Attend ance records. n Questionnaires distributed by home visitors.

Examples of BTR Data Sources and Methods n Reading diaries checked by Library staff. n Project manager/Student information forms. n Project manager/agreement forms signed by students and tutors n Project manager/Focus group with program participants. n Library staff/Telephone survey asked of BTR participants. n Tutors/Pre- and post-tests.

Plan Before Collecting Data n Train data collectors with the instruments they will be using or giving out. n Develop procedure for accurately recording the results. n Setting the scene: –Signage –Collection boxes –Return envelopes

Uses of Outcome Findings n Internal –Provide direction for staff –Identify training needs –Improve programs –Support planning –Guide budgets and justify resource allocations –Suggest outcome targets

Uses of Outcome Findings n External –Recruit talented staff and volunteers –Promote the program to potential participants and referral sources –Identify partners for collaboration –Enhance the programs public image –Retain and increase funding

The most compelling argument you make will occur when you convey the passion you have for your project!