CECC Webinar Series: Measuring Program Outcomes Presented by Terry Tolan and John Roden

Purpose To introduce you to the concept and a system for measuring your CECC’s outcomes.

What is Outcome Measurement? Outcome Measurement is the regular and systematic measuring of progress toward intended outcomes in order to: Increase the effectiveness of programs and services Communicate the value of those programs and services

Our Business Model Children Enter Kindergarten Ready High Quality Early Learning Environments Supportive Families Access to Data Participation in STARS A great early childhood workforce Families understand child health and developmental needs Common Kindergarten Entry Screener Scholarships & PD Plans Families are engaged Children have access to appropriate services Data is shared by early childhood programs

What CECCs Measure

What we useWhat we doWhat we count Program Outcome Model

What we useWhat we doWhat we count How THEY change! Program Outcome Model

SITUATION INPUTS ACTIVITIES OUTPUTS OUTCOMES INTIAL INTERMEDIATE LONG-TERM EXTERNAL INFLUENCES, ENVIRONMENTAL, RELATED PROGRAMS Outcomes Logic Model – A System of Measurement

Parenting Education Program 1.Parents from 10 families attend the workshops 2.Six group workshops are conducted 3.Parents’ understanding of children’s developmental issues increases 4.Parent provide more age appropriate guidance to children

Input Outcomes ActivityOutput Initial Intermediate Long-Term Inputs Through Outcomes: The Conceptual Chain

Children Enter School Ready Logic Model Framework

INPUTS ACTIVITIES OUTCOMES Initial Intermediate Long-term

Figuring Out Our Outcomes What do we want to be true of participants because of their involvement in the program? What do we want to be able to say about them? If we succeed with a participant (or don’t), what has changed (or hasn’t)? If we conduct the activity, then what do participants believe, know, have, or do as a result? And what benefit or change follows that?

Program Outcome Criteria for Each Outcome Is it reasonable to think the program can influence the outcome in a non-trivial way even though it can’t control it? Would measurement of the outcome help identify program successes and pinpoint problems? Will the programs various “publics” accept this as a valid outcome of the program?

SITUATION INPUTS ACTIVITIES OUTPUTS OUTCOMES INTIAL INTERMEDIATE LONG-TERM EXTERNAL INFLUENCES, ENVIRONMENTAL, RELATED PROGRAMS Outcomes Logic Model

Outcomes vs. Indicators Outcome: Benefits for participants Teens follow proper nutrition and health guidelines Indicator: The specific information that is tracked to indicate success in achieving the outcome Proper weight Does not smoke Takes a prenatal vitamin

Decision Tree

Sample Indicators ActivityOutcomeIndicator Improving ECE Programs through coaching ECE programs are high quality ECERS score of 5 or more

Data Gathering

Only 25% of Children Enter School Ready Planning Facilities Books Printing Costs Literacy Training Book Distribution Demonstration Number of Books Distributed Number of Parents Attending Training Number of Brochures Mailed OUTCOMES Gained Knowledge of Importance of Reading Awareness of Role/Impact Intentionality of Modeling Increase number of times child read to daily/weekly Increase number of parents reading Children Enter School Ready EXTERNAL INFLUENCES, ENVIRONMENTAL, RELATED PROGRAMS *Linking Parents to GED, Adult Education, and ELL Programming* Outcomes Logic Model: Parents Reading Daily to Children

Gathering Data

This checklist can help you decide which data collection methods are most appropriate for your outcome measurement. SURVEYS 1. Do I need data from the perspective of the participant, client, beneficiary, or customer? 2. Do I have a systematic way to get it from these individuals? 3. Do I need data that are standardized so that statistical comparisons can be made? (For example, will I need to report percents or other statistics?) 4. Will participants be able to understand the survey questions? (Consider age, cultural backgrounds, etc.) 5. Do participants have the necessary knowledge or awareness to accurately answer questions about the outcomes? Check List for Selecting Data Collection Methods

Don't ask a question if the answer is obvious. Avoid abbreviations and jargon. If they must be used, clearly define them. Ask yourself whether several questions are actually necessary or if you can get the information in one question. Don't try to cram too much into one question. Make your questions easy to understand. Make sure your sample population understands them. If a list of answers is provided, make sure all possible answers are present. Even with "yes" and "no" questions, it may be necessary to include a neutral "undecided" or "don't know." Don't mix "I feel" or "I think" questions with questions regarding facts. Keep factual and perception questions in separate groupings. Place sensitive demographic questions (such as age or income) at the end of the survey. General Guidelines for Survey Questions

Sometimes it takes just one word to bias a question. Avoid using inflammatory words in surveys, such as: allege, allude, arbitrary, blame, claim, demand, error, failure, fault, ignore, ill- advised, ill-informed, incompetence, ineptness, insist, just, maintain, misinformed, must, neglected, one-sided, only, overreact, peremptory, purport, questionable, rejection, rigid, so-called, unfortunately, unilateral, unreasonable Value-laden questions, especially those that attempt to be global in scope, tend to overwhelm respondents. For example, making respondents choose between a healthy environment and a vital economy will probably bias results. Don't distill complex issues into "black" or "white" scenarios. Rather, explore the "gray" areas. Avoid “Red Flag” Words

Is the question relevant? Is it consistent with survey goals? Does the question ask for "need to know" or "nice to know" information? What will be the value of a response? If 95 percent say, "Yes," would this affect decision making? Will respondents be able to answer the question? Will they have the information? Does the question lead to a particular response? (Is it a leading question?) If a set of answers is provided, are all possible answers listed? Is one side of the issue represented more than another? Does the question use negative phrases or words? Are positive adjectives or phrases used? If a scale is used for responses, is it balanced (for example, 1 to 5, with 3 being neutral)? Are "dead giveaway" words used, such as "all," "every," or "always"? Is the question wordy? Were ambiguous words used - words with more than one meaning? Is the question worded simply? Questions to Ask About Questions

Only 25% of Children Enter School Ready Planning Facilities Books Printing Costs Literacy Training Book Distribution Demonstration Number of Books Distributed Number of Parents Attending Training Number of Brochures Mailed OUTCOMES Gained Knowledge of Importance of Reading Awareness of Role/Impact Intentionality of Modeling Children being read to daily/weekly Parents reading to children routinely Children Enter School Ready Measurement of Process Indicators: Number of Trainings Held/Number of Attendees Number of Books Distributed Number of Demonstrations held/Number of Parents attending Events (Mostly Counting: Does not truly measure Impact) Outcomes Logic Model: Parents Reading Daily to Children Measurement of Outcomes Indicators: Pre-Post Survey of gained knowledge/awareness of parent impact Increase in the number of children being read to daily Increase in the number of parents reading to their children (Measuring Change: Change in Behavior/Practices)

urces/guidebooks/MeasuringOutcomes.pdfhttp:// urces/guidebooks/MeasuringOutcomes.pdf icModel.pdfhttp:// icModel.pdf Resources

Happy Measuring! Terry Tolan John Roden