Performance and Progress 2008/2009

Introduction Data collected during 2008/2009 fiscal year. Who did our programs serve? Did programs reach the intended populations? Did programs meet service goals? Did children and families meet outcome goals? Lessons learned.

Service Data Children’s Investment Fund programs served 13,165 children and 2,279 parents/caregivers during the last fiscal year. Programs exceeded service goals by 3% more children than projected. Early Childhood percentage is larger in part because funding in that program area includes two substantial programs that reach several preschool and childcare sites. We saw more significant variation at the individual program level in meeting service goals. Most programs met or exceeded goals.

Who Did We Serve? Gender and Age Group Programs served roughly the same number of boys and girls. School aged children (age 9-15), followed by children age 3 – 5, make up the largest proportions of children served. Concentration of services in the school aged (9-15) population is due to a requirement that after-school programs serve 4th through 8th graders. Large portion of ages 3 – 5 served due to Raising a Reader program serving close to 3,000 annually, the majority of whom are in preschool.

Who Did We Serve? Race/Ethnicity Over the majority of children served are children of color (over 65% of children served). Latino children are the largest population among children of color served (22%) followed by African American children (20%). Early childhood and child abuse programs served twice as many Native American children as after-school and mentoring programs. Early childhood and child abuse programs served 10% more Latino children as compared to after-school and mentoring programs. The differences among investments areas can be attributed to the differences in culturally specific programs funded in each of the investment areas.

Who Did We Serve? Primary Language Over 25% of children do not speak English as a first language. Across all program areas, over 1100 children speak a first language other than English or Spanish. From data grantee reported, at least 26 other languages are spoken, including: Bosnian, Russian, Vietnamese, Mandarin Chinese, Cantonese Chinese, Hmong, Chukese, Indonesian, Tagalog-Filipino, Somali, Mai Mai, Arabic, Amharic-Ethiopian, Liberian dialects, Nigerian dialects, Congolese dialects, Kurdish, Urdu, Persian, Pashto, Eritrean/Tigrinya dialects, Burmese, Japanese, Hindi, German, and Thai.

Who Did We Serve? Poverty Level Income data are collected differently by Early Childhood (EC) and Child Abuse Prevention & Intervention (CA) grantees than by After School and Mentoring (ASM) grantees. EC/CA grantees collect income data directly from clients; ASM grantees receive Free & Reduced Lunch qualification data about groups of children served. Altogether, over 50% of children served are living in households with incomes that range from the federal poverty level (FPL) and up through 185% of FPL. The FPL for the relevant year was $22,050 for a family of 4; 185% of FPL was $40,793 for a family of 4. As a point of reference, the median income for a family of 4 in Portland is $70,000. This means the majority of children served in our programs come from families whose annual income is, at best, just over half of the median income. The clients our programs serve primarily reside in SE, NE and North Portland which are the highest poverty areas in the city. Our programs are reaching: predominantly children of color; families who are living in the sectors of the city with the highest poverty rates; and a high proportion of children living at or near the federal poverty level. Data showing that the majority of children served are low income lets us know that the services are indeed reaching our most vulnerable children.

Outcome Data Outcome Goal Areas: Early Childhood and Child Abuse Child development Child health Parenting/family functioning Child stability and welfare Outcome Goal Areas: After-School and Mentoring School attendance School behavior Academic achievement Background on Outcome Goal Development: two approaches based on investment areas. We worked with grantees and PSU consultants at the Child Welfare Partnership and the Hatfield School of Government to establish a menu of outcomes so grantees could select outcome most applicable the types of data collection and reporting related to their specific program models. For After School and Mentoring programs, outcome goals were chosen through a series of community meetings during creation of the Request For Investment process (RFI). After the first year of program funding, CHIF staff negotiated with school districts (PPS, David Douglas, Centennial, Reynolds) to generate aggregate data for CHIF, and individual program data for grantees for the students participating in CHIF funded after-school and mentoring programs.

Outcome Data Limitations The data we are reporting are descriptive, not causative. Many data points provide information on progress made while children are enrolled. Percentages reported apply only to the portion of programs tracking the outcome and those clients who met a participation threshold. 4,965 of the children served (38%) met participation thresholds set by grantees for outcome tracking. This is an Important section of this report! Our data are descriptive about what happened with children in our programs during the time in which they were served, and, in some cases, that data is compared with the same data on the same children for the prior year. Our data neither show that our programs caused these results nor do they say that our programs did not cause these results. Our data mainly help us understand what happened with children in our programs. Our data are based on only the programs that collected and reported data for outcomes relevant to their program models. Not all programs collected data on each outcome. Percentages in following slides are not based on all 13,000 children served. The number of children for whom the percentage applies is listed with each outcome statement. In all cases, the percentages listed only apply to the children who met the participation threshold set by the grantee. The percentage of children served who met the participation floor listed above is somewhat understated because the school districts are not always able to correctly identify and report on all children listed as “served” by grantees. Some programs were not included in the calculation of the percentage meeting participation thresholds so the percentage of children served that met participation thresholds is an awkward figure. We omitted programs that served large numbers of children with drop-in or outreach services, a small portion of which received more intensive services. In these situations, grantees only measured outcomes for the portion of the population receiving the more intensive services.

Early Childhood and Child Abuse Program Data Grantees needed technical assistance in data collection and reporting. Gathering Data: Technical Assistance Project with Portland State University. After TA, grantees more effective in data collection and reporting Data collection and reporting still time consuming and costly for grantees. We found (through the EC study done for us by PSU in 2003 - 2005 and by our own site visits with grantees) that grantees needed help creating and strengthening systems for collecting, aggregating, reporting, and analyzing outcome data about the clients they served. During the first two years of program funding for early childhood and child abuse prevention programs, PSU gathered and analyzed data on a portion of programs seeking to assure children meet developmental milestones. Grantees then needed to transition to gathering, analyzing and reporting this data to us directly. CHIF engaged PSU to provide the technical assistance necessary for grantees to do this. TA provided by PSU included: 1) Revising and solidifying data collection and reporting plans with grantees; 2) Helping them select tools to collect data need to report on outcomes; 3) Creating and modifying specialized annual outcome report forms for each program, specific to their outcome goals and data; 4) Arranging extensive computer training; 5) Coordinating and hosting a two-day training to have 8 home-visiting programs use new parent-child interaction assessment tool. Technical Assistance contract with PSU ended April 2007. Grantees had a total of 18 months of technical assistance to clarify and systematize methods of data collection, reporting and analysis. TA helped significantly but additional TA in data collection, reporting and overall program evaluation is still needed by grantees in order to produce meaningful reports to CHIF. Organizations often lack the resources, such as staff time, staff expertise, and technology, to systematically collect, manage, analyze and report data. TA from CHIF has helped, but funding to help programs build their own capacity in those areas would be more sustainable in the long run.

Child Development 79% of children screened met developmental milestones. 21% who were not on track made progress on meeting milestones while enrolled. 92% of these children were referred to additional services. Children screened showed the most risk in language/ communication development. 878 children were screened at least twice per year for their progress in meeting milestones in child development. Over 1,600 developmental screenings were performed. 79% of children screened were on track in development of gross motor, fine motor, communication, problem-solving, and social/emotional skills. 21% were making progress toward milestones; 92% of those children were referred to additional services for development (such as Multnomah Early Childhood Program). Social Emotional Development and Communication are the domain in which children showed the most risk. 80/20 split has remained constant since PSU did its initial 18-month study. This trend is likely to continue because our programs are meant to identify children who might need early intervention for development in order to help them arrive in kindergarten ready to learn. These results are based on data from 13 grantees.

Child Health 95% of children eligible for health screenings were screened for health and wellness needs. 92% of children screened for immunizations were up to date. 95% of children (n=667/705) were screened for physical health and wellness needs; types of screenings include, height, weight, vision, hearing, dental, immediate medical needs, nutrition, and up-to-date immunizations. 29 children had needs that were identified by screenings and were referred to additional services. 92% of children (n=421/456) screened for immunizations are up to date with immunizations. Screening results are based on data from 10 grantees; immunization results are based on data from 8 grantees.

Parenting and Family Functioning 88% of parents increased social supports. 82% of parents increased appropriate parenting practices. 75% of parents increased knowledge of ways to manage child behavior. 89% of parents increased knowledge of child development. 91% of parents increased and demonstrated appropriate parent-child interactions. Over 780 parents participated in parenting classes or home visiting services. For parenting classes, parents attended approximately 75% or more of sessions, which lasted up to 12 weeks. For home visiting services, parents completed over 6 months of service. (Numbers are based on 15 grantees.) 88% of parents increased social supports (n= 275/313 parents, 8 grantees) 82% of parents increased appropriate parenting practices. (n= 210/255 parents, 5 grantees) 75% of parents increased knowledge of ways to manage child behavior (n= 182/244 parents, 5 grantees) 89% of parents increased knowledge of child development (111/125 parents, 3 grantees) 75% of parents demonstrated appropriate parent-child interactions (347/461 parents, 10 grantees)

Child Stability and Welfare 99% of children attending child care centers or preschools with access to mental health counselors were not removed from care due to behavioral problems. 91% of families who were referred to the child abuse hotline for suspected abuse or neglect were not re-reported within 180 days of completing services. 85% of vulnerable children experienced an increase in stability. 99% of children served at 9 child care centers and preschools with access to early childhood mental health consultation were not expelled or suspended from their preschool settings (711/714, 1 grantee serving 5 grantees plus other preschool and childcare providers in Portland; expulsion data not reported to grantee from 2 sites serving approximately 250 children). 91% of families were not RE-reported to the Multnomah County Child Abuse Hotline within 180 days of completing services (60/66 families, 1 grantee) 81% of parents with mental health issues, predominantly depression and anxiety, demonstrated a decrease in mental health issues. (30/37 parents, 2 grantees) 79% of children and youth with mental health issues demonstrated a decrease in mental health issues or an increase in emotional well-being (87/106 children and youth, 4 grantees) 91% of vulnerable children and youth struggling with academic challenges (which includes behavior, attendance, attitude and performance) experienced increases in academic success (298/327 children and youth, 3 grantees) 85% of vulnerable children and youth experienced an increase in stability in their lives, including reduction of risk of family violence, improvement in overall family functioning, and improvement in school stability (155/183 children and youth, 5 grantees)

After-School and Mentoring Program Data 79% of identified program participants attended Portland Public Schools. PPS, David Douglas, Reynolds and Centennial School Districts all provided data on academic achievement variables. PPS also provided data on attendance, behavior variables, and progress on grades.

School Attendance (PPS Only) 47% of program participants improved school attendance in the 2008/2009 school year. 69% of program participants attended 90% of school days. Improvements in attendance were measured for children for whom 2 years of data were available and who attended fewer than 97% of school days during the 2007/2008 school year. 47% improved attendance: (n=745/1602). 69% CHIF program participants who attended 90% of school days: (n =2039/2939).

School Behavior (PPS Only) 68% of program participants decreased serious behavior referrals (those that resulted in a suspension or expulsion) Tracking discipline data electronically is new at PPS which raises some reliability problems. No consistent standards for making behavior referrals between schools and districts so a child may receive a behavior referral at one school for a reason that would not trigger a similar referral at a different school. Data on behavior referral decreases only refers to students for whom two years of data was available, and who received at least one referral for suspension or expulsion during the 2007/2008 school year. 55% of program participants decreased behavior referrals (n = 319/577). 68% of program participants decreased serious behavior referrals: (n =223/327).

Academic Achievement: Grades (PPS Only) 24% improved reading/English grades 23% improved math grades All PPS elementary school use standards based report cards and do not give letter grades. Most PPS middle schools and all PPS high schools use traditional letter grading systems. No grade data could be collected for children moving between standards based and traditional grading systems thus many program participants were not included in this analysis. Only students for whom two years of data was available were included in this analysis.

Academic Achievement: Percentage of Students Meeting State Standards in Reading and Math (All Districts Reporting) Subject 2007/08 2008/09 Reading/ English 70% 68% Math 65% Achievement data was only reported for program participants who had valid scores on tests for two years.

Academic Achievement: Percentage of Students Moving to a Higher Performance Category (All Districts Reporting) 17% of program participants moved to a higher performance category in reading. 21% of program participants moved to a higher performance category in math. Performance levels were as follows: very low low nearly meets meets Exceeds

Academic Achievement: Progress toward Meeting State Standards (PPS Only) Of the students who were not meeting state standards in reading, 44% moved to a higher performance category. Of the students who were not meeting state standards in math, 49% moved to a higher performance category. This result is the most encouraging because we can see that a significant percentage of kids who most need to make progress are making it.

Outcome Data Analysis Technical assistance is necessary when grantees must gather, analyze and report data. Programs need money, time, and expertise dedicated to program reporting and evaluation. Need for strategies to increase level of participation in programs.

Considerations for the Future Need for common intermediate outcomes for after-school and mentoring programs. CHIF would need to provide technical assistance to help after-school and mentoring programs to collect, analyze and report data on intermediate outcomes. CHIF should provide ongoing technical assistance to early childhood and child abuse prevention programs for producing high quality program reports. SUN system task force is considering joint evaluation strategies. We are only tracking very high level outcomes for after-school and mentoring programs. While the information is valuable and we want to keep our eye on how kids are performing on these variables, it is difficult to say to what degree any given program affects these outcomes. It would be beneficial to develop a list of intermediate outcomes that programs could choose from to track the variables they feel they are most likely to affect. Technical assistance provided by PSU, under contract from CHIF, for early childhood programs and child abuse prevention programs assured that grantees produced higher quality and meaningful program reports. CHIF staff have continued to provide technical assistance to grantees after the PSU contract ended, but it has turned out to be more time-consuming that staff anticipated. To best support grantees in producing the type of program outcome reports CHIF desires, CHIF should continue to pay for technical assistance services (beyond what staff can currently provide).