Using Family Survey Data for Program Improvement ECO Meeting June, 2009 Using Family Survey Data for Program Improvement Pam Roush, Director WV Birth to Three October 7, 2009

Gathering Family Survey Data ECO Meeting June, 2009 Gathering Family Survey Data WV Birth to Three uses both scales of the NCSEAM surveys (IFS and FCSS) ICC assisted LA to select survey tool that would respond to the OSEP indicators, and provide additional information for program improvement Chose NCSEAM surveys at the time due to proven reliability and validity WV has been using the NCSEAM surveys for three years. We worked with our ICC to select a survey tool that would give us reliable and valid data for the OSEP indicators. We also wanted a tool that would give us more information and help to guide our improvement strategies around practices that will make a difference for children and families – including but beyond the three outcome indicators. Prior to this, we used simple yes/no surveys, with some open ended questions – this wasn’t sufficient. WV also uses the recommended RASCH analysis – in the 2007 APR summary, there were 17 states using the NCSEAM survey, with the full RASCH analysis.

Gathering Family Survey Data ECO Meeting June, 2009 Gathering Family Survey Data Surveys are mailed to all exiting families whose children have received services for 6 months or longer Cover letter is included explaining the benefits of families sharing this information Include stamped, addressed return envelope 30% response rate Analyze demographics of respondents to assure representativeness Surveys are coded to allow matching against our integrated child data base. This gives us the opportunity to correlate responses with many other demographic factors. We did revise our cover letter last year, to emphasis how families are helping other families by completing the survey. Completed surveys are returned to the state office and data entered by a separate Division in Office of Maternal, Child and Family Health

ECO Meeting June, 2009 Interpreting Results The Rasch analysis provides frequency and distribution data Look at frequencies and distribution to see if there are outliers Does data look different with outliers removed? Can look for other characteristics that distinguish families with lower measures from those with higher measures

Order of agreeability of IFS items Calibration Stem : Over the past year, Early Intervention services have helped me and/or my family: 678 Participate in typical activities for children and families in my community. 656 Know about servi ces in the community. 640 Know where to go for support to meet my family's needs. 625 Keep up friendships for my child and family. 609 Know where to go for support to meet my child's needs. 584 Be more effective in managing my child's behavior. 5 76 Make changes in family routines that will benefit my child with special needs. 576 Do activities that are good for my child even in times of stress 70 Improve my family's quality of life. 565 Feel that I can get the services and supports that my child and family need. 563 Get the services that my child and family need. 562 Feel that my family will be accepted and welcomed in the community 559 Feel more confident in my skills as a parent. Feel that my child will be accepted and welcome d in the community. 556 Communicate more effectively with the people who work with my child and family. 553 Understand how the Early Intervention system works. 546 Understand the roles of the people who work with my child and family. 539 Know about my child's and family's rights concerning Early Intervention services. 534 Be able to evaluate how much progress my child is making. 516 Understand my child's special needs. 498 Feel that my efforts are helping my child. Do things with and for m y child that are good for my child's development. Each survey item on the scale has a calibration that represents the level of difficulty in getting agreement from families. The items that respond to the OSEP Family Outcome Indicators are toward the bottom of the scale - # 539, 516, 556

ECO Meeting June, 2009 Interpreting Results Survey items at or just above the mean may require new or different practices Nearly all families are in agreement with items below the mean, so these items are where current practices can be extended or enhanced Strong correlation between items on IFS and FCSS (.93 in WV) A mean score is developed for each family, and then for all respondents. So in addition to having performance data for the 3 OSEP indicators, we also have frequency data for each item and a mean score for each scale. Since there is such strong correlation between the Impact on Families Scale and the Family Centered Services Scale, we also know that improvements in practice measurements on the FCSS scale should result in improvement on the IFS.

West Virginia’s Analysis ECO Meeting June, 2009 West Virginia’s Analysis Overall family responses are very positive Mean measure of 720.08 on IFS Needed to decide where to focus improvement strategies Highest calibrated item on survey has calibration of 678 – putting our mean score above this item. Which meant that in general, most families were in agreement with survey items. We needed to decide where to focus our improvement strategies in order to continue to increase the level of agreement. We know the survey reliability tells us that focusing on the items just below the mean should give us improvement on those items, as well as all items below

Identifying Improvement Strategies ECO Meeting June, 2009 Identifying Improvement Strategies November, 2008 – WV attended regional forum on identifying improvement strategies for APR indicators WV Birth to Three chose to focus on Family Outcome Indicator Used results of both survey scales Items on FCSS are practice items – should inform us of the practices that would result in improved measures on IFS Technical assistance from Mid South, NECTAC and other national TA

West Virginia’s Analysis The top two (most difficult) items on the Impact on Families Scale are: BTT services have helped me and/or my family: Participate in typical activities for children and families in my community Know about services in the community

West Virginia’s Analysis Mean score on the FCSS was 647 Item just below the mean is: My family was given information about organizations that offer support for parents of children with disabilities Item just above the mean is: My family was given information about how to participate in different programs and services in the community

West Virginia Analysis ECO Meeting June, 2009 West Virginia Analysis And true to the scale’s reliability, a lower level of agreement was reached on the higher calibrated practice items Someone from WV Birth to Three helped me get in touch with other parents for help and support (calibration of 734) Changes in measurements for practice items on the FCSS should result in changes on measurements for the IFS

Identifying Practices/Activities for Improvement ECO Meeting June, 2009 Identifying Practices/Activities for Improvement Task group of the ICC is working with LA to develop a ‘Who You ‘Gonna Call’ visual/written resource to get information to families Ways to connect with other parents Organizations that support families of children with disabilities Who to call when things are not going right How to find child care We had some earlier success with a visual depiction of what happens when a referral is made to BTT – stepping families through the process. ISCs and community partners found this helpful to explain process and prepare families for their participation

Identifying Practices/Activities for Improvement Recently revised the statewide/standard IFSP form and instructions Considered the measurements from the two scales Built in more emphasis on importance of involving other community partners in the IFSP process and linking families to other non-Part C needed resources Added roles/responsibilities of SCs, including advocacy, etc

Evaluating Effectiveness of Improvement Strategies ECO Meeting June, 2009 Evaluating Effectiveness of Improvement Strategies WVBTT will use future results on the FCSS and IFS to measure improvement after implementation of new practice strategies