1. 1

2. 2 Design and Funding Forum: Emerging Models for Innovation September 5, 2008 An RBS Evaluation Framework

3. 3 Goals for Today  To review the values and principles underlying the evaluation framework  To introduce the individual outcomes component  To explain how the evaluation framework will fit into the RBS program design  To get your ideas about implementing the evaluation framework

4. 4 Values and Principles Underlying the Proposed Evaluation Framework The evaluation should:  Yield the information needed to meet the evaluation mandates of AB1453  “Follow and not drive” the RBS program design  Be as unobtrusive as possible  Be affordable  Have minimal administrative and fiscal burden on participating providers and counties  Provide some of the clinical and management information needed by the local RBS program stakeholders to monitor local program operations  Be acceptable to all RBS stakeholders (children and families, providers, counties, CDSS, and members of the RBS Reform Coalition)

5. 5 The Evaluation Mandates of AB1453 Excerpted from the Legislation (emphasis added): “Voluntary agreements by counties and nonprofit agencies shall … (3) Provide for an annual evaluation report, to be prepared jointly by the county and the private nonprofit agency. The evaluation report shall include analyses of the outcomes for children and youth, including achievement of permanency, average lengths of stay, and rates of entry and reentry into group care. The evaluation report shall also include analyses of the involvement of children or youth and their families, client satisfaction, the use of the program by the county, the operation of the program by the private nonprofit agency, payments made to the private nonprofit agency by the county, actual costs incurred by the nonprofit agency for the operation of the program, and the impact of the program on state and county AFDC-FC program costs. The county shall send a copy of each annual evaluation report to the director, and the director shall make these reports available to the Legislature upon request.”

6. 6 AB1453 Evaluation Mandates Table 8.The impact of the operation of the program on state and county AFDC-FC costs 7.Actual costs incurred by the private non-profit agency for the operation of the program 6.Payments made to the private non-profit agency by the county Fiscal Outcomes 5.The operation of the program by the non-profit agency 4.The use of the program by the county Systems Operation s 3.Client satisfaction 2.Analyses of the involvement of children or youth and their families 1c.Rates of entry and reentry into group care 1b.Average lengths of stay 1a.Achievement of permanency 1.Analyses of the outcomes for children and youth including Client Outcomes

7. 7 AB1453 Evaluation Mandates Table 8.The impact of the operation of the program on state and county AFDC-FC costs 7.Actual costs incurred by the private non-profit agency for the operation of the program 6.Payments made to the private non-profit agency by the county Fiscal Outcomes 5.The operation of the program by the non-profit agency 4.The use of the program by the county Systems Operation s 3.Client satisfaction 2.Analyses of the involvement of children or youth and their families 1c.Rates of entry and reentry into group care 1b.Average lengths of stay 1a.Achievement of permanency 1.Analyses of the outcomes for children and youth including Client Outcomes The evaluation framework for these elements will be developed based on the RBS program designs proposed in the Voluntary Agreements due in mid-October

8. 8 Client Outcomes AB1453 Evaluation Mandates Table 1c.Rates of entry and reentry into group care 1b.Average lengths of stay 1a.Achievement of permanency 1.Analyses of the outcomes for children and youth including [but not limited to] 3.Client satisfaction 2.Analyses of the involvement of children or youth and their families 1a.Achievement of permanency

9. 9 Client Outcomes AB1453 Evaluation Mandates Table 1c.Rates of entry and reentry into group care 1b.Average lengths of stay 1.Analyses of the outcomes for children and youth including [but not limited to] 3.Client satisfaction 2.Analyses of the involvement of children or youth and their families RBS evaluation Challenge - meet the requirements of:  AB1453 mandated evaluation client outcomes (1, 1a-c., 2, and 3 above) and  client outcomes identified by the local RBS teams during the July Technical Assistance Workshops (1d and 1e above) while  remaining true to the seven values and principles presented earlier 1a.Achievement of permanency County Identified Outcomes 1d.Well Being (e.g. safety, stability, behavioral improvement, sense of hope, others) 1e.Educational progress and progress toward permanency

10. 10 Meeting the Challenge: The Proposed Framework  Three pre-existing data collection instruments  County Level CMS/CWS Data currently collected quarterlyCMS/CWS Data currently collected  The Youth Services Survey (YSS) andYouth Services Survey (YSS)  The Youth Services Survey for Families (YSS-F) promulgated by the California Department of Mental Health (DMH)Youth Services Survey for Families (YSS-F)  An assessment instrument with a national track record  The Child and Adolescent Needs and Strengths (CANS) assessment instrumentChild and Adolescent Needs and Strengths (CANS)instrument

11. 11 Maximum Data Collection Model Minimum Data Collection Model Treatment Planning / Utilization Review YSS/YSS-F CANS CWS/CMS YSS/YSS-F CANS CWS/CMS RBS Treatment Program Year TwoYear One Meeting the Challenge: How This Might Work Intake Exit 90 days 180 days 270 days 365 days 90 days 180 days 270 days Residential TreatmentCommunity Based Care                        2 data points 9 data points 2 data points Children & Families

12. 12 What Degree of Data Collection is Manageable? To be determined:  The frequency of data collection needed by RBS stakeholders to understand what is happening with children and families during RBS enrollment  Costs associated with and funding available to support:  Staff and client time spent in instrument administration  Data collection and compilation  Data reporting

13. 13 RBS Evaluation Framework Design Values and Principles CANS YSS & YSS-F Existing CWS/CMS Measures 4.Is it affordable? Meeting the Challenge: Why These Instruments? 5.Does its use minimize the administrative and fiscal burden on participating providers and counties? 3.Can it be or is it implemented “unobtrusively”? 2.Does it follow - and not drive - the RBS program design? 1.Will it yield the information needed to meet the legislative mandates of AB1453 - the legislation authorizing the RBS initiative? Yes 6.Will it provide some of the clinical and management information needed by the local RBS program stakeholders to monitor their program operations? Yes

14. 14 How the Data Collection Could be Operationalized 1.County Level CWS/CMS Data currently collected quarterly  Work with participating RBS counties and The Center for Social Services Research (CSSR) to establish standard procedures for identifying or “flagging” children enrolled in RBS in CWS/CMS  Generate automated reports on RBS children on current CWS/CMS measures relevant to RBS including: RBS Stakeholders’ Roles in this Process  Continue to report CWS/CMS data accurately and on time  Flag the records of children enrolled in RBS according to the standard procedures we establish ◆ Permanency ◆ Lengths of Stay ◆ Rates of entry and reentry into group care ◆ Safety ◆ Placement stability ◆ Restrictiveness, and ◆ Educational achievement

15. 15 2.The Youth Services Survey (YSS) and the Youth Services Survey for Families (YSS-F)  Assess the current use of the instruments and how data are collected and compiled  Conduct an item level review of the instruments to determine which items are most relevant to RBS  If appropriate, create abbreviated versions of the instruments RBS Stakeholder’s Role in this Process  Provide feedback to above results  Imbed client satisfaction instruments in the ongoing periodic treatment planning and/or utilization review processes  Collect, compile and submit client satisfaction data to the RBS Consortium How the Data Collection Could be Operationalized

16. 16 3.Child and Adolescent Needs and Strengths Assessment Instrument (CANS)  Assess the current use of the CANS and how data are collected and compiled  Conduct an item level review of the CANS to determine which items are most relevant to RBS  If appropriate, create an abbreviated version of the CANS RBS Stakeholder’s Role in this Process  Provide feedback to above results  Imbed the CANS in the ongoing periodic treatment planning and/or utilization review processes  Collect compile and submit CANS data to the RBS Consortium How the Data Collection Could be Operationalized

17. 17 RBS Evaluation Next Steps Proposed Timeline Client Outcomes Measures SeptemberOctoberNovemberDecember 1. Contact RBS county decision makers to assess the acceptability of the proposed evaluation framework 3.Work with CSSR and RBS counties to establish procedures for identifying RBS clients in CWS/CMS 2.Assess the current use of the YSS, YSS-F and CANS in the RBS counties 4.Report on findings from 1-3 to the RBS Consortium and Steering Committee 6.Create modified instruments as per 1-4 above 7. Submit modified instruments for review and comment to RBS Consortium and Steering Committee and RBS county decision makers 8.Finalize instruments based on 6 and 7 above 1-1516-30    1-1516-31 5.Convene RBS Evaluation Subcommittee 1-1516-311-1516-30   

18. 18 Voluntary Agreements Submitted Systems Operations Measures Fiscal Outcomes Measures SeptemberOctoberNovemberDecember 1-1516-31 1. Determine proposed Systems Operations Measures based on October Evaluation Subcommittee meeting and Voluntary Agreement 2. Propose Systems Operations Evaluation Framework at November RBS Forum 1. Determine proposed Fiscal Outcomes Measures based on October Evaluation Subcommittee meeting and Voluntary Agreement 2. Propose Systems Operations Evaluation Framework at November RBS Forum 1-1516-301-1516-31   1-1516-30 RBS Evaluation Next Steps Proposed Timeline

19. 19 Your Feedback 1.Your name 2.The RBS location you’re from 3.Whether you’re representing a county or a provider  Please sharing your thoughts on what you’ve just seen and heard by answering the following questions: 1.What, in your opinion, are the strengths of the proposed RBS evaluation design? 2.What are your concerns about the evaluation design? 3.What additional questions do you have?  Please introduce your comments by telling the group: Thank you for participating! Your feedback will help the RBS Evaluation Team and Steering Committee as we develop a practical and affordable evaluation approach

20. 20

21. 21 Themes from Demonstration Site Presentations California RBS Reform Forum Sacramento, CA September 5, 2008

22. 22 Convergence of Technologies Resource Utilization Management Comprehensive Care Coordination Practice Based on Evidence

23. 23  Maximize revenue to create best array of services possible for each child and family  Arc of enrollment  Family voice and choice  Vehicles for continuity  Pass-off the funds, rather than pass-off the service relationship Comprehensive care coordination themes

24. 24  Connections for youth/family/community  Use of standard assessment measures  Enhanced referral process & matching  Flex worker roles  Increase worker time for team participation  Open up the “black box” of facility-based care  Provider involvement in planning  selected providers informing operational details Evidence based practice themes

25. 25  Capture cost savings with initial innovations  Utilize flexible pool of AFDC-FC funds  Creative tension between “packaged” funding & “blended” funding  RCL Alternative  Treatment incentives and outcome incentives  Creating a better niche to fill the space currently occupied by group care Resource utilization themes

26. 26

27. 27 Proposed CDSS Review Process What would you attempt to do if you knew you could not fail? -Unknown

28. 28 Deliverables Submitted to CDSS RBS Analyst will coordinate internal routing and reviews. Counties identify contact(s). Electronic MS Word format. 3 Week Review Internal CDSS areas reviewing deliverables will work with county contact(s) to resolve minor issues. If needed, schedule meeting or conference call with County to clarify and resolve issues. Meet with County during 4th week (optional) Tentative Approval County finalizes deliverables and resubmits. Final CDSS Approval 10 Day Review of Resubmitted Deliverables

29. 29

30. 30 DateKey Milestone Sept 2008Technical assistance support to demonstration sites Sept 9-10 Sacramento Sept 15-16 Bay Area Oct 2008Demonstration sites submit RBS reform deliverables to CDSS for review Oct 3 Sacramento Oct 17 Los Angeles, San Bernardino, Bay Area Nov – Dec 08CDSS Review Process Nov 2008Technical assistance support to demonstration sites Nov 5-6, 2008RBS Implementation Forum Jan - Feb 09CDSS authorizes regulatory & fiscal waivers needed for approved RBS Reform Plans and secures agreements Feb – Mar 09RBS reforms implemented in demonstration sites Timeline for 2008

31. 31  All proceedings from Forum will be posted on website www.RBSreform.orgwww.RBSreform.org  Specific technical assistance topics to address in Sept & Oct?  Confirm need for, timing and format for next LIC gathering?  Questions? Feedback & next steps

32. 32 Karen Gunderson, MSW Chief, Child and Youth Permanency Branch California Department of Social Services Email: karen.gunderson@dss.ca.govkaren.gunderson@dss.ca.gov Phone: (916) 651-7464 Fax: (916) 651-0673

33. 33