1. An Overview of Performance Dashboards Presented by: Desiree A. Crevecoeur-MacPhail, Ph.D. Research Psychologist, UCLA ISAP Loretta L. Denering, M.S. Project Director, UCLA ISAP

2. Why Conduct this Training? To review purpose of Los Angeles County Evaluation System (LACES) To review the recently implemented Performance Dashboards To review the purpose of the Dashboard and how to interpret and utilize the information it contains 2

3. What do We Hope You Will Gain By the end of this training, providers will: –Be more aware of what LACES is and does –Understand some terms used in program evaluation –Understand how to read and interpret the Dashboards –Be aware of the benchmarks and how they are being used to assess performance for outpatient counseling programs 3

4. EVALUATING LOS ANGELES COUNTY SUBSTANCE USE DISORDER SERVICES LACES and LACPRS 4

5. Los Angeles County Evaluation System (LACES) Evaluation of adult alcohol and other drug services provided by the Los Angeles County. –Data analysis, reports, articles –Training, presentations and conferences –Development and implementation of surveys & tools Partnership between SAPC, contractors for SAPC services, and UCLA/ISAP. On-going evaluation, not a temporary study.

6. Purpose of LACES Evaluate LA County substance use disorder treatment services Assess treatment outcomes and program performance Disseminate evaluation data to the public Evaluate and explore innovative programs Analyze and report on drug trends See www.LACES-UCLA.org for more infowww.LACES-UCLA.org

7. Los Angeles County Participant Reporting System (LACPRS) LACPRS is the key to evaluating substance use treatment among those using County services All agencies are contractually required to input specific data into the LACPRS database For LACPRS to be effective, agencies MUST input data completely, accurately, and timely for every client! 7

8. LACPRS Admission & Discharge (A/D) LACPRS A/D has 141 questions –Questions 1-32: basic demographic asked once –Questions 33-141: information asked at admission, and again at discharge LACPRS is ONLY for treatment services, not prevention or DUI (adolescent programs have different set of questions) Provides data on those admitted to County funded AOD treatment 8

9. LACPRS Admission and Discharge Data from LACPRS A/D informs the following evaluation documents –Site Reports –Performance Dashboards –Annual Reports 9

10. ASSESS TREATMENT OUTCOMES AND PROGRAM PERFORMANCE SITE REPORTS AND DASHBOARDS 10

11. Important Terms Four important terms: –Client/Treatment Outcomes –Program Performance –Engagement –Retention 11

12. Difference between Outcomes and Performance Client Outcomes –Client Outcomes are the result of what programs do –Can be measured –Examples: Changes in drug use and employment Program Performance –Program Performance refers to areas that are under the control of the program –Can be measured –Examples: Length of stay and perception of care 12

13. Examples of Outcome Measures DomainsMeasures Alcohol/Drug UseAlcohol/drug use during past 30 days Employment/Educati on Employment/education in past 30 days Crime & Criminal Justice Criminal justice system-related activity in past 30 days, in terms of any CJS involvement arrests, jail days, and prison days Stability in HousingStable housing in past 30 days, in terms of homeless Social Connectedness Family/social problems in past 30 days in terms of serious family conflict

14. Examples of Potential Performance Measures DomainsMeasures Continuity of CareClients who had a subsequent admission to another service during 30 days after discharge from a prior service (treatment episode data with unique client ID). AccessSelf-reported wait list time at admission. EngagementStay in treatment at least 30 days and participate in 4 or more sessions. RetentionLength of treatment stay (in days) CompletionClients with a treatment completion (referred/not referred) discharge status.

15. Site Reports Details of Content and Use 15

16. What is a Site Report? Brief: six pages, double-sided. Information is gathered from the admission & discharge LACPRS. Information is included on each agency site and for all sites of the same program type (e.g. outpatient, residential, etc).

17. More on Site Reports Snapshot of program functioning and short-term treatment outcomes. Based on LACPRS admission & discharge data. Are used to provide information ONLY Divided into two sections: –Executive Summary –Full Report 17

18. Purpose of Site Report To provide information concerning key performance/outcome areas. To provide feedback to sites concerning how they are performing with regards to the focus areas of the evaluation. To report how other, similar programs are performing. 18

19. Contents of the Site Report Executive Summary –Admission and discharge totals. –Information on program functioning and treatment outcomes. Full Report –Includes demographic information. –Includes more details on treatment outcomes. –Charts showing admission to discharge changes in various areas covered by the LACPRS.

20. Site Reports and LACPRS 20 Site Report Information is based on LACPRS –Late data or inaccurate data input at admission and discharge will affect results Engagement of clients is critical Conducting Exit Interviews are essential –Compares measures/indicators input at admission with those at discharge –Lack of exit interviews impact accuracy of performance measures and program outcomes

21. Performance Dashboards What are they? How were they developed? How are they used? How do you gain access? 21

22. What is a Dashboard? Similar to a report card An easy to read summary (typically single page) Provides information on provider performance based on the identified measures Dashboard is based solely on LACPRS information input by the agency. 22

23. Difference between Site Reports and Dashboards Site Reports –Multi-page –Info on outcomes and some performance measures –No benchmarks or required level of achievement Dashboards –Single page – in some cases –Info only on performance measures –Includes benchmarks and there is a Required level of achievement 23

24. Similarities between Site Reports and Dashboards Accurate and timely data entry are important. –Inaccurate data will result in Incorrect reports (Site Reports/Dashboards) Delayed payments –Late data will result in Incorrect reports Both reports only available to the executive directors and their designees If you need access, the executive director must contact Richard Lugo 24

25. Terminology Review 25 Performance Measure: An indicator used to assess a provider's delivery of care as it conforms to guidelines or standards of quality. –Focus on program functioning –Performance measures do not directly measure these outcomes.

26. More Terminology Review Performance Benchmark : A level of achievement in reaching the goal for a performance measure that generally represents an industry-best standard. –For SAPC that industry-based standard is the average performance for all providers –Adherence to performance benchmarks is expected to lead to desirable outcomes 26

27. Performance Measure (PM) Development A list of possible PMs were distributed to contracted treatment providers Meetings were held to discuss potential PMs, availability of the data and their utility Once performance measures were settled, data examined to determine benchmark Settled on three initial performance measures - for outpatient counseling (OC) ONLY Day Care Habilitative included with OC 27

28. Performance Measure (PM) Development Performance Measures agreed upon: –30 Day Length of Stay (Engagement) –90 day length of stay (Retention) –Exit Interviews All of these performance measures were familiar to provides since they are included in site reports 28

29. Why Engagement? In AOD treatment, a significant proportion of patients leave treatment during the first four weeks Engagement = first 30 days in treatment –Typically includes at least 4 contacts Low engagement may indicate problems with intake process, counselor rapport development, program process or policies 29

30. Why 30 Days? Most of the questions on the LACPRS that we use to measure outcomes have a 30 day time frame. For example –How many days in the last 30 did you… –This time frame is used at admission and discharge If clients are not engaged for at least 30 days, the time frame of the questions asked at discharge overlap the time period covered at admissions 30

31. Why Retention? Research shows that without an adequate amount of time in treatment, few improvements are found –90 days is the magic number Retention = 90 days or more in treatment Low retention rates may indicate problems with treatment process (e.g., redundant), lack of rapport, inappropriate or ineffective policies for dealing with relapse, etc. 31

32. Interviews Responses to all discharge LACPRS questions. One of the most important of the PMs Information from both admission and discharge is required to measure treatment effectiveness No exit interviews = No measurement of changes that occurred during treatment

33. Impact of Exit Interviews and 30- days LOS Data Program ABC has 20 individuals discharged –15 completed exit interviews –10 of the 15 were in treatment at least 30 days ONLY have valid outcome data for 10 of the 20 discharged individuals –Only those in treatment at least 30 days and who have completed exit interviews will have data that can show changes that occurred during treatment 33

34. Performance Benchmarks Once PM were agreed upon, benchmarks were developed Standard levels of expected performance SAPC benchmarks are based on average from outpatient programs for the prior three years –30 Day Length of Stay = 80% –90 Day Retention = 65% –Exit Interviews = 50% Benchmarks are reported via “Dashboards” 34

35. HOW TO READ DASHBOARDS Three (or four) sections of dashboard –Introduction –Results –Next Steps (for fell below benchmark) –Definitions 35

36. HOW TO READ DASHBOARD Introduction Performance Measure Current Quarter Discharges: 7 Report QTR N Total for Fiscal Year by Quarter (%) Cumulative Performance (%) Performance Benchmarks (%) 1 st 2 nd 3 rd 4 th NYear 1Year 2Year 3 Participants in Treatment at least 30 Days 7100 1010080 Participants in Treatment at least 90 Days 51007188065 Participants with Exit Interviews 6678688050 36 Performance Dashboard Outpatient Counseling Program 1234 Any Street, Los Angeles, CA 90025 This dashboard provides information concerning how well this program is performing relative to the established performance benchmarks. Providers are expected to reach or exceed the performance benchmark. Those programs that do not reach the performance benchmark may be offered training and technical assistance. It is the responsibility of the agency to ensure that these numbers are reviewed on a quarterly basis and to contact the County if the performance of this program falls short. According to this information, nothing further is required of your program at this time.

37. HOW TO READ DASHBOARD Introduction Performance Measure Current Quarter Discharges: 7 Report QTR N Total for Fiscal Year by Quarter (%) Cumulative Performance (%) Performance Benchmarks (%) 1 st 2 nd 3 rd 4 th N Year 1 Year 2 Year 3 Participants in Treatment at least 30 Days 7100 1010080 Participants in Treatment at least 90 Days 51007188065 Participants with Exit Interviews 6678688050 37 Performance Dashboard Outpatient Counseling Program 1234 Any Street, Los Angeles, CA 90025 This dashboard provides information concerning how well this program is performing relative to the established performance benchmarks. Providers are expected to reach or exceed the performance benchmark. Those programs that do not reach the performance benchmark may be offered training and technical assistance. It is the responsibility of the agency to ensure that these numbers are reviewed on a quarterly basis and to contact the County if the performance of this program falls short. According to this information, nothing further is required of your program at this time.

38. HOW TO READ DASHBOARD Introduction Two main areas to note on all dashboards –The underlined comment above the table –The table at the top of the dashboard 38

39. HOW TO READ DASHBOARD Introduction Performance Measure Current Quarter Discharges: 7 Report QTR N Total for Fiscal Year by Quarter (%) Cumulative Performance (%) Performance Benchmarks (%) 1 st 2 nd 3 rd 4 th N Year 1 Year 2 Year 3 Participants in Treatment at least 30 Days 7100 1010080 Participants in Treatment at least 90 Days 51007188065 Participants with Exit Interviews 6678688050 39 Performance Dashboard Outpatient Counseling Program 1234 Any Street, Los Angeles, CA 90025 This dashboard provides information concerning how well this program is performing relative to the established performance benchmarks. Providers are expected to reach or exceed the performance benchmark. Those programs that do not reach the performance benchmark may be offered training and technical assistance. It is the responsibility of the agency to ensure that these numbers are reviewed on a quarterly basis and to contact the County if the performance of this program falls short. According to this information, nothing further is required of your program at this time.

40. HOW TO READ DASHBOARD Introduction Performance Measure Current Quarter Discharges: 7 Report QTR N Total for Fiscal Year by Quarter (%) Cumulative Performance (%) Performance Benchmarks (%) 1 st 2 nd 3 rd 4 th NYear 1 Year 2 Year 3 Participants in Treatment at least 30 Days 7 100 10100 80 Participants in Treatment at least 90 Days 5 10071880 65 Participants with Exit Interviews 6 6786880 50 40 Performance Dashboard Outpatient Counseling Program 1234 Any Street, Los Angeles, CA 90025 This dashboard provides information concerning how well this program is performing relative to the established performance benchmarks. Providers are expected to reach or exceed the performance benchmark. Those programs that do not reach the performance benchmark may be offered training and technical assistance. It is the responsibility of the agency to ensure that these numbers are reviewed on a quarterly basis and to contact the County if the performance of this program falls short. According to this information, nothing further is required of your program at this time.

41. HOW TO READ DASHBOARD Introduction Performance table divided into 5 columns that provide information on discharged clients entered into LACPRS. These columns include: –Performance measure –Report Quarter –Total for Fiscal Year by Quarter (%) –Cumulative Performance (%) –Performance Benchmark (%) 41

42. HOW TO READ DASHBOARD Introduction Performance Measure Current Quarter Discharges: 7 Report QTR N Total for Fiscal Year by Quarter (%) Cumulative Performance (%) Performance Benchmarks (%) 1 st 2 nd 3 rd 4 th NYea r 1 Yea r 2 Yea r 3 Participants in Treatment at least 30 Days 710 0 1010 0 80 Participants in Treatment at least 90 Days 510 0 7188065 Participants with Exit Interviews 6678688050 42 Performance Dashboard Outpatient Counseling Program 1234 Any Street, Los Angeles, CA 90025 This dashboard provides information concerning how well this program is performing relative to the established performance benchmarks. Providers are expected to reach or exceed the performance benchmark. Those programs that do not reach the performance benchmark may be offered training and technical assistance. It is the responsibility of the agency to ensure that these numbers are reviewed on a quarterly basis and to contact the County if the performance of this program falls short. According to this information, nothing further is required of your program at this time.

43. HOW TO READ DASHBOARD Introduction Performance Measure Current Quarter Discharges: 7 Report QTR N Total for Fiscal Year by Quarter (%) Cumulative Performance (%) Performance Benchmarks (%) 1 st 2 nd 3 rd 4 th NYea r 1 Yea r 2 Yea r 3 Participants in Treatment at least 30 Days 710 0 1010 0 80 Participants in Treatment at least 90 Days 510 0 7188065 Participants with Exit Interviews 6678688050 43 Performance Dashboard Outpatient Counseling Program 1234 Any Street, Los Angeles, CA 90025 This dashboard provides information concerning how well this program is performing relative to the established performance benchmarks. Providers are expected to reach or exceed the performance benchmark. Those programs that do not reach the performance benchmark may be offered training and technical assistance. It is the responsibility of the agency to ensure that these numbers are reviewed on a quarterly bass and to contact the County if the performance of this program falls short. According to this information, nothing further is required of your program at this time.

44. HOW TO READ DASHBOARD Introduction Table provides info on Performance measures and benchmarks –Benchmarks based on County average for outpatient counseling programs Current Quarter Discharges –Number of discharges for the reporting quarter Report Quarter (QTR) N –Number of clients who met the performance measure 44

45. HOW TO READ DASHBOARD Introduction Performance Measure Current Quarter Discharges: 7 Report QTR N Total for Fiscal Year by Quarter (%) Cumulative Performance (%) Performance Benchmarks (%) 1 st 2 nd 3 rd 4 th N Year 1 Year 2 Year 3 Participants in Treatment at least 30 Days 7100 1010080 Participants in Treatment at least 90 Days 51007188065 Participants with Exit Interviews 6678688050 45 Performance Dashboard Outpatient Counseling Program 1234 Any Street, Los Angeles, CA 90025 This dashboard provides information concerning how well this program is performing relative to the established performance benchmarks. Providers are expected to reach or exceed the performance benchmark. Those programs that do not reach the performance benchmark may be offered training and technical assistance. It is the responsibility of the agency to ensure that these numbers are reviewed on a quarterly basis and to contact the County if the performance of this program falls short. According to this information, nothing further is required of your program at this time.

46. HOW TO READ DASHBOARD Introduction Total for Fiscal Year by Quarter –The number in these boxes are PERCENTAGES –They tell us what percentage of discharged clients met the performance measure. In this case, 100% of discharged clients remained in treatment at least 30 days. –Agencies would compare the percent for each quarter with the performance benchmark at the end of the table. 46

47. HOW TO READ DASHBOARD Introduction Performance Measure Current Quarter Discharges: 7 Report QTR N Total for Fiscal Year by Quarter (%) Cumulative Performance (%) Performance Benchmarks (%) 1 st 2 nd 3 rd 4 th N Year 1 Year 2 Year 3 Participants in Treatment at least 30 Days 7100 1010080 Participants in Treatment at least 90 Days 51007188065 Participants with Exit Interviews 6678688050 47 Performance Dashboard Outpatient Counseling Program 1234 Any Street, Los Angeles, CA 90025 This dashboard provides information concerning how well this program is performing relative to the established performance benchmarks. Providers are expected to reach or exceed the performance benchmark. Those programs that do not reach the performance benchmark may be offered training and technical assistance. It is the responsibility of the agency to ensure that these numbers are reviewed on a quarterly basis and to contact the County if the performance of this program falls short. According to this information, nothing further is required of your program at this time.

48. HOW TO READ DASHBOARD Introduction Performance Measure Current Quarter Discharges: 7 Report QTR N Total for Fiscal Year by Quarter (%) Cumulative Performance (%) Performance Benchmarks (%) 1 st 2 nd 3 rd 4 th NYear 1 Year 2 Year 3 Participants in Treatment at least 30 Days 7100 1010080 Participants in Treatment at least 90 Days 51007188065 Participants with Exit Interviews 6678688050 48 Performance Dashboard Outpatient Counseling Program 1234 Any Street, Los Angeles, CA 90025 This dashboard provides information concerning how well this program is performing relative to the established performance benchmarks. Providers are expected to reach or exceed the performance benchmark. Those programs that do not reach the performance benchmark may be offered training and technical assistance. It is the responsibility of the agency to ensure that these numbers are reviewed on a quarterly basis and to contact the County if the performance of this program falls short. According to this information, nothing further is required of your program at this time.

49. HOW TO READ DASHBOARD Results The narrative description of the data is below the performance table under the section “Results” –There are two periods in which results are listed: Current Quarter Cumulative Results –There are two types of results: Met/exceeded Not met 49

50. HOW TO READ DASHBOARD Results RESULTS Current Quarter This program has met/exceeded the County Benchmarks for 30 Day LOS. This program has met/exceeded the County Benchmarks for 90 Day LOS. This program has met/exceeded the County Benchmark for Exit Interviews. Cumulative Results (Year to Date) This program has met/exceeded the County Benchmarks for 30 Day LOS. This program has met/exceeded the County Benchmarks for 90 Day LOS. This program has met/exceeded the County Benchmark for Exit Interviews. 50

51. HOW TO READ DASHBOARD Results RESULTS Current Quarter This program has met/exceeded the County Benchmarks for 30 Day LOS. This program has met/exceeded the County Benchmarks for 90 Day LOS. This program has met/exceeded the County Benchmark for Exit Interviews. Cumulative Results (Year to Date) This program has met/exceeded the County Benchmarks for 30 Day LOS. This program has met/exceeded the County Benchmarks for 90 Day LOS. This program has met/exceeded the County Benchmark for Exit Interviews. 51

52. HOW TO READ DASHBOARD Results RESULTS Current Quarter This program has met/exceeded the County Benchmarks for 30 Day LOS. This program has met/exceeded the County Benchmarks for 90 Day LOS. This program has met/exceeded the County Benchmark for Exit Interviews. Cumulative Results (Year to Date) This program has met/exceeded the County Benchmarks for 30 Day LOS. This program has met/exceeded the County Benchmarks for 90 Day LOS. This program has met/exceeded the County Benchmark for Exit Interviews. 52

53. HOW TO READ DASHBOARD “Fell Below: 1-19%” Performance Measure Current Quarter Discharges: 8 Report QTR N Total for Fiscal Year by Quarter (%) Cumulative Performance (%) Performance Benchmarks (%) 1 st 2 nd 3 rd 4 th N Year 1 Year 2 Year 3 Participants in Treatment at least 30 Days 675 157580 Participants in Treatment at least 90 Days 3465074765 Participants with Exit Interviews 65075126050 53 The performance of this program requires improvement of 1-19% on one or more of three performance measures.

54. HOW TO READ DASHBOARD “Fell Below: 1-19%” All sections are the same for all reports – only the data presented is different –Look to “Results” and to learn what performance measure needs improvement. Current Quarter This program has not met the County Benchmarks for 30 Day LOS. This program has not met the County Benchmarks for 90 Day LOS. This program has met/exceeded the County Benchmark for Exit Interviews. Cumulative Results (Year to Date) This program has not met the County Benchmarks for 30 Day LOS. This program has not met the County Benchmarks for 90 Day LOS. This program has met/exceeded the County Benchmark for Exit Interviews. 54

55. HOW TO READ DASHBOARD “Fell Below: 1-19%” Also, look to Next Steps: Provides tips on how to improve performance in each of the performance areas Tips are straightforward and should not require additional assistance from SAPC 55

56. 30-Day Length of Stay (LOS): Check out the NIATx website at www.niatx.net to learn of ways to improve your programs’ 30 day LOS. www.niatx.net 90-Day Length of Stay (LOS): Program participants who are not in treatment at least 90 days may not fully benefit from treatment. The patient does not have to be in this program for the full 90 days if he or she transferred from a briefer treatment stay elsewhere. In order to ensure the LOS is calculated correctly, be sure that the client ID is identical to what was used with the patient in the prior program. If you are transferring the patient to another level of care, be sure to follow-up with the program to determine if the patient enrolled. If your program does not span for 90 days, please notify your program auditor. 56 HOW TO READ DASHBOARD Next Steps

57. Exit Interviews (Administrative Performance Measure): Exit interviews (or completed LACPRS discharges) are important in order to adequately measure how the patient improved over the course of treatment. If you are having problems with patients who leave treatment prior to the scheduled interview, try one of these strategies: –Inform the patient at admission that an exit interview is required prior to the patient discharging from the program. –As the date of discharge nears, remind the patient that there is an exit interview that needs to be completed prior to discharge – regardless of the patient’s discharge status. –Have counselors complete the Concurrent Recovery form – this information can then be used if the patient leaves treatment prior to the scheduled discharge. If the patient is present, complete the discharge as usual – do not use the form, even if completed as it does not collect all of the required discharge information and should only be used if necessary. If this program requires additional assistance, please contact your program auditor. 57

58. HOW TO READ DASHBOARD “Fell Below 20% or more” This dashboard will be posted when any single measure is 20% or more below the benchmark All sections are the same for all reports – only the data presented is different –Look to “Results” and to learn what performance measure needs improvement 58

59. HOW TO READ DASHBOARD “Fell Below 20% or more” Performance Measure Current Quarter Discharges: 98 Report QTR N Total for Fiscal Year by Quarter (%) Cumulative Performance (%) Performance Benchmarks (%) 1 st 2 nd 3 rd 4 th N Year 1 Year 2 Year 3 Participants in Treatment at least 30 Days 534154874980 Participants in Treatment at least 90 Days 514394677765 Participants with Exit Interviews 232123302350 59 The performance of this program requires improvement of 20% more on one or more of three performance measures.

60. HOW TO READ DASHBOARD “Fell Below 20% or more” Also look to “Expected Performance…” table –This process improvement project is designed to assist with 30 Day engagement –For other performance measures look to “Next Steps” for tips to improve 60

61. 61 Performance Measure Current Performance (%) Expected Performance FQ 3 (%) Expected Performance FQ 4 (%) Participants in Treatment at least 30 Days 546474 Participants in Treatment at least 90 Days --- Participants with Exit Interviews 233343 Also look to “Expected Performance” table HOW TO READ DASHBOARD “Fell Below 20% or more”

62. HOW TO READ DASHBOARD Definitions The dashboard also includes definitions of the terms used in the report LOS: Length of Stay. Participants in Treatment at Least 30 Days: Are those individuals who are in treatment at least 30 days, as measured by the LACPRS admission date and discharge date (last face to face) and had four treatment sessions during that time. The treatment sessions can include the individual counseling sessions to complete the assessment and treatment plan as well as any form of group counseling. 62

63. HOW TO READ DASHBOARD Definitions Participants in Treatment at Least 90 Days: Are those individuals who were in treatment at 30 days and remained in treatment for 90 days or more, as measured by the LACPRS admission date and discharge date (last face to face). Participants with Exit Interviews: This performance measure is more of an administrative performance measure in that it assesses the ability of the program to collect the information necessary to produce patient outcomes. This measure is collected based on the response to the LACPRS discharge question, “Is the client available for an exit interview?” This performance measure is only counted for those clients who remained in treatment at least 30 days or more. 63

64. HOW TO READ DASHBOARD In looking at the dashboard: –Compare total fiscal year by quarter with the performance benchmark for each measure –Compare Cumulative Performance with the performance benchmark –Read the results section for further clarification –Then, if necessary, read the next steps and expected performance ONLY NECESSARY WHEN ONE OR MORE PERFORMANCE BENCHMARKS IS NOT MET 64

65. Additional Dashboard Info For the most accurate dashboards –A/D data entered weekly –All data entered by the last day of the month Dashboards posting in same area on the LACPRS system as the site reports –Posted on the 10 th of the month subsequent to the end of each quarter –e.g. 3 rd quarter dashboards will be posted by April 10 th (end of quarter is March 31 st ) –ONLY Executive Directors and their designee(s) have access 65

66. Final Note on Dashboards Located in LACPRS, like the site reports Based on LACPRS data –Late data or inaccurate data will affect results –Advise executive directors to review reports If you want more info on LACES, outcome vs. performance measures, site reports, etc. go to www.laces-ucla.org.www.laces-ucla.org 66

67. Questions? 67

68. Contact Your Presenters Desiree A. Crevecoeur-MacPhail, Ph.D. (310) 267-5207 email: desireec@ucla.edudesireec@ucla.edu Loretta L. Denering, M.S. (310) 267-5312 email: lransom@ucla.edulransom@ucla.edu