1. National Child Traumatic Stress Initiative (NCTSI) Data and Evaluation Webinar

2. Webinar Overview Overview of data and evaluation for the NCTSI:  National Center for Child Traumatic Stress (NCCTS) Data and Evaluation Program/Core Data Set (CDS)  Cross-site Evaluation  Transformation Accountability (TRAC)

3. NCCTS Data and Evaluation Program Range of quality improvement, data collection, and evaluation initiatives  CDS: Captures quantitative data on Network-supported treatments and services Types of traumatic events Treatments delivered Functioning of youth over time

4. Cross-site Evaluation Eight study components:  Child and family outcomes and satisfaction with services  Network functioning with regard to trainings, collaboration, product development, and product adoption and dissemination

5. TRAC TRAC:  Uniform tool for collecting data on mental health outcomes across programs  Grantees are required to collect TRAC data as part of SAMHSA’s reporting obligations

6. NCCTS Data and Evaluation Program: Overview and Next Steps Ernestine Briggs-King, Ph.D. National Center for Child Traumatic Stress Duke University School of Medicine

7. Objective: How Do All the NCTSI Data Collection Initiatives Fit Together? QPR Core Data Set eSUF, Metrics, Product Evaluation Cross-site Evaluation TRAC Surveys

8. Data and Evaluation Program Ernestine Briggs-King, Ph.D. 919-682-1552, ext. 254 brigg014@mc.duke.edu Bart Evans, B.S. Project Leader 919-668-8981 bart.evans@duke.edu Rebecca Wilgus, RN, M.S.N. 919-668-8897 rebecca.warlick@duke.edu

9. Data and Evaluation Program: What Do We Do? The Data and Evaluation Program is broadly concerned with data collection, analysis, and dissemination. The Duke Clinical Research Institute (DCRI) provides the following services: ●Data operations (includes project management, regulatory, contracts, finance) ●Data management ●Data coordinating activities The NCTSN Data Repository is housed at the DCRI. Data Operations Committee includes key members from NCTSN Centers, SAMHSA, Macro, UCLA, and Duke.

10. Regulatory Issues Data Collection Initiatives ●Electronic Service Utilization Form (eSUF) Aggregate data on clients, services, and trainings ●NCTSN member-driven data collection initiatives Surveys (e.g., New Grantee Surveys) Product evaluation Collaborative group initiatives ●Evaluation Training, implementation, and dissemination efforts ●Core Data Set (CDS) Client-level data about youth and families served Data and Evaluation Program: What Do We Do?

11. Introduction: CDS  Quality improvement initiative  Network-wide data collection that will provide answers to central Network questions  Essential for ensuring that the work done in the Network is systematically measured, disseminated, and recognized  Numerous benefits for participating clinicians, individual sites, and the Network as a whole ●Clinical improvements and utility ●Real-time reports ●Sustainability

12. NCTSN Clinical Summary Report

13. What is in the CDS?  Demographic and living situation information  Trauma history and detail  Indicators of severity  Treatment services and interventions  Standardized assessment measures Behavioral and Emotional Difficulties ●Child Behavior Checklist ●UCLA PTSD Reaction Index ●Trauma Symptom Checklist for Children-Alternate  CDS measures: administered at treatment entry, end of treatment (if short-term) or every 3 months

14. Bidirectional Data Flow ●All centers will have continuous (interactive) access to client-level reports (i.e., scoring of measures, summaries of client history) ●All centers will have continuous (interactive) access to data status reports that describe data entered by their center ●Each center can receive from Data Repository a ready-to- analyze data set of its own data

15. Next Steps: Data Collection Data collection and data entry resource requirements are determined by your implementation strategy ●Real-time—Clinicians collect and enter data online during assessment of client and family ●Ex post facto—Clinicians collect data on paper during their assessment, and data entry staff enter data into InForm at a later date ●Hybrid model—Combination of above models What will work best at your center??

16. Next Steps: Regulatory Processes Start the regulatory process soon!! Assign a “point person” Obtain documentation from your IRB Submit a Public Health Information Letter Signed Data Use Agreement Templates, roadmaps and technical assistance are available - Web site: www.nctsn.org - Data and Evaluation staff: Just a phone call away! If your center is not affiliated with a local IRB, the DUHS IRB may serve as your IRB of record. Contact Bart Evans at the Data and Evaluation Program for more information.

17. Next Steps: Training Training—Technology  How to use the InForm system Training—Clinical Content  How to complete the case report forms  How to administer the standardized assessments

18. NCTSN Data and Evaluation Program’s Commitment to You Customer support DCRI EDC Help Desk: edchelp@duke.edu Phone: 888-372-7743 NCTSN Data and Evaluation Team: nctsn@dcri.duke.edu Phone: 919-668-8182 Technical consulting and training Clinical expertise and guidance

19. Data and Evaluation Webinar Overview: Data Collection and Evaluation Activities

20. Introduction to Macro  Macro International Management and consulting research organization – More than 1,000 employees – Multiple offices in the United States and overseas – Many contracts with local, State, and Federal governments (80–90 percent of Macro’s work) More than 40 years as a government contractor – Applied health services research – Information technology – Organizational development – Communications 20

21. 21 The Common Goal To raise the standard of care and improve access to services throughout the United States for children and adolescents and their families who have experienced trauma

22. Monitoring and Evaluating the NCTSN  Evaluation has always played a large role in the NCTSN  Initial congressional authorization required a “rigorous evaluation plan” for each center (P.L. 106-310, 2001)  Evaluation plans were to include methods for assessing the effectiveness of: Processes Treatment/intervention outcomes 22

23. What is Cross-site Evaluation (CSE)? 23 Network Collaboration National Impact NREPP Descriptive and Clinical Outcome Study Satisfaction Study Trauma- Informed Services Adoption and Implementation CLIENT & FAMILY LEVEL NETWORK LEVEL Product Development and Dissemination Study PROVIDER LEVEL

24.  Macro wins competitive bid to conduct CSE  Macro begins year of evaluation design, development, and review History of NCTSN Growth and CSE Participation 24  CSE implementation ongoing  44 grantees currently funded  70 grantees ever funded  OMB approves CSE design  Macro begins implementing CSE  44 grantees currently funded  70 grantees ever funded  16 new grantees join NCTSN  12 grantees continue on consecutive awards  44 grantees currently funded  70 grantees ever funded 53 grantees currently funded 36 grantees currently funded 20 grantees currently funded NCTSI program authorized under Children’s Health Act of 2000  10 new grantees join NCTSN  5 grantees continue on consecutive awards  44 grantees currently funded  80 grantees ever funded 200020012002200320042005200620072008 Fiscal Years Note: Counts of “currently funded” grantees exclude the National Center for Child Traumatic Stress  5 new centers join NCTSN  3 grantees continue on consecutive awards  54 grantees currently funded  85 grantees ever funded

25. Steps Toward Implementation Ongoing Technical Assistance 25 Continuum of Training and Technical Assistance MONTHLY UPDATES Reviewing Details of the Cross-site Evaluation Creating Center- based Strategies for Implementation Regional Training EVALUATION LIAISON SUPPORT Component- specific Updates by E-mail/Ongoing Consultation Cross-site Procedures Manual Individual Training and Technical Assistance

26. Getting Started: Practical Help Tips

27. Getting Started 27 Engaging Stakeholders in the Evaluation Staff members Community members Other relevant stakeholders Ensuring Organizational Readiness Staff hiring plan and roles Staff orientation to grant activities Communication with current grantees Beginning Institutional Review Board (IRB) Process Local IRB plans Local IRB application process and deadlines

28. 28 Getting Started (cont.)

29. Contact Information If you have questions or comments or need more information, feel free to contact the following members of the CSE team: –Christine Walrath, Officer in Charge E-mail: cwalrath@macrointernational.com –John Gilford, Project Director E-mail: jwgilford@macrointernational.com –Bhuvana Sukumar, Deputy Project Director E-mail: bhuvana.sukumar@macrointernational.com 29

30. NCTSI CMHS Transformation Accountability (TRAC) Center for Mental Health Services (CMHS) and Performance Management NCTSI CMHS Transformation Accountability (TRAC)

31. Agenda Performance management and CMHS TRAC and NCTSI Next steps

32. CMHS Performance Management Federal/Annual GPRA PART Review National Outcome Measures (NOMs) TRAC Eval Federal/4 yrs SAMHSA CMHS NCTSI

33. TRAC Overview CMHS-TRAC is a Web-based, centralized, data platform that collects and reports outcome measures on CMHS programs for the Government Performance and Results Act (GPRA).

34. TRAC: TRAC: A New Approach to Accountability TRAC is a strategic imperative for CMHS Driven by: –Government-wide requirements –SAMHSA data strategy –Center commitment to performance management Outcome measures targeted at: –Consumer services –Technical assistance –Infrastructure development –Prevention

35. SAMHSA-wide Domains: NOMs SAMHSA is obtaining OMB approval to use NOMs for GPRA reporting purposes: –Functioning –Stability in Housing –Employment and Education –Crime and Criminal Justice Status –Perception of Care –Social Connectedness –Access/Capacity –Retention –*Cost-effectiveness –*Use of Evidence-based Practices *Indicates that measures have not yet been developed for these domains

36. CMHS GPRA Data Collection: Current Status  Most programs were collecting GPRA data in some form  Lack of standardized outcome data: –Limits the ability to demonstrate what CMHS has achieved –Creates difficulties in aggregating data to summarize program performance  NOMs tool is now the standard data collection method for all service programs

37. TRAC and NCTSI Service Treatment Grants NOMs tool used to collect interviews on all consumers enrolled in services: –Paper NOMs tool/electronic NOMs tool Data collection points—baseline, reassessment, and discharge Reassessment is collected every 3 months while consumer is receiving services No followup after discharge or if consumer is lost to contact for 90 days or more

38. What Can TRAC Do for You?  Reports: –Enrollment and reassessment rates –NOMs Outcome Report –Cross Tabs and Frequencies  Data download: –Grantees can download data through CMHS-TRAC system and run analysis

39. How Can Grantees Prepare? Visit the TRAC Web site for more information at https://www.samhsa-gpra.samhsa.gov/home/ index.htm (Click CMHS TRAC  General Information  Service Programs  Related Links ) Contact TRAC Help Desk with your questions at 888-219-0238 or trachelp@westat.com If necessary, seek IRB approval immediately Determine processes for data collection

40. Q & A