1. Cricket Mitchell, PhD CIMH Evaluation Consultant CiMH Palette of Measures Evaluation Training: Skill Streaming Checklists

2. Palette of Measures Evaluation: What You Will Need (slide 1 of 2) Palette of Measures Data Entry Shell v2 (Excel file) –Developed by CiMH and customized for each participating agency –Holds all data for clients served thru the Palette of Measures project –Demographics, service delivery information, pre- and post- outcome measure data Palette of Measures Data Dictionary v2 (Word document) –A guide for using the associated data entry shell –Defines each column in the excel file 2

3. Palette of Measures Evaluation: What You Will Need (slide 2 of 2) Outcome measures from the two-pronged approach –General Outcome Measure –Target-Specific Outcome Measure(s) –For example… Skill Streaming Checklists –Part of the Skillstreaming curriculum published by Research Press Must be purchased by interested Palette of Measures project participating agencies 3

4. Overview of Training Brief Overview of Palette of Measures evaluation protocol Skill Streaming Checklists: Target-specific measure for disruptive behavior –Administration –Scoring –Clinical Utility Instructions for Palette of Measures data entry and data submissions –Data entry: Skill Streaming Checklists –Data entry: Demographics & Services –Data submissions to CiMH 4

5. 5 Brief Overview of Palette of Measures Evaluation Protocol

6. Outcome Assessment Palette of Measures providers will track outcomes using data from pre- and post- administrations of standardized measures of functioning Pre- and Post- a “dose” of treatment / an intervention interval –General measure of youth mental health functioning (e.g., YOQ/YOQ-SR, CANS, Ohio Scales) –Target-specific measure linked to focus of treatment/intervention (e.g., AQ, ECBI, PHQ-9, PTSD-RI, RCADS) Providers may choose to administer mid-course assessments as well –e.g., at 3-month intervals in usual care 6

7. A note about the use of standardized assessment measures… (slide 1 of 2) Assessment is the beginning of developing a relationship with the child and family –Demonstrates a desire to know what the child and family are experiencing –By incorporating standardized assessment measures of functioning, the efficiency and thoroughness of assessment is enhanced 7

8. A note about the use of standardized assessment measures… (slide 2 of 2) Using standardized assessment measures of functioning… –Assists in initial clinical impressions –Provides valuable information to guide treatment/interventions –Assesses sufficiency of treatment delivered –Demonstrates treatment-related improvements in child functioning *Although the Skill Streaming Checklists are not standardized, they are structured assessments that provide information specific to treatment planning and outcome assessment. 8

9. 9 Skill Streaming Checklists

10. CiMH Skill Streaming Checklist Training Information on the administration, scoring, and clinical utility of the Skill Streaming Checklists was obtained from the Aggression Replacement Training™ book, written by Arnold P. Goldstein, Barry Glick and John C. Gibbs, and published by Research Press in 1998; and, from discussions with Education & Treatment Alternatives, Inc.’s Master Trainers in Aggression Replacement Training™ 10

11. Skill Streaming Checklist Description Target-specific measure for disruptive behavior Measures the extent to which a youth exhibits pro-social skills (a strength-based measure) Elementary and Adolescent versions available –Youth self-report –Parent/caregiver report –Teacher/staff report 5 – 15 minutes to complete 11

12. Skill Streaming Checklist Description Part of the Skillstreaming curriculum published by Research Press –After purchase of Skillstreaming program materials (~$45 per age version), there is no per-use cost **20% discount if purchased through CIMH –http://www.researchpress.com/skillstreaming/a dolescent/product/item/6576/http://www.researchpress.com/skillstreaming/a dolescent/product/item/6576/ 12

13. Skill Streaming Checklist Description 50 items (adolescent) 60 items (elementary) 5-point Likert scale response options –Almost never (1) –Seldom (2) –Sometimes (3) –Often (4) –Almost always (5) Total Score –Note: The SS Checklists are not standardized measures; therefore, a subset of items can be administered, corresponding to the specific set of pro-social skills youth are being taught. This is referred to as a “targeted assessment”. 13

14. Example: Items from the Skill Streaming Checklist (adolescent) Does the youngster pay attention to someone who is talking and make an effort to understand what is being said? –Skill: Listening Does the youngster request assistance when he/she is having difficulty? –Skill: Asking for Help Does the youngster let others know which emotions he/she is feeling? –Skill: Expressing Your Feelings 14

15. Skill Streaming Checklist Administration Administer pre- and post- a dose of treatment, or an intervention interval, focused on disruptive behavior –“Student” version of the Skill Streaming Checklist completed by client –“Parent” version completed by parent or caregiver –“Teacher” version completed by teacher or staff member 15

16. Let’s take a look at the Skill Streaming Checklist (Student version)... 16

17. Administration and Clinical Utility of the Skill Streaming Checklist Informs specific pro-social skills for which youth have the greatest need for intervention –Can be used to determine the subset of skills that are selected for skills training curriculum Administer checklist to program staff members and ask them to complete it with the average client in mind Administer to clients to determine actual skill needs prior to intervention 17

18. Clinical Utility of the Skill Streaming Checklist “Targeted Assessment” –Pre- and Post- test only those skills that are taught in your curriculum –i.e., if 10 skills are taught – those same 10 skills are selected for pre- and post- outcome assessment When all 50 skills are assessed, the gains youth make in 10 skills tend to be washed out by the 40 skills that youth have not had an opportunity to learn and practice –If the entire checklist is administered pre-treatment to determine the subset that will be taught Score only those that are selected for the curriculum Administer only the subset at the post-assessment 18

19. Skill Streaming Checklist Scoring Each item on the checklist has five response options with values ranging from 1 – 5 To obtain the Total Score, sum the values of the responses circled by the respondent –If all 50 skills have been assessed, Total Scores will range from 50 – 250 –If a subset of skills are assessed in a “targeted assessment”, the range will change e.g., if 10 skills have been assessed, Total Scores will range from 10 – 50 Missing items are not scored 19

20. Skill Streaming Checklist Scoring Given the variability in number of skills that can be assessed, Total Scores are not used for outcome analyses For pre-/post- outcome analyses, a mean item response is calculated (the average item response) –Total Score ÷ Number of Skills Assessed –Scores for outcome analyses range from 1.0 to 5.0 20

21. Clinical Utility of the Skill Streaming Checklist Assesses youth’s use of pro-social skills (a strengths-based measure) –Higher scores indicate better use of pro-social skills –Lower scores indicate poorer use of pro-social skills Informs which pro-social skills youth have the greatest need for intervention Comparisons of pre/post scores demonstrate treatment-related outcomes in youth’s use of pro- social skills 21

22. 22 Palette of Measures Data Entry and Data Submissions

23. Palette of Measures Data Entry Shell There is a separate spreadsheet in the excel workbook (aka database) for each type of data: –Demographics & Services –Pre- General outcome measure –Post- General outcome measure –Pre- Target-specific measure(s) –Post- Target-specific outcome measure(s) Specific outcome measure spreadsheets included in each agency’s database varies across Palette of Measures project participants 23

24. Palette of Measures Data Entry Shell In addition to the spreadsheets that hold data... –There is an Instructions spreadsheet Basic data entry instructions Contact information for T.A. (Cricket Mitchell) –There is a Data Lists spreadsheet at the end of the workbook that you will not use Data Lists populate the pull-down menus in other spreadsheets 24

25. Palette of Measures Data Entry: Skill Streaming Checklist 25 There is a separate spreadsheet for Pre-SS data, Post-SS data, and Mid-SS data In each spreadsheet, there is a field for the Version, # of Skills Assessed, and Total Score for all possible informants In the event of missing data, leave the fields blank/empty. Do not enter text into any of the fields. Pre- SSChecklists Date of Skill Streaming ChecklistsSelf-Report:Parent Report: Teacher/Staff Report: Client ID#AssessmentSSVersion1 C- SS#Skills1 C- SSScore1 P- SS#Skills1 P- SSScore1 S- SS#Skills1 S- SSScore1

26. Palette of Measures Data Entry: Skill Streaming Checklist 26 For agencies who will conduct mid-treatment assessments, indicate the Assessment Interval in the Mid-SS spreadsheet by selecting from the available pull-down menu e.g., 1 st mid-treatment assessment, 2 nd mid-treatment assessment Mid- Self-Report: SSChecklistsDate ofAssessment Skill Streaming Checklists Client ID#AssessmentIntervalSSVersionMC-SS#SkillsMC-SSScoreM

27. But, before you enter any outcome data, you’ll enter Demographics & Service Delivery Information... But, before you enter any outcome data, you’ll enter Demographics & Service Delivery Information... 27

28. Palette of Measures Data Entry: Demographics & Services (1 of 2) 28 Use a unique identifier for Client ID# Categorical variables will have pull-down menus from which you’ll select an option (e.g., gender, ethnicity, language) Dates should be entered as xx/xx/xxxx Axis I diagnoses s/b the numeric DSM-IV code Therapist ID is optional Client Information: Primary(DSM-IV code) Client ID#DOBGenderEthnicityLanguagePrimary Axis ISecondary Axis ITherapist ID

29. Palette of Measures Data Entry: Demographics & Services (2 of 2) 29 Select Focus from pull-down menu (e.g., anxiety, depression) The shell will hold data for up to 4 foci, or treatment targets Enter Date of First Session The remaining fields are to be completed at the end of treatment targeting this particular focus (e.g., Date of Last, # Sessions) 2 levels of “Completed?” Treatment targeting this particular focus Overall service delivery Focus 1 of Treatment Date of Total #Completed (if Focus 1 not completed)Completed (if Services not completed) Focus1 First Session Last Session of SessionsFocus 1?ReasonServices?Reason

30. Palette of Measures Data Submissions (slide 1 of 3) Data submissions to CiMH will occur twice a year throughout the duration of the project –The end of each May (reflecting all clients served from the initiation of the project through the end of that April) –The end of each December (reflecting all clients served from the initiation of the project through the end of that November) –Note that this is the anticipated schedule; actual data submission dates may vary slightly An email notice will be sent to Palette of Measures site leads approximately one month in advance of each data submission deadline 30

31. Data Submissions (slide 2 of 3) Providers may choose from among the following methods for submitting their Palette of Measures Excel databases to CiMH: –Use YouSendIt, or another secure web-based transfer site, to submit data electronically YouSendIt (www.YouSendIt.com) is a vendor that supports the secure transfer of electronic data (encrypted and password- protected)www.YouSendIt.com –Mail a password-protected CD to CiMH and submit the password separately (via email or phone) –Email an encrypted, password-protected file(s) to CiMH and submit the password separately (via email or phone) 31

32. Data Submissions (slide 3 of 3) After data are submitted, sites continue to enter new data into the same database –Always reflects an ongoing, historical record of clients served through the Palette of Measures project Every effort is made to distribute reports within two months of each data submission –Aggregate and site/agency-specific reports 32

33. 33 Questions

34. 34 The End Contact Information Cricket Mitchell, PhD Email: cmitchell@cimh.orgcmitchell@cimh.org Cell phone: 858-220-6355