1. Kendon ConradBarth Riley University of Illinois at Chicago Michael L. Dennis Chestnut Health Systems

2. Overview  Global Appraisal of Individual Needs (GAIN)  Benefits of Computerized Adaptive Testing  CAT: How Does it Work  Examples of CAT in Clinical Assessment Triage of persons around treatment decisions for starting and stopping rule Content Balancing over multiple clinical dimensions Identification persons with atypical symptom presentations

3. The GAIN  Comprehensive biopsychosocial instrument designed for intake into substance abuse treatment. Provides 5 axes DSM-IV diagnoses Also supports treatment planning, outcome monitoring and program evaluation Versions varying from 2-5 minute screener, 20-30 minute quick, and 1-2 hour full Over 103 scales, 1000 created variables, and text based narrative report The biggest problem is how long it takes

4. The Benefits of Computerized Adaptive Testing

5. General and Targeted Measures  Generalized Heavy response burden Lack specificity  Targeted Floor and ceiling effects Limited content validity Don’t “talk with each other.”

6. Tailoring Outcome Measurement Instrument A Instrument B Instrument C CAT Selects items from Item Bank administer item ?

7. Benefits of CAT & Item Banking CAT Item Bank Respondent Burden Tailoring/ Specificity Coverage of content domains Floor and ceiling effects

8. CAT vs. Short Forms  CAT has been found to be superior to “short forms” of tests, yielding more precise measures.

9. CAT: What Is It and How Does It Work?

10. Computerized Adaptive Testing Decreased Difficulty Typical Pattern of Responses Increased Difficulty Middle Difficulty Score is calculated and the next best item is selected based on item difficulty +/- 1 Std. Error CorrectIncorrect

11. Item Selection  There are several methods for selecting items during a CAT.  The most common method is to find the item that provides the most information given the current estimate of the measure.

12. Item Selection cont.

13.  Item selection can also take into account the types of domains of items to be represented in the CAT session.  Examples: Items necessary for DSM-IV diagnosis

14. Stop Rules  The stop rule, which determines when the item administration process of the CAT ends, can be based on: Measurement precision Number of items administered Test-taking time Some combination of the above

15. Item Bank Size  The more items there are in an item bank, the more likely it is that items that are tailored to an individual’s level on the measured variable will be available.  Typically, item banks consist of hundreds of items.  The number of items will likely depend on The number of constructs or domains being assessed. Whether one wishes to estimate a measure or classify persons into groups.

16. CAT for Clinical Assessment  The application of CAT to clinical research and assessment raises several new measurement issues.  Triage of persons around treatment decisions for starting and stopping rule  Content Balancing over multiple clinical dimensions  Identification of persons with atypical presentation of symptoms

17. Example 1 Triage of Individuals to Support Clinical Decision-Making

18. Classifying Persons Using CAT  CAT is typically used to estimate a measure  Few studies have examined the use of CAT to place persons into diagnostic groups.  For placing persons into diagnostic groups, it is desirable to vary the level of measurement precision depending on the category in which the person is placed.  Current CAT procedures do not allow one to vary measurement precision during the CAT session.

19. Triage of individuals to support clinicaldecision making  Strategy: Use of screener measures to set the value of thee initial measure and variable stop rules designed to maximize precision and efficiency for identification of persons in low, medium or high symptom severity  Implications: Taking into account initial location and/or precision around decision points can further improve the efficacy of assessment without hurting precision for decision making

20. Clinical Decision Making  To facilitate clinical diagnoses, it would be desirable for a CAT to: Classify patients by symptom severity Maximize measurement place within the area of the measure that is most critical for decision making. Use previously collected information to increase the efficiency of the CAT.

21. Study  We examined the ability of CAT to place persons into low, moderate and high levels of substance abuse and substance dependency.  The Substance Problem Scale (SPS) is a 16 item instrument that measures recency of substance use. “When was the last time you used alcohol or other drugs weekly?

22. Defining Cut Points  Cut points can be established by examining where persons with different levels of severity fall onto the measurement continuum.

23. The Start Rules Random: randomly select an item with difficulty calibrations between -0.5 and 0.5 logits (average level of difficulty). Screener: Select an item that has a difficulty level that most closely approximates the respondent’s measure on a previously administered screener (SDScr).

24. The Variable Stop Rule  Stop rules for the CAT were defined in terms of maximum standard error of measurement for the low, mid and high range of substance abuse severity.  The mid range stop rule was set to SE=0.35 for all simulations.  Low and High range SE ranged from SE=0.5 to 0.75 logits.

25. CAT Standard Error Middle range where decisions and made and precision is controlled High & Low ranges where there is little impact on clinical decisions and precision is allowed to vary more

26. The Item Selection Algorithm Start Rule Using Screener Select item Administer item Re-estimate measure & SE Stop rule met? End test Yes No Measure in high range? In mid range? Low range stop rule High range stop rule Mid range stop rule Yes No

27. Results  Screener starting rule improved efficiency of the CAT by approximately 7 percent compared to standard CAT procedures.  Variable stop rules improved efficiency by 15 to 38 percent, depending on definition of the mid range of severity, compared to standard stopping rules.

28. Results  Pre-calibration and variable stop rules resulted in accurate and efficient estimation of substance abuse severity.  The screener start rule had only a small effect on classification precision.

29. Next Step: Refining the Algorithm

30. Example 2: Content Balancing over Multiple Dimensions

31. Measuring Multiple Dimensions  Strategy: Use of content balancing methods in combination with conventional item selection procedures to ensure selection of items from each substantive domain  Implications: Assessment of an individual’s clinical profile can be conducted both efficiently and comprehensively at both the total and subscale level.

32. Internal Mental Distress Scale  The IMDS consists of the following subscales: Depression Symptom Scale Anxiety/Fear Symptom Scale Traumatic Distress Scale Homicidal/Suicidal Scale  IMDS also has 4 general somatic items as part of the total scale score.  Clinicians want to estimates for the overall severity and in each of the subscale areas.

33. Internal Mental Distress Scale by Content Area IMDS Subscale Item Calibrations -3 -2 0 1 2 3 Logits H/S Trauma Anxiety Depression Somatic

34. Example: No Content Balancing All Screener Items Administered

35. Example: No Content Balancing Depression: 2 H/S: 1 Anxiety: 1 Trauma: 1 Think other people don’t understand you: “Yes”

36. Example: No Content Balancing Depression: 3 H/S: 1 Anxiety: 1 Trauma: 1 Lost interest in things: “Yes”

37. Example: No Content Balancing Depression: 3 H/S: 1 Anxiety: 2 Trauma: 1 Thoughts people taking advantage of me: “No”

38. Example: No Content Balancing Depression: 4 H/S: 1 Anxiety: 2 Trauma: 1 Shyness: “No”

39. Example: No Content Balancing Depression: 4 H/S: 1 Anxiety: 3 Trauma: 1 Have to repeat action over and over: Yes

40. Results  If continued to 13 items –  Except for screener items, no hostility/suicide or trauma items were administered during the CAT session.  Mixed precision on the subscales

41. No Content Balancing Low Precision Estimates

42. IMDS by Content Area

43. IMDS Screener Items -3 -2 0 1 2 3 Logits Suicidal Trauma Anxiety Depression Somatic M1C2 Suicidal thoughts M2A Traumatic memories give distress M1D10 Thoughts should be punished M1B1 Trapped, lonely, sad, depressed M1A2 Sleep trouble Answers N, Y, Y, N, N and estimate overall Rasch severity as IMDS= -0.5 logit

44. IMDS Subscale Calibrations -3 -2 0 1 2 3 Logits DepressionAnxietyTraumaSuicidal five screener items change in rank order of severity

45. IMDS Subscale Item Calibrations -3 -2 0 1 2 3 Logits DepressionAnxietyTraumaSuicidal five screener items

46. Re-estimating IMDS -3 -2 0 1 2 3 Logits Suicidal Trauma Anxiety Depression Somatic Revised Estimate five screener items

47. Cont. Balancing: CAT to Full IMDS Subscale & Total now have good precision

48. Example 3: Identifying Persons with Atypical Symptom Presentations

49. Overview  Strategy: Rasch person fit statistics can identify persons with atypical clinical presentations in a computerized adaptive testing context  Implications: Clients sometimes endorse severe clinical symptoms that are not reflected by overall scores on standard assessments. Using statistics that can identify persons with such an atypical presentation has important clinical implications.

50. Rasch Fit Statistics  Both infit and outfit follow a chi-square distribution where the high scores are of primary concern  Infit or “Randomness”: More changes between yes/no that would be expected based on overall severity. Low – almost too perfect fit High –more transitions than expected  Outfit or “Atypicalness”: Focuses more on the tail ends –Group of answers Used to detect unexpected outlying, off-target responses. Outlier sensitive Low – almost too perfect fit High – endorsed high severity items, but not the percursor items. (e.g.., easier items)

51. Problems with Fit Responses by Severity Low HighRandomnessAtypicalness 1111111110000000000.30.5 1111010110001000000.61.0 1111110101000000001.0 1110000111000000000.91.3 0111111111000000003.81.0 1111111110000000013.81.0 10101010101010101010104.02.3 00000000000011111112.64.3

52. Clinical Implications of Misfit  Misfit in the context of clinical assessment can reflect: Difficulty understanding the assessment Cross-cultural effects Differential effects of treatment on some symptoms but not others  Our analyses indicate that there are subgroups who endorse severe symptoms without endorsement of milder symptoms.  Example: atypical suicide profile

53. Example: Atypical Suicide  Depression is regarded as the major risk factor for suicide.  However, there is a less common profile characterized by suicide-related symptoms but in the absence of depressive symptoms.  This profile can be identified through the use of fit statistics (atypicalness). 00000000000011111 Depression Suicide

54. Atypical Suicide

55. Fit Statistics in CAT  Fit statistics such as infit and outfit become less sensitive to atypical response patterns as the number of items is reduced.  Since CAT usually administers items that the respondent has a 50% probability of endorsing, either a “yes” or a “no” response to a dichotomous question is equally likely, and therefore, consistent with the Rasch model.

56. Randomness by Number of Items Number of Items Randomness Categories < 0.750.75-1.33> 1.33 1623.6%58.2%18.2% 1228.2%55.6%16.2% 835.2%52.8%12.0% 451.1%44.0%4.9%

57. Atypicalness by Number of Items Number of Items Atypicalness Categories < 0.750.75-1.33> 1.33 1630.2%48.1%21.7% 1234.3%51.1%14.6% 838.4%53.2%8.4% 458.2%40.0%1.8%

58. Next Steps: Alternatives to Infit and Outfit  Several measures/procedures for detecting misfit have been developed, specifically for use with short tests and/or CAT. These include: Adjustment of critical values for fit statistics Statistical process control procedures Modified t, modified H and modified Z statistics (Dimitrov and Smith, 2006).

59. Potential of CAT in Clinical Practice  Reduce respondent burden  Reduce staff resources  Reduce data fragmentation  Streamline complex assessment procedures  Assist in clinical decision making  Identify persons with atypical profiles

60. Future Research  How do we put it all together?  Much of the research in the area of CAT has used computer simulation. There is a need to test working CAT systems in clinical practice.

61. Contact Information  A copy of this presentation will be at: www.chestnut.org/li/posters  For information on this method and a paper on it, please contact Barth Riley at barthr@uic.edu  For information on the GAIN, please contact Michael Dennis at mdennis@chestnut.org or see www.chestnut.org/li/gain