Verity Questionnaire Development

The truth about outcomes questionnaires!  All patient self report outcome questionnaires tend to load on a common factor: “global distress”.  Due to the high degree of correlation between items, well constructed questionnaires of items can have coefficients of reliability and construct validity comparable to measures of 30 or more items.  Even ultra brief questionnaires of 4-9 items may have adequate reliability and validity for must measurement needs.

Measurement 2.0 Measurement Reliance on copyrighted and published questionnaires Copyright holder may charge fees for the use of questionnaires Copyright holder may place conditions or restrictions on the use of questionnaires Measurement 2.0  Item banks and resulting questionnaires belong to community of users.  No fees for questionnaires constructed from items in the shared item bank  Each organization is responsible for their own measurement methods and determines the appropriate content and use of questionnaires

Measurement 2.0 Questionnaire Development Measurement A pool of items are tested in various samples Item analysis used to select items for final questionnaire Questionnaire validated, usually in correlation studies with questionnaires measuring the same construct Questionnaire published in final form  Manual published Many years may pass before a new version is published Measurement 2.0  A pool of items are tested in various samples  Item analyses employ both Classical Test Theory and Item Response Theory  Item analysis used to select items for multiple versions of the questionnaires, depending on the needs of the users  Construct validity determined by factor analysis and comparison of results to known constructs.  Various organizations may conduct correlation studies to satisfy internal skeptics.  Multiple versions available, with the community of users determining which versions offer the greatest utility  Online manual constantly updated as data accumulates and needs of users dictate

Item Pool  Item content drawn from multiple sources  Items reworded for consistent format, brevity and ease of use  Item formats tested to identify optimal format for brief questionnaires

Item pool

Item Format  5 point Likert scale found to produce best results  Items scored 0 – 4  Mean item response tends to be in middle of range  4 or 3 point Likert scales tend to produce responses skewed toward on end of the scale  Liker scales with more than 5 points provide no additional benefit

Ease of use  Items written to 4 th grade reading level  Simple to understand frequency anchors  Never, Hardly Ever, Sometimes, Often, Very Often  Common sentence structure aids rapid completion How often in the past two weeks did you …feel unhappy or sad? …have little or no energy?

Reliability  Reported as coefficient of correlation (r)  Test-retest reliability- correlation between measure at two points in time  Less relevant to measures of change  Coefficient alpha - measure of internal consistency; reflects degree of inter-correlation between items on the same scale.  Useful for measures designed to be sensitive to change.

Low reliability Low validity  Validity is a function of reliability!!!!!  Low reliability leads to low correlations with other measures

Validity  You better have face validity!!!!!  Construct validity based on correlation to a known “construct” such as Global Distress  Factor analysis used to identify “constructs”  Concurrent validity based on correlations between measures of a common construct  ACORN correlation with Beck Depression=.78  ACORN correlation with Patient Health Questionnaire-9=.82

Reliability & item count  More items=greater reliability  Assuming they correlate with the same factor (construct)  Each additional item adds less additional reliability  Little additional gain beyond 15 well chosen items  Coefficient alpha =>.9 recommended for measures of individual change

Slope of change  Outcome questionnaires need items that change in therapy.  Items vary in their likelihood to show change.  Linear regression used to calculation the slope and intercept for each item  Intake score used to predict last score  Final score = Intake Score * slope + intercept

Regression lines  Feel unhappy or sad (Slope=.45; Intercept=.88)  Hear voices when alone (Slope=.63; Intercept=.06)

Item Change Calculator  Excel tool with normative data on all items  Mean intake, slope & intercept  Used to predict psychometric properties of new questionnaires  Used to assure equivalency of alternative forms of questionnaires

Item Response Theory  Permits us to evaluate how well items works for clients at varying levels of distress  Identify items sensitive for clients with very high levels of global distress scores  Thoughts of suicide; Worthless  Outpatient questionnaires ideally targeted for moderate to severe cases with a mix of items

Youth and child measures  Different mix of items for children or adolescents  More complex factor structure  Relatively weak sub factors for “externalizing” and “internalizing” items  Items tapping ADHD & conduct problems  Items tapping symptoms of depression and anxiety  Subscales both correlate with Global Distress  Child/Youth measures tend to require more items to achieve.9 reliability on Global Distress scale

Child or parent completed?  Questionnaires may be completed by the child or by an adult who knows the child well.  Different norms depending on who completes the questionnaire.  Adolescent & child measures produce similar scores for self or parent completed forms

“Recovery” questionnaires  Designed for clients in substance abuse recovery programs.  Contains both Global Distress Scale and Substance Abuse Recovery Scale  Items on Substance Abuse Recovery Scale have high face validity for recovery settings.

Substance Abuse Recovery Sample items  …feel accepted by your recovery support group  …feel sad or angry that you can’t use drugs or alcohol like other people  …participate in activities that support a clean and health lifestyle.  …feel a need to use alcohol or drugs

Client registration form  Completed at start of treatment  Used to collect case mix variables such as:  Age  Prior treatment history  Sex  Language & ethnicity  Diagnosis  Health status  History of physical and/or sexual abuse

Focusing on the relationship “…decades of research consistently demonstrate that relationship factors correlate more highly with client outcome than do specialized treatment techniques… therapists need to remember that the development and maintenance of the therapeutic relationship is a primary curative component of therapy and that the relationship provides the context in which the specific techniques exert their influence.” - Lambert & Barley (2001) Psychotherapy, 38(4):

Relationship building is an Evidence Based Practice! Practitioners are encouraged to routinely monitor patients’ responses to the therapy relationship and ongoing treatment. Such monitoring leads to increased opportunities to repair alliance ruptures, improve the relationship, modify technical strategies, and avoid premature termination. - Norcross & Lambert (2006) in Evidence-Based Practices in Mental Health, Norcross, Beutler & Levant (Eds), p. 218

Therapeutic alliance as a key ingredient  Large body of research on therapeutic relationship and working alliance show that it is an important factor in the outcome of psychotherapy  Horvath and Symonds, 1991: Meta-analysis of 24 studies attributed 26% of the difference in case outcomes to differences in alliance

Concept of Therapeutic Alliance Three Components:  Tasks: Behaviors and processes within the therapy session that constitute the actual work of therapy  Bonds: The positive interpersonal attachment between therapist and client of mutual trust, confidence, and acceptance  Goals: Objectives of therapy that both client and therapist endorse

Therapeutic Alliance Scale Sample items  I felt like we talked about the things that were important to me  I felt like the therapist liked and understood me  I felt the session was helpful.  I felt confident that the therapist and I worked well together  Did you feel that the clinician understood what it was like to be you

Alliance Scale Psychometrics  Items are heavily skewed in positive direction.  Scale scores are not normally distributed.  Cannot calculate reliability & validity using parametric statistics that assume normality of distribution  Items are only as “valid” as clinician’s ability to illicit honest and frank responses!

Alliance Results Measurement make a difference

High praise for clinicians Alliance scores at start of treatment

Alliance changes in treatment

Alliance change & outcome