17 March 2017 | DIA Statistics Community Webinar Developing PRO Instruments in Clinical Trials: Issues, Considerations, and Solutions 17 March 2017 | DIA Statistics Community Webinar

Developing PRO Instruments in Clinical Trials: Issues, Considerations, and Solutions Chairs: Laura Lee Johnson (FDA) Cheryl Coon (Outcometrix) Speakers: Wen-Hung Chen (FDA) Dennis Revicki (Evidera) Lisa Kammerman (AstraZeneca)

The views expressed in this presentation are those of the speakers, and do not necessarily represent an official FDA position.

Let’s Begin with a Poll... Please raise your hand if you have heard of the term Clinical Outcome Assessments (COAs) before today? If you have, do you know what COAs refer to? Please raise your hand if you have no idea what I am talking about? By the way, this is not uncommon.

A New Era of Patient Empowerment Dr. Janet Woodcock (2015): “It turns out that what is really bothering the patient and what is really bothering the doctor can be radically different things….patients are true experts in their disease. It's clear you have to start with an understanding of the impact of the disease on the people who have it, and what they value most in terms of alleviation before you set up a measurement and go forward with truly patient-focused drug development.” (PDUFA V Clinical Outcome Assessment Public Workshop, April 2015)

What are COAs? Clinical Outcome An outcome that describes or reflects how an individual feels, functions, or survives Clinical Outcome Assessment (COA) Assessment of a clinical outcome can be made through report by a clinician, a patient, a non-clinical observer or though a performance based assessment. There are four types of COAs: Clinician-reported outcomes (ClinROs) Observer-reported outcomes (ObsROs) Patient-reported outcomes (PROs) Performance outcomes (PerfOs) BEST (Biomarkers, EndpointS, and other Tools) http://www.ncbi.nlm.nih.gov/books/NBK326791/

Patient-Reported Outcomes (PROs) A measurement based on a report that comes directly from the patient (i.e., study subject) about the status of a patient’s health condition without amendment or interpretation of the patient’s response by a clinician or anyone else. A PRO can be measured by self-report or by interview provided that the interviewer records only the patient’s response. Symptoms or other unobservable concepts known only to the patient can only be measured by PRO measures. PROs can also assess the patient perspective on functioning or activities that may also be observable by others.

PRO Questions: Good/Bad/Ugly REAL example. Asked… We want to know the severity of bowel movement issue “What is the severity of your bowel movement? Very severe, moderately severe, mild, none” What does this mean? Unfortunately, not a whole lot Usually not this bad Source: Johnson 2016

Goals of this webinar This webinar will address when and how to evaluate the psychometric properties of a patient-reported outcome (PRO) instrument or other clinical outcome assessment (COA) used to construct a primary endpoint in a clinical trial What level of evidence is needed in terms of psychometric properties? What is the ideal setting for evaluating psychometric properties? What are the risks and ramifications for deviating from the ideal setting? What other things should be considered when planning for a psychometric evaluation using clinical trial data (e.g., power, blinding, timing)?

What are psychometric properties? Francis Galton defined psychometrics as “The art of imposing measurement and number upon operations of the mind”1 (1879!) Psychometric properties are statistical information intended to demonstrate that an instrument measures what it is supposed to measure (aka validity) and measures it in a precise manner (aka reliability) While not a psychometric property, score interpretation is a critical component to psychometrics – once you know that your instrument is reliable and valid, what does a score, or a score change, mean? Psychometrics provide confidence that scores from an instrument are truly representative of the concept in a clinical trial endpoint 1. Galton, F. (1879). Psychometric experiments. Brain: A Journal of Neurology, 11, 149-162.

How are psychometric properties evaluated? Psychometric methods may be similar to or different from biostatistical methods E.g., Construct validity can be assessed by comparing scores from groups known to be different using ANOVA E.g., Reliability can be computed across various ranges of the scale using item response theory While hypothesis testing is relevant, psychometric analyses are generally more exploratory in nature than traditional biostatistical analyses Methods to assess many psychometric properties can rely on cross- sectional data (e.g., reliability, validity, dimensionality), while others require longitudinal data from an interventional study (e.g., responsiveness to change, score interpretation)

Part One: Setting the Stage Wen-Hung Chen (WHC) US Food and Drug Administration Clinical Outcome Assessment Staff, OND/CDER Lisa Kammerman (LK) AstraZeneca Regulatory Specialist, Oncology, Biometrics and Information Sciences Dennis Revicki (DR) Evidera Senior Vice President, Outcomes Research

Wen-Hung Chen Pieces and level of evidence required in terms of demonstrating an instrument’s psychometric properties Ideal setting(s) to evaluate a COA’s psychometric properties

Evaluation of Psychometric Properties of Clinical Outcome Assessment for use in Clinical Trial: A Regulatory Perspective Wen-Hung Chen, PhD Clinical Outcome Assessment Staff Office of New Drugs (OND) Center for Drug Evaluation and Research (CDER) Food and Drug Administration (FDA)

Disclaimer The views expressed in this presentation are those of the speaker, and do not necessarily represent an official FDA position.

Evidence of Treatment Benefit Direct evidence of treatment benefit is derived from studies with endpoints that measure survival, or how patients feel and function in daily life Indirect evidence of treatment benefit is derived from studies with endpoints that measure other things (e.g., biomarkers*, distal measures) that are related to how patients survive, feel, or function *Biomarker: a physiologic, pathologic, or anatomic characteristic that is objectively measured and evaluated as an indicator of some normal or abnormal biologic function, process or response to a therapeutic intervention

Types of Clinical Trial Study Endpoints as Seen by FDA/CDER Performance Outcomes (PerfOs) Clinician-reported Outcomes (ClinROs; includes overall survival) Observer-reported Outcomes (ObsROs) Patient-reported Outcomes (PROs) Clinical Outcome Assessments (COAs) Often a biomarker* that is intended as a substitute for how a patient feels, functions, or survives Surrogates

When is a Clinical Outcome Assessment Adequate for use? Regulatory standard: measures are well-defined and reliable Empiric evidence demonstrates that the score quantifies the concept of interest in the targeted context of use What does this mean? This means measuring the right thing (concept of interest), in the right way in a defined population (targeted context of use), and the score that quantifies that ‘thing’ does so accurately and reliably, so that the effects seen in the outcome assessment can be interpreted as a clear treatment benefit.

FDA PRO Guidance Defines good measurement principles to consider for “well-defined and reliable” (21 CFR 314.126) PRO measures intended to provide evidence of clinical benefit All COAs can benefit from the good measurement principles described within the guidance Provides optimal approach to PRO development; flexibility and judgment needed to meet practical demands

Well-defined and Reliable The tool adequately measures the concept of interest in the context or clinical setting of interest To assess this, we review the tool’s measurement properties: Content validity Construct validity Reliability Ability to detect change Information to support interpretation of change

Development Process for a Clinical Outcome Assessment

Evaluation of Measurement Properties Focus on basic analysis and build up the evidence methodically using a systematic approach The quantitative/psychometric evidence described in the next few slides should be gathered in a sample of patients with characteristics consistent with the targeted patient population expected in trials and targeted context of use

Evaluation of Measurement Properties (continued) What quantitative evidence that we would like to see  Item descriptive statistics including frequency distributions of both individual items and overall score, floor and ceiling effects, and percentage of missing responses Inter-item correlation and dimensionality analysis (e.g., factor analysis or principal components analysis, evaluation of conceptual framework). Item inclusion and reduction decisions, identification of subscales (if any), and modification to conceptual framework

Evaluation of Measurement Properties (continued) What quantitative evidence and summaries that we would like to see Preliminary scoring algorithm including how missing data will be handled Reliability (e.g., Test-retest reliability, internal consistency, inter-rater reliability where applicable) Construct Validity Convergent and discriminant validity (e.g., correlation with other instruments assessing similar concepts) Known groups validity methods (e.g., difference in scores between subgroups of subjects with known status) Final instrument, conceptual framework, and scoring algorithm including the handling of missing data

Last but not the Least: Thinking about Meaningful Change How much change is meaningful? Anchor-based method Cumulative distribution function

Other Considerations COA ≠ Study Endpoint Study endpoint should be constructed, from the COA, in the way that can be used to clearly describe the treatment benefit Endpoint hierarchy

Conclusion A COA is evaluated in its totality in the context of use Content validity Other measurement properties The interpretation of the score and what is clinically meaningful change Early in the drug development and prior to confirmatory studies Plan ahead the study endpoint and the endpoint hierarchy

Lisa Kammerman Risks/ramifications of deviating from ideal setting Do risks/ramifications depend on position in endpoint hierarchy

Risks and ramifications when deviating from the ideal setting Lisa A. Kammerman, Ph.D.

What is the ideal setting? (Recap from Dr. Chen’s talk) Goal: A COA that is adequate for use in a clinical trial How? Evaluate psychometric properties of COA When? Early in medical product development program Prior to confirmatory studies What type of setting? Non-interventional observational stand-alone validation study Phase 2 Study Post hoc secondary analysis from earlier study

What is a less than ideal setting? Sometimes, plans don’t go as expected – bad luck You started early in your development program, but Results of psychometric evaluations weren’t great More time needed to revise instrument and retest Multi-cultural translation work doesn’t go well Result: May need to do more work during the confirmatory study

What is a less than ideal setting? Sometimes, you’re working with a rare disease You may not be able to start early; often there’s only 1 study May need to create COAs based on expert and patient input only May not have the time or capability to do the recommended evaluations Result: May need to do more work during the confirmatory study

What is a less than ideal setting? Sometimes, you started too late You didn’t allow enough time May have some preliminary results But results are inconclusive or not comprehensive in order to conclude the COA is appropriate for use Result: May need to do more work during the confirmatory study

What work can you do during a confirmatory study? Create a substudy to evaluate psychometric properties Use a subset of data from a confirmatory study Perhaps an interim analysis from an ongoing study Perhaps a subset from a completed study Evaluate psychometric properties from subset Use evaluation to update instrument for use at the final analysis Questions What is the size of the substudy Who is included in the substudy Does the final analysis include all subjects

Risks of deviation from ideal setting: Psychometric evaluation during the confirmatory study Type of endpoint Primary Secondary, intended for labeling Exploratory Reason Rare disease Started early, but unforeseen problems Started late If using an interim analysis, Will results be highly data dependent, and can’t generalize beyond study Risks of unblinding study

Dennis Revicki Who takes on risk in each case? Sponsors? The FDA? The general public?

WHO TAKES ON RISK IN EACH CASE? SPONSORS? FDA? GENERAL PUBLIC? Dennis Revicki, PhD Outcomes Research Evidera, Bethesda, MD, USA

Two main possible approaches Risks and Implications for Deviating from Ideal Approach to Evaluating Psychometric Properties of COAs Sometimes clinical development programs require psychometric evaluation of clinical outcome assessment (COAs) in Phase III clinical trials Attributable to timing of COA development and schedule for conducting Phase III clinical trials Tension between systematic development of new COAs and interest in advancing clinical development program (time is money) Two main possible approaches Design and conduct parallel stand-alone psychometric evaluation study Planned psychometric sub-study and blinded analysis of clinical trial data

Psychometric study may demonstrate inadequate psychometric properties Risks Associated with Parallel Stand-alone Psychometric Evaluation Study: Sponsor Risks Recruiting patients for psychometric study may compete with recruiting for clinical trial Psychometric study may demonstrate inadequate psychometric properties Unless sufficient changes in clinical status are observed, may not be able to identify thresholds for interpreting change (i.e., responder definitions) Psychometric study may not be completed by clinical trial data lock

Risks Associated with Psychometric Sub-study: Sponsor Risks Administrative issues and difficulty in maintaining fidelity of psychometric sub-study Resistance from clinical investigators and research staff Risk that psychometric sub-study sample differs in terms of demographic or clinical characteristics from Phase III clinical trial sample Challenges in maintaining mask to treatment group assignment for the analysts conducting the psychometric analyses Psychometric sub-study may demonstrate inadequate psychometric properties

Risks Associated with Phase III Clinical Trial: Sponsor Risks Conducting and maintaining fidelity of psychometric sub-study may impact conduct of the clinical trial Reduction in clinical trial sample size (assuming sub-study patients not included in efficacy analyses) May be minimized by increasing overall sample size to maintain statistical power for efficacy analyses Increase in clinical trial expenditures Can psychometric sub-study data be included in the clinical trial efficacy analyses? Regulatory bodies recommend not including the sub-study data May be possible to include these data in a sensitivity analysis

Risks Associated with Phase III Clinical Trial: COA is Primary or Secondary Endpoint Increased risk associated with taking the psychometric sub-study approach Risk that psychometric analyses demonstrate that the COA does not have adequate measurement properties (reliability, validity, responsiveness) May find that estimated responder definition criteria is not demonstrated and/or require larger sample sizes May be unknown whether lack of responsiveness is attributable to treatment or the COA Basically proceeding with Phase III clinical trial with a COA endpoint with unknown psychometric characteristics COA designated as primary endpoint more risky than COA designated as secondary endpoint Not advisable to have COA with unknown psychometric qualities as primary endpoint

Risk to pharmaceutical or biotech sponsor Overall Risk of These Approaches to Psychometric Evaluation During Phase III Risk to pharmaceutical or biotech sponsor Risk to regulatory agencies (Food and Drug Administration, European Medicines Agency, others) Risk to study participants and the general public

Risk to Pharmaceutical or Biotech Sponsor Some risk to sponsors in conducting stand-alone psychometric evaluation studies for new COAs Substantial risk to sponsors associated with psychometric sub-study in Phase III clinical trials Risk is reduced if there is some evidence (even from small studies) supporting psychometric properties of COA (dimensionality, reliability, validity) and content validity Depending on findings of the psychometric analyses, potential for delay in clinical development program

Risk to Regulatory Agencies Little overall risk to regulatory agencies Regulatory agencies will not have sufficient evidence on COA to make a confident decision about efficacy of treatment May come under criticism for delaying clinical development programs by recommending more ideal COA development and psychometric evaluation

Risk to Study Participants and General Public Study participants may be exposed to adverse effects of treatment unnecessarily in clinical trial with inadequate COA endpoints For general public and health care system, may delay in gaining access to potentially effective treatments

Part Two: Practical Implications Panelists will provide 2 minute introduction to each of four topics, followed by audience questions and panel discussion Ideal clinical trial scenario (WHC) Using single clinical study to both develop instrument and demonstrate efficacy of medical product (LK) Issues with using an interim analysis to evaluate measurement properties (LK) Masking psychometricians to treatment group (DR)

Practical implications: Ideal clinical trial scenario Wen-Hung Chen, PhD Clinical Outcome Assessment Staff Office of New Drugs (OND) Center for Drug Evaluation and Research (CDER) Food and Drug Administration (FDA)

The Setting to Evaluate the Measurement Properties of a COA An important component of patient-focused drug development is describing the patient’s perspective of treatment benefit in labeling based on data from patient-focused COAs. Therefore, early in product development, we encourage sponsors to consider incorporating well- defined and reliable patient-focused COAs as efficacy endpoints in clinical trials, when appropriate, and to discuss those measures with the FDA in advance of confirmatory trials. Non-interventional observational stand-alone validation study Phase 2 Study Post-hoc secondary analysis from earlier study

Advantages and Disadvantages of Conducting Psychometric Analyses Using Clinical Trial Data* Ideally using phase II rather than phase III clinical trial data Ideal situation is when some subjects are improving, some subjects remain the same, and some subjects are worsening in clinical status Problem situation #1: Active treatment is highly effective and there is only a small placebo group May inflate estimates of responder definition and responders Problem situation #2: Active treatment is not very effective and small sample size May attenuate estimates of responder definition and responders * Slide courtesy of Dr. Dennis Revicki

Practical implications: Issues with using an interim analysis to evaluate measurement properties Lisa A. Kammerman, Ph.D.

Using a single clinical study to develop instrument and assess efficacy in that clinical study Instrument will be highly dependent on data collected in the study Certain characteristics will be unique to the study, for example: Enrollment criteria Study sites and investigators Item selection and deletion dependent on population How can we be assured that we are measuring what we believe we are measuring without verification from an independent set of data. If we conclude there’s a difference between treatment groups, how do we interpret the difference when we have only the single study. If we want to use the instrument in a new study, can we use the same instrument? Trial characteristics may be different Study population may be different

Using an interim analysis to evaluate measurement properties Need processes to preserve integrity of study Need to consider timing of analysis A study of long duration that is slowly enrolling (e.g., a rare disease) A study of long duration that enrolls rapidly Etc Need sufficient number of subjects with endpoint of interest If you conclude the instrument needs modification … How will you do that How many subjects remain to be enrolled How many subjects currently enrolled Scoring algorithm likely to be data dependent on interim analysis data set Implications for overall efficacy analysis Implications for generalizing to other studies that use instrument If endpoint is not primary, then wait until end of the study

Risks of deviation from ideal setting: Psychometric evaluation during the confirmatory study Type of endpoint Primary Secondary, intended for labeling Exploratory Reason Rare disease Started early, but unforeseen problems Started late If using an interim analysis, Will results be highly data dependent, and can’t generalize beyond study Risks of unblinding study

MASKING PSYCHOMETRICIANS TO TREATMENT GROUPS FOR PSYCHOMETRIC ANALYSES Dennis A. Revicki, PhD Outcomes Research, Evidera, Bethesda, MD, USA

Masking Psychometricians to Treatment for Psychometric Analyses of Clinical Trial Data Essential to mask psychometricians to treatment group for psychometric analyses of phase II or III clinical trial data Decisions about item retention and deletion needs to be made without reference to treatment group membership Usual approach is to provide psychometrician data files without any reference to treatment group status Contract research organization vendors In-house biostaticians (although often without psychometric expertise) No adverse event data is provided, which could potentially be used to identify treatment group Provide psychometrician with only those data files necessary for conducting the planned psychometric analyses

Masked to treatment group membership Masking Psychometricians: Independent Psychometric Monitoring Committee Set up like an independent data monitoring committee for a clinical trial Includes psychometricians, clinicians and biostatisticians not directly involved with clinical trial Masked to treatment group membership Reviews psychometric analyses and makes independent decisions about item retention and deletion, and responder definitions

Part Three: Q&A Comments 2017 DIA Annual Meeting Session Please type and submit your questions Include the presenter’s name 2017 DIA Annual Meeting Session Developing and Evaluating PRO Instruments in Clinical Trials Overview and Importance of Key Psychometric Characteristics Statistical Issues for Applications using Registration Trials Risks and Advantages for Applications using Registration Trials