1. PRO-CTCAE Face-To-Face Meeting #2 Advancing the Science of Adverse Symptom Monitoring in Cancer Treatment Trials Amy P. Abernethy, MD Task 6: Usability testing May 24, 2010 National Cancer Institute Rockville, MD

2. Aims To determine performance of the PRO CTCAE system in five domains: ease of learning efficiency of use memorability error frequency and severity subjective satisfaction

3. Background Purpose of usability testing = to determine how well users can learn and use a product, and how satisfied they are with that process. Definition of usability = the system’s capacity to allow users to carry out tasks safely, effectively, efficiently, and enjoyably Target users = patients with diverse backgrounds, educational levels, computer experience, and physical illness; clinical research staff

4. Background Differentiation between tasks: cognitive interviewing vs. usability testing – cognitive interviews = define verbiage and refine presentation FOCUS = REFINING QUESTIONS – usability testing = incorporate the refined questions/presentation, and focus on the technology/platform FOCUS = REFINING TECHNOLOGICAL INTERFACE FUNCTIONALITY & USABILITY

5. Design Participate in system design (Task 5) Setting = outpatient oncology clinics, inpatient oncology; spans academic and community Sites = MSKCC, MDACC, Duke, plus respective satellite clinics Create realistic scenarios that replicate future uses of the system; observe in situ inter- actions; identify areas for improvement

6. Users (participants) 2 iterations of testing in 2 groups of users: clinical research staff, patients 1.Clinical research staff (n=54) 9 nurses, 9 clinical research assistants, 9 physician investigators [x2] 2.Patients (n= 150) ≥10 over age 75, ≥10 with less than high school education, ≥10 non-white, ≥10 inpatients or severely ill (ECOG >2), ≥30 metastatic After 40, enrich in targeted manner

7. Methods 3 main components: 1.Observe users interacting with the system  think-aloud protocols, extensive field notes 2.Collect user feedback  semi-structured interviews, system usability scale, short surveys 3.Employ web analytics  tracking for different web pages, clickstream analysis

8. Creating PRO Form

9. Review Reports

10. Complete Forms (patients)

11. Sample Exit Interview Questions What was your impression of the way information is entered? – It was easy to do – It was somewhat easy – It was neither easy nor hard – It was not easy Did you think the size and color of the words made them easy to read? (circle one) Yes No Did you think it was easy to navigate through this site or find the information you needed? (circle one)Yes No If you were to describe this site to someone, what would you say?

12. System Usability Scale © Digital Equipment Corporation, 1986.

13. Benchmarking metrics total time required to use the PRO CTCAE system by new users to perform specified tasks aspects of the system that create unnecessary delays (e.g., web pages) memorability (after creating a new form in a specified way, can the user create their own without difficulty) order of pages viewed number of times the user backtracks number of missed or skipped items

14. Key personnel Amy Abernethy, Task Leader (Duke) Deborah Schrag, Deputy Task Leader (DFCI) Marwan Shouery, Deputy Task Leader (MSKCC) Andrea Denicoff (NCI, DCTD) Lori Minasian (NCI, DCP) Mike Montello (NCI, CTEP) Lori Hudson (Duke) Other Duke personnel (Mary Ruth Pupa, Paul Pooley, Robin Fowler, Danielle King, Jessica Ray)

15. Timeline Activity Month Protocol development 12/08-1/09 IRB submission 11/09-3/10 Training of research staff at each site 3/10-8/10 Participant recruitment and enrollment Start 5/10 Study conduct 5/10-10/10 Coding and data entry 10/10-11/10 Analysis 11/10-1/11

16. Deliverables Summary reports of usability testing Source transcripts of usability testing Documentation of resulting system modifications Refined PRO-CTCAE platform based on results of usability testing

17. Potential barriers Recruitment of sufficient sample Distribution of user types to represent population of potential users Logistical, e.g., impact on clinic flow Technical, e.g., technology requirements

18. Input is welcome… Do you have feedback on the design? Are there additional benchmarking items that should be analyzed? Can you point out other barriers that we should anticipate?