What Really Works Methods and Measures for Research & Evaluation Charlene C. Quinn

What Really Works: Methods Informal Systematic Review funded research –Search sites NIH RePORT, clinical trials.gov, HSRProj, Federal agencies other than NIH including AHRQ and the VA –Previous six years

What Really Works: Methods Informal Systematic Review funded research –486 projects identified –29 actual mobile health interventions –Types of mobile interventions varied

What Really Works: Methods Importance of informal search findings –What’s in the pipeline? –Where is the focus of non-industry funded research? –Who is doing the research, potential collaborators? –What are the interventions, what measurement tools, what research design?

What Really Works: Methods Mobile Health Projects –Funded pilots (3), state technology funds –Working with technology start-ups –RCT funded by industry partnership Technology start-up, major insurer, glucose meter company, wireless mobile phone company

What Really Works: Methods Designing the Clinical Trial –Randomized –Larger number study participants –One year intervention –Control group, stepped intervention (dose) –Enrolled providers and patients (cluster randomized trial) Community primary care providers

What Really Works: Measures Primary research outcome: mean change in HbA1c (a gold standard in diabetes research) Secondary outcomes to explain if change in HbA1c –Symptom measure (McColl) –Depression (PHQ-9) –Diabetes distress –Diabetes Stages of Change –Provider and patient satisfaction

What Really Works: Evaluation Published a research design article describing the Mobile Diabetes RCT Study results-primary research outcome in manuscript review Working on analyses secondary outcomes, including –Differences by age, race

What Really Works: Evaluation Intervention themes observed –Include expectations of persons with condition/disease –Feedback that is actionable –Continuous –Linked to health providers

What Really Works: Evaluation Challenges to operationalize evaluations –Gold standard for academic research is defined largely by NIH R01 standards Needed for FDA review? –IRB understanding of mobile health and protection of human subjects –Where data resides. Industry servers not covered by IRB protection. Integrating server data with research data –Collaborations mobile health industry, scientific community and payors-transparency

What Really Works Not only ask if mobile health works, ask how does it work If you build it, they will come Charlene C. Quinn –