1. What Really Works Methods and Measures for Research & Evaluation Charlene C. Quinn cquinn@epi.umaryland.edu 410.706.2406

2. What Really Works: Methods Informal Systematic Review funded research –Search sites NIH RePORT, clinical trials.gov, HSRProj, Federal agencies other than NIH including AHRQ and the VA –Previous six years

3. What Really Works: Methods Informal Systematic Review funded research –486 projects identified –29 actual mobile health interventions –Types of mobile interventions varied

4. What Really Works: Methods Importance of informal search findings –What’s in the pipeline? –Where is the focus of non-industry funded research? –Who is doing the research, potential collaborators? –What are the interventions, what measurement tools, what research design?

5. What Really Works: Methods Mobile Health Projects –Funded pilots (3), state technology funds –Working with technology start-ups –RCT funded by industry partnership Technology start-up, major insurer, glucose meter company, wireless mobile phone company

6. What Really Works: Methods Designing the Clinical Trial –Randomized –Larger number study participants –One year intervention –Control group, stepped intervention (dose) –Enrolled providers and patients (cluster randomized trial) Community primary care providers

7. What Really Works: Measures Primary research outcome: mean change in HbA1c (a gold standard in diabetes research) Secondary outcomes to explain if change in HbA1c –Symptom measure (McColl) –Depression (PHQ-9) –Diabetes distress –Diabetes Stages of Change –Provider and patient satisfaction

8. What Really Works: Evaluation Published a research design article describing the Mobile Diabetes RCT Study results-primary research outcome in manuscript review Working on analyses secondary outcomes, including –Differences by age, race

9. What Really Works: Evaluation Intervention themes observed –Include expectations of persons with condition/disease –Feedback that is actionable –Continuous –Linked to health providers

10. What Really Works: Evaluation Challenges to operationalize evaluations –Gold standard for academic research is defined largely by NIH R01 standards Needed for FDA review? –IRB understanding of mobile health and protection of human subjects –Where data resides. Industry servers not covered by IRB protection. Integrating server data with research data –Collaborations mobile health industry, scientific community and payors-transparency

11. What Really Works Not only ask if mobile health works, ask how does it work If you build it, they will come Charlene C. Quinn –cquinn@epi.umaryland.educquinn@epi.umaryland.edu –410.706.2406