1. Research and Evaluation in Telehealth
Suparna Qanungo, PhD
Assistant Professor, College of Nursing
Director, Telehealth Research Program
Jillian Harvey, PhD
Assistant Professor, Healthcare Leadership & Management
Director, Division of Doctor of Health Administration

2. Key Steps to Develop a Telehealth Research Study
Idea generation
Partnership/collaboration
Research Methods and Design
Data Collection
Data Analysis
Evaluation of Outcomes
Dissemination

3. Idea Generation It all starts with a research or program-related question/idea
Is the telehealth research question/idea something I/others care about?
Will the telehealth research idea address a problem?
Will the telehealth research idea reach the intended population/community?
Is the telehealth research idea feasible within the given time frame and location?
Does the telehealth research question/idea lead into a testable hypothesis?
Can the telehealth research question/idea produce data/ results of the tested hypothesis that can be replicated?
Does the telehealth research idea have a clear purpose

4. Telehealth Idea is Here: What Do you Do Next?
Literature Search
What is out there?
How have others done similar work?
Where is the gap?
How can you bridge the gap
Partnership/Collaboration
Courtesy 123 RF.com
Sustainable Funding

5. Telehealth Idea is Here: What Do you Do Next?
Partnership/Collaboration
Strong organizational/community partners
Telehealth needs and interest match
Strong leadership and dedicated commitment
Key stakeholders and consumers engaged
Open to change in culture
Trusted team spirit
Evaluation of outcomes

6. Research Methods
Develop appropriate telehealth research protocol
Study Design (Qualitative, Quantitative, Mixed Method)
Participant Recruitment and Informed Consent
Data Collection and Analysis
Evaluation of Outcomes

7. Qualitative vs. Quantitative Study Design
ASPECT
QUALITATIVE
QUANTITATIVE
Focus of research
Quality (nature, essence)
Quantity (how much, how many)
Philosophical Roots
Phenomenology, symbolic interaction
Positivism, logical empiricism
Common Purpose
Understanding, interpretation, discover ideas, hypothesis generating
Prediction, control, confirmation, causal explanation, hypothesis testing
Approach
Observe and interpret
Measure and test
Design Characteristics
Flexible, free form, evolving, emergent
Predetermined, structured
Setting
Natural, familiar
Artificial, unfamiliar
Sample
Small, non-random, theoretical
Large, random, representative, generalizability
Researcher independence
Researcher is intimately involved, “Emic” (insider perspective)
Researcher uninvolved observer,
“Etic” (outsider perspective)
View of reality
Subjective
Objective
Data collection
Researcher as primary instrument, interviews, observation
Inanimate instruments (scales, tests, surveys, questionnaires)
Mode of analysis
Inductive (by researchers)
Deductive (by statistical methods)
Findings
Comprehensive, holistic
Precise, narrow, reductionist

8. Recruitment and Informed Consent
Traditional Paper Consent
Teleconsent
E-Consent
All Need IRB Approval…… Attend Telehealth Seminar Series April 18, 2018 at MUSC Center for Telehealth…. Considerations for IRB Approval

9. Teleconsent (Remote Recruitment)

10. Telehealth Use in Research

11. Examples of Telehealth Research at MUSC
Mealtime Interventions for Caregivers/Dementia Patients at Respite Care Centers
HIV and substance use prevention program in Latina adolescents
Telehealth research for prenatal care
Trauma-focused Cognitive Behavioral Therapy
Peer Navigator interventions for individuals with spinal cord injury
Changes in smoking behavior with e-Cigs
Tele-mental Health in PTSD
Tele-stroke
School-based Telehealth

12. Data Collection Techniques/Sources (Qualitative)
Observations (e.g. photovoice)
Documents
Artifacts
Interviews
Focus groups

13. Data Collection Techniques/Sources (Quantitative)
Primary and Secondary Data Sources
Closed/structured Questionnaires
Surveys
Instruments/Scales/Logs
Experiments
Comparative Methods

14. Telehealth Data Evaluation and Outcomes

15. Rigorous evaluations of telemedicine outcomes and cost are needed
“Telehealth research in real-world settings requires alternative research designs,
new research methods, and innovative analytic techniques that supplement
traditional randomized, controlled trials and should be supported with enhanced
funding and an expanded workforce”
Source: Tuckson, R.V., Edmunds, M., & Hodgkins, M.L. (2017). Telehealth. The New England Journal of Medicine. 377:16,

16. Telehealth Research
Literature indicates several telehealth programs have a sufficient evidence base1-4
Effectiveness for remote monitoring, communication, counseling for chronic conditions and psychotherapy support of behavioral interventions
Evidence that telehealth can improve access to care
Patient satisfaction and Provider technical satisfaction
Some evidence on improved efficiencies (reducing wait times, travel costs, transports)
Mixed results related to utilization/cost
More research is needed1-4
Implementation
Delivery and payment models
Specific interventions lack empirical evidence (Intensive Care, Maternal & Child Health, Triage for urgent care, primary care)
Financial and cost effectiveness

17. Telehealth Methodological Challenges
Small n across specific telehealth programs (and counties)
Matching issues
Apples to Apples Comparisons
Vast differences in care protocols & equipment across organizations
Differences in state level Medicaid and Parity coverage and reimbursement
New terminology: What defines a telehealth consult?
Traditional control variables may not fit
Two Institutions (e.g. jail & hospital)
Selection Bias
Time

18. Telehealth Data Challenges
Telehealth presents unique data challenges
Claims data (identifying claims and rurality restrictions)
Patient registries (missing data and sharing information across organizations)
Electronic Medical Records (pulling information, accuracy)

19. Research? Program Evaluation? Quality Improvement?5
Can be indistinguishable
Examine relationship between variables
Qualitative, Quantitative, and mixed-methods
Systematic
Research:
Seeks conclusions and/or building the scientific knowledge base (independent from routine care)
Rigid protocol
Generalizable
IRB: Often randomized or testing new intervention. Possible risk in participating
Evaluation:
Explores program, intervention, treatment, or policy
Facilitates decisions
Can be formative or summative
Participants may be selected by the program leadership
Seeks understanding of program objectives/outcomes, program characteristics, program cost/value, quality, fidelity, etc.
What?, So what?, Now what? (Patton, 2008)
May be context specific
QI:
Data guided activities to bring about change or improvements
Used for ongoing management of care and processes
Adaptive, iterative

20. Evaluation of Outcomes
Requires careful planning and complete data collection
Five key steps of outcomes research6
Researchable Question
Conceptual Model
Literature Review
Operationalize the Variables
Research Plan

21. National Quality Forum6: Domains & Subdomains of Telehealth Measurement
Examples
Access to Care
-Access for patient, family, and/or caregiver
-Access for care team
-Access to information
Financial Impact/Cost
-Financial impact to patient, family, and/or caregiver
-Financial impact to care team
-Financial impact to healthcare system or payer
-Financial impact to society
Experience
-Patient, family, and/or caregiver experience
-Care team member experience
-Community experience
Effectiveness
-System effectiveness
-Clinical effectiveness
-Operational effectiveness
-Technical effectiveness

22. Data Collection Plan Hypothesis/ Question
Metric Operational Definition
Collection Process
Data Source
Timing
Collected by:
Analysis & Notes
1. Readmissions will decline in telehealth communities, and this decline will be greater in telehealth communities than nationally.
30 day readm. Rate- all cause
Secondary
Dartmouth
Annual, Oct,
Jim
Difference-in-difference modeling to estimate changes in intervention communities versus nationally
2. How does the implementation of telehealth interventions impact Door to needle time?
Mean door-to-needle time in minutes from patient registration to receipt of tPA
REACH
REACH export
Monthly
Pat
Exclude: those who had a thrombectomy.
QA check on any DTN over 270 minutes

23. Dissemination
Dissemination: Purposeful distribution of information and intervention materials to a specific audience
US healthcare system has done a poor job of putting research findings into practice7
Innovations of unknown/poor evidence base are frequently spread8
Dissemination Plan: Develop at start of project
Who is the audience?
Study participants, funders, communities, policy makers,
How to reach the audience?
Different wording, formats & methods
Present!
Publish!
Outside of tele journals

24. Conclusion
Rigorous evaluations of telemedicine outcomes and cost are needed
There are many research methods that can be applied (selection needs to be driven by the research question)
The telehealth infrastructure in South Carolina provides a fantastic opportunity to explore telehealth models
Collaboration & planning are essential for telehealth research (e.g. clinicians, administrators, staff, researchers)
Statewide resources are available: Pilot grants, CTSA, especially for cross organization/discipline partnerships

25. References
1. Tuckson, R.V., Edmunds, M., & Hodgkins, M.L. (2017). Telehealth. The New England Journal of Medicine. 377:16, Ashwood, J.S., Mehrotra, A., Cowling, D., & Uscher-Pines. (2017). Direct-to-Consumer telehealth may increase access to care but does not decrease spending. Health Affairs. 36(3). 3.Edmunds, Margo; Tuckson, Reed; Lewis, Joy; Atchinson, Brian; Rheuban, Karen; Fanberg, Hank; Olinger, Lois; Rosati, Robert; Austein-Casnoff, Cheryl; Capistrant, Gary; and Thomas, Latoya (2017) "An Emergent Research and Policy Framework for Telehealth," eGEMs (Generating Evidence & Methods to improve patient outcomes): Vol. 5: Iss. 2, Article 1. DOI: Available at: 4. Kane, R.L. & Radosevich, D.V. (2011). Conducting health outcomes in research. Jones & Bartlett Learning: Sudbury, MA. 5. CHOP (2018). Quality Improvement vs. Research. Available at: 6.National Quality Forum. (2017). Creating a Framework to Support Measure Development for Telehealth. Available at: 7. NIH. (2018). National Information Center on Health Services Research and Health Care Technology Dearing, J.W., (2009). Applying diffusion of innovation theory to intervention development. Research on Social Work Practice, 19(5):