D&I Grant Funding: Tips for Success and Possible Futures Russell E. Glasgow, PhD University of Colorado School of Medicine Department of Family Medicine Adult and Child Consortium for Health Outcomes, Research and Delivery Science (ACCORDS) www.ucdenver.edu/accords/implementation http://cufamilymedicine.org/evaluation_hub/

Thank You to: Dr. Paul Estabrooks and planning committee Drs. Ross Brownson and Enola Proctor My CU School of Medicine D&I program colleagues Faculty and coordinators of prior national and CU SOM D&I training and mentoring programs

Overview Rationale and Need for D&I Research Grant Funding Tips – primarily for NIH, but many generalizable to the IDeA CTR Pilot & Scholar program applications General D&I specific The future and final thoughts Resources Q&A Is this what you meant by fonts the same level, Russ? And you want the space between The future… and Resources?

Bench to Bookshelf

Research to Pipeline Green, LW et al. Diffusion theory and knowledge dissemination. Ann Rev Public Health 2009;30:151-74

Implementation Science Implementation science is the study of methods to promote the integration of research findings and evidence into healthcare policy and practice. Dissemination science is the study of methods of distribution of information and intervention materials to a specific public health or clinical practice audience.

When are you ready to study D&I? Adapted from Figure 11.1 Implementation and dissemination of prevention programs. (Brownson et al. Dissemination and Implementation Research in Health, 2012)

Effectiveness/Implementation Hybrids Curran, et al. Medical Care, 2012

Impact, Communication, Credibility Specific aims…aims…aims…it’s all about aims and delivering on aims Have aims read by multiple experts in other fields and non-experts Why will this make a difference? Why is this grant compelling? Demonstrate succinctly your and your team’s capacity

Video on “Importance of Specific Aims” Dr. Ross Brownson, Washington University https://youtube/twKT2CL7Z6o

Aims…good, bad and ugly Poor We will improve depression by: using a well known cognitive behavior therapy program X testing it in a controlled study analyzing which patients benefit the most disseminating the results

Aims…good, bad and ugly (cont’d) Better Refine existing EBP X based on Theory Y guided by prior findings, stakeholder engagement, and pilot testing of implementation strategy Z Reduce depression by at least 25% among diverse community health center patients compared with a minimal intervention control in a cluster randomized pragmatic trial Study the following implementation outcomes based on the RE-AIM model for evaluation and a mixed methods approach: reach, effectiveness; and implementation Advance D&I science by testing the generalizability of this EBP in diverse, low resource settings and patients, and studying contextual factors predicted to be related to implementation outcomes

Question What sections / components of grants do you think are weighted most heavily in review?

Methods – Key Issues (most heavily weighted) Congruence across sections - can map from theory to intervention to measures to analyses to impact to dissemination Select and justify a theory or conceptual model Consider relevance of theory and fit to this situation Consider pros and cons of adapting theory or combining theories

www.dissemination-implementation.org www.re-aim.org

Measures and Analyses Linked and clear rationale for hypotheses Are reliable and valid in this population Address potential downsides, unanticipated consequences, attrition Are pragmatic – feasible, brief, sensitive to change, not perfect Grid Enable Measures? Glasgow RE and Riley T (2013). Pragmatic Measures: What They Are and Why We Need Them. American Journal of Preventive Medicine, August;45(2):237-43. Grid Enabled Measures..... https://www.gem-beta.org/Public/wsmeasures.aspx?cat=8&aid=1&wid=1

Show your work & specify WHY it’s important (Personal style) Do not hide tough decisions (Briefly) discuss choices – show you are aware of alternatives State succinctly how this is innovative and why this team in this setting at this point in time (write reviewers evaluation for them)  How does this project: advance science, and impact public health (or some other societal or funding agency) goal?

Talk with project officers and “fit” your project to agency, key need or priority Project Officer(s) plural – different agencies and possible funders

Team Science Days of one investigator – unless descendent of Thomas Jefferson – are over Demonstrate you have all the relevant expertise, e.g., economics, biostatistics, maybe demography, qualitative expertise, etc. on your team Show how you have or can work together…if/how you are using team science for “real transdisciplinary collaboration” https://www.teamsciencetoolkit.cancer.gov/

Somewhat Specific to D&I Grants Differentiate between intervention and implementation strategy Example: intervention may be cognitive behavior therapy. IT should be evidence- based (separate and detailed discussion) Implementation strategy may be facilitator training; or delivery via internet Implementation strategy should have some evidence but can be innovative, if justified and “fits” Sa?

Context is King Demonstrate you are aware of both the general and relevant – usually multi-level contextual factors (social ecologic model) Discuss how your program or policy or study addresses or fits well with local context Ask yourself, is this program, policy or finding likely to be generalizable or fit different contexts Feldstein A and Glasgow RE. Pragmatic Robust Implementation and Sustainability Framework. Joint Commission Journal on Quality and Safety, 34(4):228-243.

Context Reporting Recommendations* Important to report conditions under which program was delivered. To what extent is the program generalizable: To similar settings? To different settings? Goal – to what intervention do you compare it (real world alternative)? * Standards for Reporting Implementation Studies(StaRI) Statement. Hilary Pinnock, Melanie Barwick,Christopher R Carpenter, et al. BMJ 2017;356:i6795 http://dx.doi.org/10.1136/bmj.i6795

Ultimate Use Question: “What program/policy components are most effective for producing what outcomes for which populations/recipients when implemented by what type of persons under what conditions, with how many resources and how/why do these results come about?”

Stakeholder Engagement In D&I, this is multi-level…at least 3 levels System or even national organization Local system or organization, e.g., school, YMCA, voluntary health agency, extension service Supervisor or local decision-makers Delivery staff or agents Recipients, potential beneficiaries or citizens Be SERIOUS about engagement – not lip service Usually requires ongoing relationship, not ‘helicopter research’ Advisory boards, PBRNs, etc. Should have already engaged stakeholders

Dissemination Plan Do NOT say will ‘do publications and scientific presentations’ (at least as primary strategy) Think creatively – how will you reach key target audience? Go where they are – just because you build it, they will NOT come Provide training, facilitation, resources and ongoing contact and FAQs – not one-time brief presentation

The Future???

Need for Rigor, Relevance and Transparency New NIH guidelines on rigor and transparency – to address failure to replicate and data-sharing issues Personal ‘wish’ and need: Combining issue of transparency, context reporting, and generalizability… “Expanded-CONSORT” References and link

Expanded CONSORT Figure Total Number Potential Settings Settings Eligible (n and %) Settings and Agents Who Participate(n and %) Settings and Agents Who Decline(n and %) Other (n and %) Excluded by Investigator (n, %, and reasons) Total Potential Participants (n) Individuals Eligible (n and %) Individuals Enroll (n and %) Individuals Decline (n, %, and reasons) Not Contacted/Other (n and %) Excluded by Investigator (n, %, and reasons) ADOPTION REACH RE-AIM Issue Extent Tx Delivered by Different Agents as in Protocol Adaptations and Reasons Component A = XX% Component B = YY% etc. IMPLEMENTATION Complete Tx (n and %) Amount of Change (by Condition) Drop Out of Tx (n, %, and reasons) Amount of Change (by Condition) EFFECTIVENESS Present at Follow-up (n and %) and Amount of Change or Relapse (By Condition) Lost to Follow-up (n, %, and reasons) Amount of Change or Relapse (By Condition) Setting in which Program is Continued and/or Modified After Research is Over (n, %, and reasons) Settings in which Program Adapted (n, %, and reasons) MAINTENANCE Settings in which Program Discontinued (n, %, and reasons)

Feasible and Pragmatic (Russ’s Dream) How fits with local settings and resources, priorities Estimated costs – time at minimum; replication costs; cost-effectiveness How pragmatic is your project along different PRECIS dimensions for study design – Pragmatic Explanatory Continuum Indicator Summary (PRECIS-2)

https://www.precis-2.org/ We’ll have to fix these 2 slides. RG- OK- do not look bad to me though.  The font is still a bit hard to read…if you have a clearer image, we can use that. https://www.precis-2.org/

RE-AIM Precision (Personalized) Medicine Questions Determine What percent and types of patients are Reached? For whom is the intervention Effective in improving what outcomes, with what unanticipated consequences? In what percent and types of settings and is this approached Adopted? How consistently are different parts of it Implemented at what cost to different parties? And how well are the intervention components and their effects Maintained? Gaglio B, Glasgow RE. Evaluation approaches…In:Brownson R, Colditz G, Procter E, (Eds). Dissemination and implementation research in health: Translating science to practice. New York: Oxford University Press; 2012, pp 327-56. NEW EDITION DUE OUT DECEMBER

Personal Advice to Mentees (most important) Develop a thick skin Persist – get back up and try again It always takes longer than you think – budgets, especially contracts and multi-site studies, colleagues delayed in getting you their sections Remember, it is not a fair world… “But if you try sometimes, you get what you need” ~Mick Jagger

First, Last and Always… Check with your potential program officer(s)

Questions/Comments??? I’m all ears…

Key Resources- see handouts U Col. SOM D&I website www.ucdenver.edu/accords/implementation 10 Tips to Getting Funded. Proctor et al- see above Ross Brownson –Funding in D&I Research-see above Examples of successful D&I grants UNC: https://impsci.tracs.unc.edu/index.php/get-funded/sample-grants NCI: https://cancercontrol.cancer.gov/IS/pdfs/DandI-PAR-Grant FundedContentAnalysis.pdf

Key Differences between Traditional Randomized Control Trials (RCTs) and Pragmatic Controlled Trials (PCTs) A traditional RCT tests a hypothesis under ideal conditions A PCT compares treatments under everyday clinical conditions GOALS To determine causes and effects of treatment To improve practice and inform clinical & policy decisions DESIGN Tests the intervention against placebo using rigid study protocols & minimal variation *Tests two or more real-world treatments using flexible protocols & local customization* PARTICIPANTS Highly defined & carefully selected More representative because eligibility criteria are less strict MEASURES Require data collection outside routine clinical care Brief and designed so data can be easily collected in clinical settings RESULTS Rarely relevant to everyday practice Useful in everyday practice, especially clinical decision making Do you want this slide to be in a table similar to #20 in the OSU set?

Scientific and Societal Trends Behavioral or social science for its own sake decreasingly likely to be funded Much larger collaborations; big data Possibly more international collaborations Even more transdisciplinary – and partner with at least one discipline and data source that ends in ‘omics’