Translating New Knowledge from Technology Based Research Projects: Randomized Controlled Study of an Intervention Presenter: Vathsala I. Stone

Slides:



Advertisements
Similar presentations
LAO PDR Summary Findings from NOSPA Mission and Possible Next Steps.
Advertisements

Introduction to Monitoring and Evaluation
KT for TT – Ensuring Technology- based R&D matters to Stakeholders
Strategies for Implementing Outcomes in Practice Carolyn Baum, PhD, OTR, FAOTA.
Begin with Knowledge Translation; Have the End – Technology Transfer – in Mind Begin with Knowledge Translation; Have the End – Technology Transfer – in.
Contextualized Knowledge Translation Packages for Technology Transfer and Product Development ATIA Orlando, Florida January 2012 James A. Leahy Center.
Knowledge Translation Curriculum Module 1: An Introduction to KT Lesson 1 - Knowledge Translation: The Basics.
Achieving Knowledge Translation for Technology Transfer: Implications for Evaluation Presenter: Vathsala I. Stone University at Buffalo/
TTO Role in University / Corporate Partnership
How to Translate Knowledge in Three States: Discovery, Invention, Innovation Joseph P. Lane Center on Knowledge Translation for Technology Transfer University.
Getting from Knowledge to Action: Effectively communicating Research & Development value to multiple Stakeholder Groups. Joseph Lane & John Westbrook RESNA.
Innovation in Universal Design “Universal integration of research, education, innovation and enterprise at DIT GrangeGorman” Joseph P. Lane, University.
Industry – The missing link between S&T Policy and Societal Benefit. Joseph P. Lane Center on Knowledge Translation for Technology Transfer University.
Evaluation. Practical Evaluation Michael Quinn Patton.
From Theory to Practice: Operationalizing Knowledge Translation for Successful AT Commercialization Joseph P. Lane Center on Knowledge Translation for.
Evaluation and Policy in Transforming Nursing
Session Code: AAC-07 Translating New Knowledge from Technology Based Research Projects: A Randomized Controlled Study Presenter: Vathsala I. Stone Center.
Too expensive Too complicated Too time consuming.
1 Introduction to Evaluating the Minnesota Demonstration Program Paint Product Stewardship Initiative September 19, 2007 Seattle, WA Matt Keene, Evaluation.
Best Practices in Technology Transfer Jennifer L Flagg Center on Knowledge Translation for Technology Transfer University at Buffalo.
KT for AT: Knowledge Translation Tools for R&D Projects Joseph P. Lane Center on Knowledge Translation for Technology Transfer
Expanding Product Accessibility with Primary Market Research Techniques Jennifer L Flagg Center on Knowledge Translation for Technology Transfer, University.
Management of assessments and decision making: execution, facilitation, evaluation Mikko V. Pohjola, Nordem Oy (THL)
Knowledge Translation: Research Into Practice Ian D Graham, PhD FCAHS University of Ottawa Ottawa Hospital Research Institute August 9, 2012.
Integrating Knowledge Translation and Exchange into a grant Maureen Dobbins, RN, PhD SON, January 14, 2013.
Maria E. Fernandez, Ph.D. Associate Professor Health Promotion and Behavioral Sciences University of Texas, School of Public Health.
1 of 27 How to invest in Information for Development An Introduction Introduction This question is the focus of our examination of the information management.
Information commitments, evaluative standards and information searching strategies in web-based learning evnironments Ying-Tien Wu & Chin-Chung Tsai Institute.
Tools for Technical, Business & Consumer Analysis in AT Product Development: Expanding the Need to Knowledge Model Joseph P. Lane, Center on KT4TT University.
The 7th Campbell Colloquium May 14-16, 2007 London, UK Knowledge Translation and Disability and Rehabilitation Research _______________________________________________.
Managing & Communicating Knowledge in Three States Joseph P. Lane Center on Knowledge Translation for Technology Transfer School.
New England Vocational Rehabilitation Quality Assurance System
Analytic Tools: Ensuring industry relevance for university-based R&D projects intending transfer. Joseph P. Lane & James Condron Center on Knowledge Translation.
Bridging the Evidence Gap: Level Of Knowledge Use Survey - LOKUS as a Validated Instrument Joseph P. Lane Center on Knowledge Translation for Technology.
Bridging the Deliverable Gap: Improving Government’s approach to innovation intending social benefit. Joseph P. Lane Center on Knowledge Translation for.
Knowledge Translation Conference KT Solutions for Overcoming Barriers to Research Use Hosted by SEDL’s Center on Knowledge Translation for Disability and.
Knowledge Translation Conference KT Solutions for Overcoming Barriers to Research Use Hosted by SEDL’s Center on Knowledge Translation for Disability and.
The KT4TT Knowledge Base: Steps and Supporting Evidence to Improve Your Process! Webcast sponsored by SEDL September 29, 2010, 2:00 pm (Central) / 3:00.
Is One Minute Madness?? Joseph P. Lane Center on Knowledge Translation for Technology Transfer University at Buffalo.
Evaluation What is evaluation?
Stages of Research and Development
A Framework for Evaluating Knowledge Transfer and Exchange (KTE) in Public Health Paper Presented at the Annual Conference of the American Evaluation Association.
DARM 2013: Assessment and decision making
Presenter: Vathsala I. Stone
Four Models to Guide AT Projects Intending Innovative Technology Development Outcomes Joseph P. Lane Center on Knowledge Translation for Technology Transfer.
A prospective study of the translational process in the technology development and transfer projects of NIDILRR’s technology grantees: a qualitative study.
Center on Knowledge Translation for Technology Transfer
From Theory to Practice: Operationalizing Knowledge Translation for Successful AT Commercialization Joseph P. Lane Center on Knowledge Translation.
KT for AT: Knowledge Translation Tools for R&D Projects
Targeting Stakeholders and Tailoring Knowledge as Communication Strategies in Assistive Technology: Three Randomized Controlled Case Studies Presenter:
The Social Model for A/T Technology Transfer – AAATE 2010 “From Problem Identification to Social Validation: An Operational Model” Joseph P. Lane,
Bridging the Deliverable Gap: Improving Government’s approach to innovation intending social benefit. Joseph P. Lane Center on Knowledge Translation.
Translating New Knowledge from Technology Based Research Projects: an End-of-Grant Intervention Evaluation Study. Rationale and Methods Vathsala I. Stone.
Managing & Communicating Knowledge in Three States
A prospective study of the translational process in the technology development and transfer projects of NIDILRR’s technology grantees: a qualitative study.
Expanding Product Accessibility with Primary Market Research Techniques Jennifer L Flagg Center on Knowledge Translation for Technology Transfer, University.
Four Models to Guide AT Projects Intending Innovative Technology Development Outcomes Joseph P. Lane Center on Knowledge Translation for Technology Transfer.
Tools for Technical, Business & Consumer Analysis in AT Product Development: Expanding the Need to Knowledge Model Joseph P. Lane, Center on KT4TT.
Disseminating, Tracking and Evaluating New Knowledge in P&O
Joseph Lane & John Westbrook
Industry – The missing link between S&T Policy and Societal Benefit.
NARRTC Annual Meeting, Apr. 27, 2011
AEA Annual Meeting , Nov , 2009 Achieving Knowledge Translation for Technology Transfer: Implications for Evaluation Presenter: Vathsala I. Stone.
RESNA 2018 Annual Conference
Knowledge Translation Across RERC Activities
Joseph P. Lane Center on Knowledge Translation for Technology Transfer
Reconciling Government Policies and Programs with Public Expectations: The Case of Innovation in AT Joseph P. Lane Center on Knowledge Translation.
Knowledge Utility results from Rigor in Methods & Relevance in Content
The Need to Knowledge (NtK) Model: Orienting Scholar “Technology Grantees” to Best Practices in Transfer & Commercialization Joseph P. Lane, Director.
Center on Knowledge Translation for Technology Transfer
Presentation transcript:

Translating New Knowledge from Technology Based Research Projects: Randomized Controlled Study of an Intervention Presenter: Vathsala I. Stone University at Buffalo Center on Knowledge Translation for Technology Transfer AEA Annual Meeting, Nov. 5, 2011

2 Background Context: Knowledge Translation for Technology Transfer (KT4TT) Knowledge Translation ( CIHR, 2004; 2005; 2009; Sudsawad, 2007)  Problem:  Under-utilized Research (Weiss, 1979) ;  Accountability of funded Research (GPRA; Wholey et al, 2004)  Proposed solution: Research-to-practice Technology Transfer (Lane, 2003) Technology based R&D  K Outputs  Market Outcomes (products & services)  Societal Impact (User Benefits) KT4TT: Links two processes to increase results (likelihood of intended impact).

3 Knowledge Output: Vocabulary and Symbol Sets for adult users of AAC (Bryen, 2008) Beneficiaries: C onsumers with complex communication needs Expected Outcomes: manufacturers – transform vocabulary for AAC machines ; clinicians - fit AAC for consumers brokers – facilitate use of AAC policy makers –regulate the use of AAC Researchers – advance the work. Impact: Improved function & quality of life for persons with disabilities. Key: Strategic Communication of New Knowledge to Stakeholders with potential to value and apply, to facilitate implementation and use to benefit society. KT4TT: Example related to Augmentative and Alternative Communication (AAC) technology

4 KT intervention studies: Purpose Problem: Sub-optimal demonstration of impact from R&D investment. Purpose: Develop and evaluate KT intervention strategies that are feasible for use by technology R&D projects and effective in increasing use of new knowledge by potential users (stakeholders). Utility: K producers (technology grantees) can document evidence of impact from their project outputs

5 Intervention Targeted Dissemination of K (TDK) Through stakeholder affiliated organizations  Value mapping (Rogers, 2004; Lane and Rogers, 2011) (K user expectations and values regarding research)  Recruitment Targeted and Tailored Dissemination of K (TTDK)  Relevant audience targeted ( as above)  Contextualized knowledge Packages (CKPs)  Formats of communication (accessible, usable)  Multi channel delivery – tailored webinar; tailored tech assistance offer

6 Primary Research Question Are there differences in effectiveness among the 3 methods of communication, (i.e., TTDK, TDK and Passive Diffusion), in terms of raising overall levels of K use by stakeholders?

7 Research Design for the KT Intervention EvaluationResearch Design for the KT Intervention Evaluation Baseline Assessment Intervention Delivery (4 Mo.) Follow/up Test 1 Intervention Delivery (4 Mo.) Follow/ up Test 2 R T 1 OX 1a OX 1b O R T 2 OX2X2 OO R COOO Where T1=group exposed to TTDK; T2=group exposed to TDK; C=Control group; O=Observation (via LOKUS); X1a and X1b are components of TTDK method; & X2= TDK method. Five stakeholder types

8 Instrument Level of Knowledge Use survey (LOKUS) Web based survey Development (Stone et al, in preparation )  Initial framework based on Hall, et al (2006);  Measures Awareness, Interest and Use  Current model: Levels, Dimensions and Activities Questions on findings from 3 Studies (2 distracters). Psychometrics (Tomita et al, in preparation)  Adequate content validity, exceptional test-retest reliability (1.0), strong convergence with a conventional pencil and paper survey, and solid construct validity to detect changes

9 Non-Awareness Awareness Interest Orientation (B, G, S, A, I)* Preparation (B, G, S, P, I) Intended Use Initial Use (G, A, I) Routine Use (B, A, P, I) Modified Use Collaboration (B, G, S, A, P, I) Expansion (B, G, S, A, P, I) Integration (B, G, A, I) Modification (B, G, A, I) *Activities: B: Being Aware, G: Getting Information, S: Sharing, A: Assessing, P: Planning, I: Implementing Conceptual Model of LOKUS

10 STUDY GROUP STAKEHOLDER TYPE T 1 (TTDK)T 2 (TDK)ControlTotal BROKER CLINICIAN MANUFACTURER RESEARCHER CONSUMER TOTAL * The 3 groups were equivalent in Demographic characteristics ; there were no significant differences in age, years of experience, gender, race/ethnicity, education and work status. Study Sample*

11 Results: Comparative Effectiveness of 3 methods KU Level Means for Study A* at Base, F/up 1, and F/up 2 (N=207) N Baseline Mean (S.D.) Follow/up 1 Mean (S.D.) Follow/up 2 Mean (S.D.) Difference χ² (p) Post-hoc test Z (p) T1 (TTDK) (.68) 1.79 (1.16) 1.69 (1.03) (<.001) Base vs F/up (<.001) Base vs F/up (<.001) T2 (TDK) (.77) 1.76 (1.19) 1.74 (1.16) (.001) Base vs F/up (.001) Base vs F/up (.001) Control (.97) 1.51 (1.05) 1.73 (1.22) (.039) *Bryen’s research Both TTDK and TDK moved up significantly in K Use levels from baseline. They differed from the Control group, but not between each other.

12 Comparative Effectiveness of Methods (Contd.) Mean Change in KU Level: Differences among Three Groups for Study A* (All; N=207 ) KU Level Change → T1(TTDK) Mean (S.D.) T2(TDK) Mean (S.D.) Control Mean (S.D.) Difference  2 (p) Baseline to F/up 1.57 (1.12).50 (1.17).13 (1.01)7.044 (.030) Baseline to F/up 2.47 (.82).47 (1.19).35 (1.19)2.371 (.306) F/up 1 to F/up (1.20)-.03 (.75).22 (1.13)3.443 (.179) *Bryen’s research K Use level changes were significantly different among the 3 groups from baseline to Follow/up 1.

13 RESULTS (contd) Freq. comparisons between Baseline and F/Up1 reg. Non-Awareness/ Awareness + (McNemar Test ;N=207) T1Group- TTDK (N=72) Follow/UP 1 Non- Awareness Awareness + TotalExact Sig. (2-sided p=) Baseline Non-Awareness Awareness+ 279 Total (100%)

14 T1 TTDK (Study-A: N=72) Follow/UP 1 Non-UseUseTotalExact Sig. (2-sided p=) Baseline Non-Use Use213 Total T2 TDK (Study-A:N=72) Follow/UP 1 Non-UseUseTotalExact Sig. (2-sided p=) Baseline Non-Use Use224 Total Control (Study-A:N=63) Follow/UP 1 Non-UseUseTotalExact Sig. (2-sided p=) Baseline Non-Use Use437 Total56763 Frequency Comparisons between Baseline & F/Up1 reg. Non-Use/Use (McNemar Test: N=207)

15 Summary of Results: Comparative effectiveness- TTDK, TDK and Passive Diffusion 1.TTDK and TDK were effective in terms of change in level of K use. (Table 3) 2.Both TTDK and TDK were effective in raising K use level from Non-Awareness to Awareness and beyond; as well as from Non- Use to Use. Cell frequencies and exact levels of significance suggest TTDK and TDK were more effective in terms of raising awareness than in terms of moving non-users to use. 3.TTDK and TDK were more effective than passive diffusion method (control) from Baseline to Follow/up 1, but neither between Follow/up 1 and Follow/up 2, nor between baseline to Follow/up 2.

16 Conclusions Conclusions are tentative; replication RCTs are underway. 1.Targeting stakeholders for dissemination (common component of TTDK and TDK) is an effective way to raise K use; although Tailoring did not add to KT effectiveness. 2.Within TTDK, the tailored CKP was effective (intervention between baseline and Follow/up 1); however, the tailored webcast was not(intervention between Follow/ups 1 and 2). 3.Both TTDK and TDK were more effective in moving stakeholders beyond non-awareness than in moving non-users to use. (Approx. 30% Vs. 15%) You can lead horses to water but the horse decides to drink!

17 Discussion Effects of CKP vs. webcast/Tech assistance: Did the order of intervention play a role? Did the duration of intervention play a role? Nevertheless, the main results are not surprising: In End-of-Grant/Integrated KT, investigator generates K assuming audience have needs for the K; solution is to match the problem (find audience) to fit the capabilities of K. The opposite is argued in the Prior-to-grant KT approach proposed in the NtK model (Lane & Flagg, 2010).  Need should be validated prior to initiating any technology based R&D project (Based on Project’s TT experience)  Future RCTs to test this may shed further light.

18 Evaluation Quality

19 This is a presentation of the KT4TT Center which is funded by the National Institute on Disability and Rehabilitation Research of the U.S. Department of Education, under grant number H133A The opinions contained in this presentation are those of the grantee and do not necessarily reflect those of the U.S. Department of Education. We also acknowledge collaboration and expert input from the RERC on Communication Enhancement during the implementation phase of the study. Acknowledgement

20 References 1.Bozeman, B., & Rogers, J. D. (2002). A churn model of scientific knowledge value: Internet researchers as a knowledge value collective, Research Policy, 31, Bryen D. N. (2008). Vocabulary to Support Socially-Valued Adult Roles. Augmentative and alternative Communication, Vol. 24, No. 4, Pages CIHR. About knowledge translation. Retrieved October 25, 2009, from Graham, I.D., Logan, J., Harrison, M.B., Straus, S.E., Tetroe, J., Caswell, W., & Robinson, N. (2006). Lost in translation: time for a map? The Journal of Continuing Education in the Health Professions, 26(1), Hall, G.E., Dirksen, D.J., and George, A.A. (2006). Measuring Implementation in Schools: Levels of Use. Austin, TX: Southwest Educational Development Laboratory (SEDL). 6.Lane JP (editor), “The science and practice of technology transfer: implications for the field of technology transfer,” Journal of Technology Transfer: 28, 3/4, Lane, JP and Flagg JL(2010). Translating three states of knowledge--discovery, invention, and innovation, Implementation Science 2010, 5:9. 8.Lane, JP and Rogers JD, Engaging national organizations for knowledge translation: comparative case studies in knowledge value mapping, Implementation Science 2011, 6: Rogers, J.D. (2000). Theoretical consideration of collaboration in scientific research. In J.S. Hauger and C.McEnaney (Eds.), Strategies for competitiveness in Academic Research (Chapter 6). 10.Stone VI and Colleagues. Development of LOKUS. (Manuscript in preparation for submission to Implementation Science) 11.Sudsawad, P (2007). Knowledge Translation: Introduction to Models, Strategies, and Measures. Austin: Southwest Educational Development Laboratory, National Center for the Dissemination of Disability Research. (p.4; 21-22) 12.Tomita, MR, Stone VI and Telang SR. Psychometric Properties of LOKUS. (Manuscript in preparation for submission to Implementation Science) 13. Wholey J S., Hatry H P., and Newcomer, K E (eds.) (2004). Handbook of Practical Program Evaluation, San Francisco: Jossey-Bass.

21 Questions? Thank you!

22 Need to Knowledge (NtK) Model (Lane and Flagg, 2010) A KT Framework for Technology Based Innovations Need & Envisioned Solution RDP Impact on Beneficiary KtA 3 processes; 3 states of K; 3 outputs Introduces Prior-to-grant KT