NARRTC Annual Meeting, Apr. 27, 2011 Psychometric Properties of the Level of Knowledge Use Survey (LOKUS) Tool  (Shreya Telang, Machiko Tomita, Vathsala Stone) Presenter: Vathsala I. Stone vstone@buffalo.edu University at Buffalo/Center on Knowledge Translation for Technology Transfer http://kt4tt.buffalo.edu NARRTC Annual Meeting, Apr. 27, 2011

Background Overall Context: Knowledge Translation (KT) (CIHR, 2009; Sudsawad, 2007). Knowledge (Evidence)  Practice  Impact on beneficiaries Specific Context: Technology based Research - Sub-optimal level of demonstrated impact from R&D investment KT4TT Center: Develop KT models, methods & metrics for technology based R&D Conducting “end-of-grant” KT interventions in 3 technology areas – Augmentative and Alternative Communication (AAC), Environmental Access & Wheeled Mobility. LOKUS measures Knowledge Use as indicator of intervention effectiveness.

Background (contd.) KT Intervention Project in AAC Selected AAC Study (new knowledge): End-of-grant; NIDRR funded; Innovative Intervention strategy: Contextualized Knowledge Package (CKP) + Training (webinars) + Technical Assistance Effect on 6 types of Knowledge Users (stakeholders): Manufacturers; Clinicians; Transition Brokers; Researchers; Policy makers; Consumers with disabilities

Level of Knowledge Use Survey (LOKUS) Level of Knowledge Use Survey (LOKUS) tool seeks to: identify the level of use (and the corresponding categories) of new knowledge generated by technology-based research (attained) by stakeholders (users) of that knowledge. Developed by the Knowledge Translation on Technology Transfer (KT4TT) Center Based on Hall et al (2006) Psychometric study of LOKUS conducted by Shreya Telang (2011).

Development of LOKUS: Content Validation Tool based on Hall’s (2006) framework - Levels and Categories - educational setting. Expert testing of items – 3 KT scholars and 4 Technology Transfer (TT) experts. Tool showed 100% Face Validity. Tool altered: added/changed levels, eliminated irrelevant categories; closer to KT4TT context. Web version pilot tested by 6 individual stakeholders. Resulted in LOKUS, subject of this study.

Framework for Item distribution in LOKUS (Based on Halls et al, 2006)   CATEGORIES Being Aware Getting Information Sharing Assessing Planning Implementing LEVELS 0 - NON-AWARENESS 1- AWARENESS: 2 – ORIENTATION:  x 3 – PREPARATION: x 4 - INITIAL USE: 5 - ROUTINE USE: x  6 – EXPANSION: 7 –COLLABORATION: 8 – INTEGRATION: 9 – MODIFICATION:

Focus of the Psychometric Study Reliability Test-Retest Alternate Assessment method (Web based Vs. Paper-and- Pencil method) Responsiveness to change Ability to detect changes in knowledge use over time; Examination of Developmental nature of levels and categories.

Method: Research Design Study (NK) in LOKUS T1 (Baseline) T2 (at 1 week) Intervention (CKP) T3 (at 4 weeks) R Web based group A O X B C Paper-and-pencil Group

Method: Participants Represent one of the 5 stakeholder types in the KT4TT Center intervention project – the “Clinicians”. Inclusion: College Students / faculty members from allied health disciplines (Occupational therapy, nursing…..) and clinicians experienced in AAC; 18 years or older; Exclusion: participation in similar survey/focus group within past 6 months. Sample Size: Based on power analysis. large effect size d =.94 (Colbert, 1977); N = 64 (needed); 72 recruited, pre-screened and randomly assigned to two groups; Final N = 69 (35, paper-and-pencil, 34 web based).

Method: Intervention LOKUS items asked about New Knowledge from 3 NIDRR funded published studies in AAC - Study A, Study B and Study C. To identify Responsiveness to changes, a simulated Intervention condition was introduced between T2 and T3 by providing a CKP on Study A (treatment) only to both groups. and no CKP on the other two studies (controls).

Method: Data Analysis Focus Data Analysis Equivalence of participant groups (WB and P&P) Demographics on age, gender, marital status, highest educational level, experience in AAC Independent t-tests and chi-square tests. Test-Retest Reliability LOKUS scores at T1 and T2, on levels and categories. T1 - T2 comparison using ICC (3,1) Compare Alternate Assessment methods LOKUS scores at T1 and T3, on levels and categories. Mann-Whitney U-tests, for difference between groups regarding (1) change (dichotomous) in levels and (2) number that changed their level status. Responsiveness to change LOKUS scores at T1 and T3, on levels. Identify significant change (i.e., # of levels changed >Std Error of Measurement); Compare Studies A , B and C. Developmental nature of levels Participant changes in level status. Developmental, if majority of changes move up a level.

Results: Demographics There was no significant difference between the two groups regarding: Age ( t = .757; p = .452); AAC experience (t = .880; p = .382); Gender (chi sq = 1.292; p = .256); Marital status (chi sq = 4.872; p = .301) and Education (chi sq = 1.003; p = .793)

Results: Test-Retest Reliability Levels Number who maintained Levels at T2 Categories 1 2 3 4 5 6 StudyA Corr. =1.0) 0- Non Awareness 50 n/a 1-Awareness 12 2-Orientation 7-Collaboration StudyB (Corr.=1.0) 49 7 10 3-Preparation 4- Initial Use Study C 58 8 Note: Identical responses at T1 & T2

Results: Responsiveness to Change Study (NK) Change from T1 to T3 T1 Mean T3 Mean Mean Diff. SD SEM f % Study A Total 66 95.7 0.43 1.88 1.45 1.36 0.71 P &P 33 100 Web 31 91.2 Study B 5 7.2 0.65 0.86 0.21 1.13 0.44 2 5.7 3 8.8 Study C 8 11.6 0.17 0.33 0.16 0.60 0.25 4 11.4 11.8

Results: Difference between the Methods Study (NK) Change between T1 and T3 Difference between Methods (p-level) f % Changed/Not Changed Number of levels Study A Total 66 95.7 0.038* 0.194 P &P 33 100 Web 31 91.2 Study B 5 7.2 0.147 0.125 2 5.7 3 8.8 Study C 8 11.6 0.961 0.100 4 11.4 11.8 Note: * Significant; p < 0.05

Number who moved up levels at T3 1 2 3 4 5 6 7 8 Tot 37 19 0- Non Awareness (n=97) 62 1-Awareness (n=45) - 14 2-Orientation (n=41) 3-Preparation (n=18) 4- Initial Use (n=3) 5-Routine Use (n=1) 6-Expansion (n=1) 7-Collaboration (n=0) 8- Integration (n=1)

Summary and Conclusions LOKUS demonstrated: Good Face Validity. Excellent Test-Retest Reliability for both levels & categories. Good responsiveness to detect change regarding use of New Knowledge. Developmental nature for lower levels. LOKUS is a usable tool at least for AAC field. Longitudinal study needed to extend conclusion for higher levels and for categories.

Key References CIHR. About knowledge translation. Retrieved October 25, 2009, from http://www.cihr- irsc.gc.ca/e/29418.html Hall, G.E., Dirksen, D.J., and George, A.A. (2006). Measuring Implementation in Schools: Levels of Use. Austin, TX: Southwest Educational Development Laboratory (SEDL). Lane, J.P., Stone, V.I., Bauer, S. M., Leahy, J.A., and Tomita, M.R. (2008). Center on Knowledge Translation for Technology Transfer. Proposal submitted to National Institute for Disability and Rehabilitation Research (NIDRR)’s Disability and Rehabilitation Research (DRRP) Program (84.133A-7). Sudsawad, P 2007. Knowledge Translation: Introduction to Models, Strategies, and Measures. Austin: Southwest Educational Development Laboratory, National Center for the Dissemination of Disability Research. (p.4; 21-22) Telang, S.R. Establishing Psychometric Properties of the Level of Knowledge Use Survey (LOKUS) Questionnaire for Knowledge Translation for Technology Transfer. (Unpublished Masters Thesis). University at Buffalo, State University of New York.

Acknowledgement This is a presentation based on Shreya Telang’s (2011) work for her masters’ thesis which was partially supported by the KT4TT Center under funding by the National Institute on Disability and Rehabilitation Research of the U.S. Department of Education, under grant number H133E030025. The opinions contained in this presentation are those of the grantee and do not necessarily reflect those of the U.S. Department of Education.

Contact: vstone@buffalo.edu Thank you!! Contact: vstone@buffalo.edu http://kt4tt.buffalo.edu Questions?

Appendix Study A Light, J. and Drager, K. (2007). AAC technologies for young children with complex communication needs. State of the science and future directions. Augmentative & Alternative Communication, 23 (3), 204-16. Study B Light, J., McNaughton, D., Weyer, M. & Karg, L.(2008). Evidence-based literacy instruction for individuals who requireAugmentative and Alternative Communication: A case study of a student with multiple disabilities. Semin Speech Lang, 29 (2), 120-132. Study C Quach, W.(2007). Facilitating children’s learning of Augmentative and Alternative Communication systems. Retrieved from Proquest Digital Dissertations. (AAT 3275080).

LOKUS and the LoU Scale: Differences    LoU Scale LOKUS  Purpose  To measure Use of Innovations To measure Use of Innovations Context Educational setting - part of Concerns Based Adoption Model (CBAM) Broader, social setting – focus on knowledge from technology based research Framework 8 Levels linked by decision points; 7 Categories in each level 10 Levels, with 3-6 Categories under levels Observation Method Systematic interviews Web based, branched items eliciting self reported responses - large scale survey capability