1. NARRTC Annual Meeting, Apr. 27, 2011
Psychometric Properties of the Level of Knowledge Use Survey (LOKUS) Tool  (Shreya Telang, Machiko Tomita, Vathsala Stone)
Presenter:
Vathsala I. Stone
University at Buffalo/Center on Knowledge Translation for Technology Transfer
NARRTC Annual Meeting, Apr. 27, 2011

2. Background Overall Context:
Knowledge Translation (KT) (CIHR, 2009; Sudsawad, 2007).
Knowledge (Evidence)  Practice  Impact on beneficiaries
Specific Context:
Technology based Research - Sub-optimal level of demonstrated impact
from R&D investment
KT4TT Center:
Develop KT models, methods & metrics for technology based R&D
Conducting “end-of-grant” KT interventions in 3 technology areas – Augmentative and Alternative Communication (AAC), Environmental Access & Wheeled Mobility.
LOKUS measures Knowledge Use as indicator of intervention effectiveness.

3. Background (contd.) KT Intervention Project in AAC
Selected AAC Study (new knowledge):
End-of-grant; NIDRR funded; Innovative
Intervention strategy:
Contextualized Knowledge Package (CKP) + Training (webinars) + Technical Assistance
Effect on 6 types of Knowledge Users (stakeholders):
Manufacturers; Clinicians; Transition Brokers; Researchers; Policy makers; Consumers with disabilities

4. Level of Knowledge Use Survey (LOKUS)
Level of Knowledge Use Survey (LOKUS) tool seeks to:
identify the level of use (and the corresponding categories)
of new knowledge generated by technology-based research
(attained) by stakeholders (users) of that knowledge.
Developed by the Knowledge Translation on Technology Transfer (KT4TT) Center
Based on Hall et al (2006)
Psychometric study of LOKUS conducted by Shreya Telang (2011).

5. Development of LOKUS: Content Validation
Tool based on Hall’s (2006) framework - Levels and Categories - educational setting.
Expert testing of items – 3 KT scholars and 4 Technology Transfer (TT) experts.
Tool showed 100% Face Validity.
Tool altered: added/changed levels, eliminated irrelevant categories; closer to KT4TT context.
Web version pilot tested by 6 individual stakeholders.
Resulted in LOKUS, subject of this study.

6. Framework for Item distribution in LOKUS (Based on Halls et al, 2006)
CATEGORIES
Being Aware
Getting Information
Sharing
Assessing
Planning
Implementing
LEVELS
0 - NON-AWARENESS
1- AWARENESS:
2 – ORIENTATION:  x
3 – PREPARATION: x
4 - INITIAL USE:
5 - ROUTINE USE: x 
6 – EXPANSION:
7 –COLLABORATION:
8 – INTEGRATION:
9 – MODIFICATION:

7. Focus of the Psychometric Study
Reliability
Test-Retest
Alternate Assessment method (Web based Vs. Paper-and- Pencil method)
Responsiveness to change
Ability to detect changes in knowledge use over time;
Examination of Developmental nature of levels and categories.

8. Method: Research Design
Study (NK) in LOKUS
T1 (Baseline) T2
(at 1 week)
Intervention (CKP) T3
(at 4 weeks) R
Web based group A O X B C
Paper-and-pencil Group

9. Method: Participants
Represent one of the 5 stakeholder types in the KT4TT Center intervention project – the “Clinicians”.
Inclusion: College Students / faculty members from allied health disciplines (Occupational therapy, nursing…..) and clinicians experienced in AAC; 18 years or older;
Exclusion: participation in similar survey/focus group within past 6 months.
Sample Size: Based on power analysis. large effect size d =.94 (Colbert, 1977); N = 64 (needed); 72 recruited, pre-screened and randomly assigned to two groups; Final N = 69 (35, paper-and-pencil, 34 web based).

10. Method: Intervention
LOKUS items asked about New Knowledge from 3 NIDRR funded published studies in AAC - Study A, Study B and Study C.
To identify Responsiveness to changes, a simulated Intervention condition was introduced between T2 and T3 by providing
a CKP on Study A (treatment) only to both groups.
and no CKP on the other two studies (controls).

11. Method: Data Analysis Focus Data Analysis
Equivalence of participant groups (WB and P&P)
Demographics on age, gender, marital status, highest educational level, experience in AAC
Independent t-tests and chi-square tests.
Test-Retest Reliability
LOKUS scores at T1 and T2, on levels and categories.
T1 - T2 comparison using ICC (3,1)
Compare Alternate Assessment methods
LOKUS scores at T1 and T3, on levels and categories.
Mann-Whitney U-tests, for difference between groups regarding (1) change (dichotomous) in levels and (2) number that changed their level status.
Responsiveness to change
LOKUS scores at T1 and T3, on levels.
Identify significant change (i.e., # of levels changed >Std Error of Measurement); Compare Studies A , B and C.
Developmental nature of levels
Participant changes in level status.
Developmental, if majority of changes move up a level.

12. Results: Demographics
There was no significant difference between the two groups regarding:
Age ( t = .757; p = .452);
AAC experience (t = .880; p = .382);
Gender (chi sq = 1.292; p = .256);
Marital status (chi sq = 4.872; p = .301) and
Education (chi sq = 1.003; p = .793)

13. Results: Test-Retest Reliability
Levels
Number who maintained Levels at T2
Categories 1 2 3 4 5 6
StudyA
Corr. =1.0)
0- Non Awareness 50
n/a
1-Awareness 12
2-Orientation
7-Collaboration
StudyB
(Corr.=1.0) 49 7 10
3-Preparation
4- Initial Use
Study C 58 8
Note: Identical responses at T1 & T2

14. Results: Responsiveness to Change
Study (NK)
Change from T1 to T3
T1 Mean
T3 Mean
Mean Diff. SD
SEM f %
Study A
Total 66
95.7
0.43
1.88
1.45
1.36
0.71
P &P 33
100
Web 31
91.2
Study B 5
7.2
0.65
0.86
0.21
1.13
0.44 2
5.7 3
8.8
Study C 8
11.6
0.17
0.33
0.16
0.60
0.25 4
11.4
11.8

15. Results: Difference between the Methods
Study (NK)
Change between T1 and T3
Difference between Methods
(p-level) f %
Changed/Not Changed
Number of levels
Study A
Total 66
95.7
0.038*
0.194
P &P 33
100
Web 31
91.2
Study B 5
7.2
0.147
0.125 2
5.7 3
8.8
Study C 8
11.6
0.961
0.100 4
11.4
11.8
Note: * Significant; p < 0.05

16. Number who moved up levels at T31 2 3 4 5 6 7 8
Tot 37 19
0- Non Awareness (n=97) 62
1-Awareness (n=45) - 14
2-Orientation (n=41)
3-Preparation (n=18)
4- Initial Use (n=3)
5-Routine Use (n=1)
6-Expansion (n=1)
7-Collaboration (n=0)
8- Integration (n=1)

17. Summary and Conclusions
LOKUS demonstrated:
Good Face Validity.
Excellent Test-Retest Reliability for both levels & categories.
Good responsiveness to detect change regarding use of New Knowledge.
Developmental nature for lower levels.
LOKUS is a usable tool at least for AAC field. Longitudinal study needed to extend conclusion for higher levels and for categories.

18. Key References
CIHR. About knowledge translation. Retrieved October 25, 2009, from irsc.gc.ca/e/29418.html
Hall, G.E., Dirksen, D.J., and George, A.A. (2006). Measuring Implementation in Schools: Levels of Use. Austin, TX: Southwest Educational Development Laboratory (SEDL).
Lane, J.P., Stone, V.I., Bauer, S. M., Leahy, J.A., and Tomita, M.R. (2008). Center on Knowledge Translation for Technology Transfer. Proposal submitted to National Institute for Disability and Rehabilitation Research (NIDRR)’s Disability and Rehabilitation Research (DRRP) Program (84.133A-7).
Sudsawad, P Knowledge Translation: Introduction to Models, Strategies, and Measures. Austin: Southwest Educational Development Laboratory, National Center for the Dissemination of Disability Research. (p.4; 21-22)
Telang, S.R. Establishing Psychometric Properties of the Level of Knowledge Use Survey (LOKUS) Questionnaire for Knowledge Translation for Technology Transfer. (Unpublished Masters Thesis). University at Buffalo, State University of New York.

19. Acknowledgement
This is a presentation based on Shreya Telang’s (2011) work for her masters’ thesis which was partially supported by the KT4TT Center under funding by the National Institute on Disability and Rehabilitation Research of the U.S. Department of Education, under grant number H133E The opinions contained in this presentation are those of the grantee and do not necessarily reflect those of the U.S. Department of Education.

20. Contact: vstone@buffalo.edu
Thank you!!
Contact:
Questions?

21. Appendix
Study A
Light, J. and Drager, K. (2007). AAC technologies for young children with complex communication needs. State of the science and future directions. Augmentative & Alternative Communication, 23 (3),
Study B
Light, J., McNaughton, D., Weyer, M. & Karg, L.(2008). Evidence-based literacy instruction for individuals who requireAugmentative and Alternative Communication: A case study of a student with multiple disabilities. Semin Speech Lang, 29 (2),
Study C
Quach, W.(2007). Facilitating children’s learning of Augmentative and Alternative Communication systems. Retrieved from Proquest Digital Dissertations. (AAT ).

22. LOKUS and the LoU Scale: Differences
 LoU Scale
LOKUS
 Purpose
 To measure Use of Innovations
To measure Use of Innovations
Context
Educational setting - part of Concerns Based Adoption Model (CBAM)
Broader, social setting – focus on knowledge from technology based research
Framework
8 Levels linked by decision points; 7 Categories in each level
10 Levels, with 3-6 Categories under levels
Observation Method
Systematic interviews
Web based, branched items eliciting self reported responses - large scale survey capability