1 Developing Indicators of Quality for AT Reuse Programs Lindsey Bean Kampwerth and Trish Redmon Moderators: Carolyn Phillips, Joy Kniskern and Liz Persaud Pass It On Center

2 Agenda –Vision and goals of Indicators of Quality –Overview of the development process –Themes of feedback on first two versions –Design of Prototype IQ - ATR Online Assessment Tool –Next steps for IQ and Online Tool

3 Vision and Goals for Indicators of Quality for AT Reuse IQ - ATR IQ - ATR is a living document that grows and develops as AT Reuse Programs develop.IQ - ATR is a living document that grows and develops as AT Reuse Programs develop. IQ - ATR Online will be a dynamic, interactive tool to assess an AT Reuse Program.IQ - ATR Online will be a dynamic, interactive tool to assess an AT Reuse Program. IQ - ATR will link to Pass It On Knowledge Base to provide references to information and models to assist with improving the Reuse Program.IQ - ATR will link to Pass It On Knowledge Base to provide references to information and models to assist with improving the Reuse Program. 3

4 Vision and Goals for IQ - ATR 4 Knowledge Base supports IQ-ATR Self-Assessment Tool IQ-ATR and Knowledge Base inform Successful Practices Successful Practices inform Knowledge Base IQ - ATR

5 This is not the first project: Other Related QI Initiatives The SMART ExchangeThe SMART Exchange –Led by Shelley Kaplan, UCP in Atlanta , NIDRR grant , NIDRR grant Developed indicators for AT useDeveloped indicators for AT use The rationale for many of the SMART Exchange indicators are relevant to AT reuse indicators.

6 Other Related QI Initiatives QIATQIAT –Quality Indicators for Assistive Technology Services in School Settings –Developed in 1998 QIAT is an outstanding working model of quality indicators for AT use in a specific environment.

7 Validating the IQ-ATR 1.Have we defined a set of factors or processes known to lead to successful outcomes? 2.Can we create a review instrument to determine if the factors/processes are in place? 3.Is the indicator useful for internal quality improvements? 4.Can the indicator be used as an external measure of quality across programs of the same type and activities?

8 Developing the IQ - ATR Two Webinars on Quality IndicatorsTwo Webinars on Quality Indicators A meeting of program professionals and PIOC team members in Atlanta to develop initial draftA meeting of program professionals and PIOC team members in Atlanta to develop initial draft Distribution of draft to developers and interested partiesDistribution of draft to developers and interested parties Editing of draft based on feedback, including alteration of restrictive formatEditing of draft based on feedback, including alteration of restrictive format Development of assessment tool prototypeDevelopment of assessment tool prototype

9 Developing the first draft Reviewed past quality indicatorsReviewed past quality indicators Reviewed guidelines for writing quality indicatorsReviewed guidelines for writing quality indicators Small groups edited and added to draft of quality indicatorsSmall groups edited and added to draft of quality indicators

10 First draft done: Critique time! QI for AT Reuse Development Meeting in Atlanta, March 23 & 24 with occupational therapists, program directors and other key informants. Thank you!

11 Categories of Indicators Program OperationsProgram Operations User ServicesUser Services SustainabilitySustainability ManagementManagement Human ResourcesHuman Resources Marketing Accounting Organizational Structure Emergency Preparedness Partnerships and Collaboration

12 Suggestions from reviewers: Reorganize format: categories and sequenceReorganize format: categories and sequence Combine some topics; separate some topicsCombine some topics; separate some topics Consider strictures of tabular formatConsider strictures of tabular format Add rationale for categoriesAdd rationale for categories Reorganize into “core” indicators and options based on program typeReorganize into “core” indicators and options based on program type Add specific AT State program indicatorsAdd specific AT State program indicators

13 Version 2 Restrictive five-cell response format was abandoned in favor of a flexible checklist of key factors for considerationRestrictive five-cell response format was abandoned in favor of a flexible checklist of key factors for consideration Indicators were combined or split as neededIndicators were combined or split as needed New version was presented to National Task Force – which recommended a thorough language reviewNew version was presented to National Task Force – which recommended a thorough language review

14 Version 2.1 NTF sub-committee (Rik Opstelten, Brian Bard, Kerri Morgan and Lindsey Bean) reviewed the language and recommended changes.NTF sub-committee (Rik Opstelten, Brian Bard, Kerri Morgan and Lindsey Bean) reviewed the language and recommended changes. Oops! Name change! QIAT recommended against expansion of their name and possible confusion. (Hence, IQ-ATR.)Oops! Name change! QIAT recommended against expansion of their name and possible confusion. (Hence, IQ-ATR.) Language updated, and some major issues were confronted before building the Online Tool.Language updated, and some major issues were confronted before building the Online Tool.

15 Major Challenges Organizing the indicators into CORE indicators and implementing appropriate ‘branching’ based on program typeOrganizing the indicators into CORE indicators and implementing appropriate ‘branching’ based on program type Weighting the categories and the indicators to create an appropriate “result” measure for the programWeighting the categories and the indicators to create an appropriate “result” measure for the program Implementing online assessment tool and links to Knowledge BaseImplementing online assessment tool and links to Knowledge Base

16 Organizing the Indicators Variables affecting grouping: Type of organizationType of organization AT reuse activities (reassign, refurbish, exchange, recycle) AT reuse activities (reassign, refurbish, exchange, recycle) Types of equipment (DME, computers, all?)Types of equipment (DME, computers, all?) How to deal with the possible combinations?

17 Weighting Categories & Indicators 1. Survey designed and individuals asked to rank: Categories in order of importanceCategories in order of importance Indicators in order of importance within categoriesIndicators in order of importance within categories 2. Ph.D. statistician recommended weighting methodology based on ranking, but it required a standard number of responses.

18 To weight or not to weight? Based on the ranking of categories and indicators, it is possible to produce a numeric “score” from the online tool.Based on the ranking of categories and indicators, it is possible to produce a numeric “score” from the online tool. For the initial assessment tool, it was decided not to produce numeric results, but advisory messages supported by lists of resources for each indicator.For the initial assessment tool, it was decided not to produce numeric results, but advisory messages supported by lists of resources for each indicator.

19 Version 3 Challenge: A return to the five-cell format would lose ease-of-use that made the flexible checklist of key factors so attractiveChallenge: A return to the five-cell format would lose ease-of-use that made the flexible checklist of key factors so attractive Challenge: Too many variables to group indicators by program typeChallenge: Too many variables to group indicators by program typeSolution:  Retain the flexible lists of key factors for consideration.  Create standard set of responses to assess compliance.  Allow program to determine if indicator applies.  No numeric scores – even hidden – at this time

20 IQ-ATR and the Online Program Assessment Tool Sharon Meek and the PIOC web developers from REACH, Inc. developed the prototype of the interactive toolSharon Meek and the PIOC web developers from REACH, Inc. developed the prototype of the interactive toolwww.passitoncenter.org/iq-atreuse

21 Home Page: A brief introduction to how the IQ-ATR tool works

22 Registration Create an account for personal use or register as a guest.

23 Directions explain the response choices.

24 Category in the yellow bar. The name of the indicator and the text of the indicator are followed by the Rationale and the Key Factors for Consideration. The standard set of responses follows. User selects a response and clicks Next.

25 User receives a results page for each category and may stop or continue to the next category. KB or other sources are suggested for indicators that do not meet all Key Factors for the promising practice.

26 Next Steps IQ-ATR: Share and solicit additional feedback Share and solicit additional feedback Revise IQ - ATR based on recommendations Revise IQ - ATR based on recommendations Design a validation process Design a validation process

27 Next Steps Online Program Assessment Tool: Gather feedback from pilot users Gather feedback from pilot users Revise online tool and make “updatable” to keep synchronized with Knowledge Base (KB) Revise online tool and make “updatable” to keep synchronized with Knowledge Base (KB) Allow user to choose sequence of categories Allow user to choose sequence of categories Have NTF review KB content for appropriateness and usefulness as recommended at May meeting Have NTF review KB content for appropriateness and usefulness as recommended at May meeting Identify or create content to fill reference gaps in Knowledge Base Identify or create content to fill reference gaps in Knowledge Base Decide whether responses should be weighted and tallied for a quantitative (rather than general) result. Decide whether responses should be weighted and tallied for a quantitative (rather than general) result.

28 Not enough knowledge... Referencing Indicators of Quality to content in the Knowledge Base highlighted the gaps in supporting information.Referencing Indicators of Quality to content in the Knowledge Base highlighted the gaps in supporting information. The content that does exist needs to be strengthened with more comprehensive presentation of needed policies and procedures.The content that does exist needs to be strengthened with more comprehensive presentation of needed policies and procedures. The existing content has not been reviewed by NTF panels as recommended in May 2009.The existing content has not been reviewed by NTF panels as recommended in May 2009.

29 How you can help Review the IQ-ATR and send commentsReview the IQ-ATR and send commentshttp://passitoncenter.org/content Try the online tool and send feedbackTry the online tool and send feedbackhttp://passitoncenter.org/IQATReuse Contribute content or suggest resources for the Knowledge BaseContribute content or suggest resources for the Knowledge Base Please send comments and suggestions to:

30 DISCLAIMER This work is supported under a five-year cooperative agreement # H235V awarded by the U.S. Department of Education, Office of Special Education and Rehabilitative Services, and is administered by the Pass It On Center of the Georgia Department of Labor – Tools for Life. However, the contents of this publication do not necessarily represent the policy or opinions of the Department of Education, or the Georgia Department of Labor, and you should not assume endorsements of this document by the Federal government or the Georgia Department of Labor.