Dr Sally Boa and Dr Joan Murphy Professor Pam Enderby Funded by NHS Education Scotland Conducted by Talking Mats Limited © Talking Mats Ltd 2014

Background A Right to Speak (Scottish Government 2012) 1. AAC services to demonstrate the effectiveness of AAC interventions. 2. National statistics on AAC to be gathered by relevant agencies to support future gathering of cost effectiveness data on AAC to ensure that AAC funding is sustained in the longer term. © Talking Mats Ltd 2014

Two stage project 1.Scoping of current methods of measuring outcomes 2.Testing the validity and inter-rater reliability of TOM AAC (Enderby et al 2006) © Talking Mats Ltd 2014

Main points arising from scoping project Agreed on what we mean by ‘Outcome Measure’ Agreed on what is the best measure to use in AAC Identified the need for a measure to be workable across agencies © Talking Mats Ltd 2014

Definition of an outcome measure Outcome Measures:  Should measure change  Can be repeated over time  Can be used across clients, settings and disciplines  Can be used to give feedback to services, professionals, carers and clients (about what works as well as what doesn’t)  Should be robust/standardised/well respected  Should inform discharge  Should be short and straightforward to use  Should allow for comparison with different types of AAC/No AAC © Talking Mats Ltd 2014

Main contender for outcome measure in AAC TOMs AAC (still in development – needed to be tested for reliability and validity) © Talking Mats Ltd 2014

Stage 2 Testing TOM AAC for validity and inter-rater reliability © Talking Mats Ltd 2014

Therapy Outcome Measure (TOM) An Outcome Measure used by many rehabilitation professionals to measure the impact of their interventions Based on the World Health Organisation’s International Classification of Functioning, Disability and Health (WHO ICF, 2001) Administered following assessment/intervention of an individual by a professional Individuals are rated using an 11 point ordinal scale with 6 defined points. People are rated in relation to four descriptors: Impairment, Activity, Participation and Well-being. © Talking Mats Ltd 2014

Therapy Outcome Measure © Talking Mats Ltd 2014

TOM AAC Adapted scale for use with an AAC population (children and adults) Domains of activity and participation can be rated with and without AAC © Talking Mats Ltd 2014

Recruitment of AAC professionals (n = 29) © Talking Mats Ltd 2014

Validity testing Training provided Each professional was asked to rate 10 AAC users using TOM AAC Feedback provided through: – Online survey – Focus groups © Talking Mats Ltd 2014

Findings from the survey Coherence and relevance of the descriptors Strong agreement that the descriptors in TOM AAC were appropriate and relevant for use with an AAC population Respondents reported a good understanding of the difference between impairment, activity, participation and well-being Usability of the measure in practice Most agreed that the measure was quick and easy to use Most agreed that the TOM AAC would help them to report on outcomes on their work with individuals who use AAC © Talking Mats Ltd 2014

Findings from Focus Groups TOM AAC was straight forward to use – “I think it’s really useful. I think it’s a really quick, useful tool.” (Participant from focus group 3) Comments about wording of the impairment sections - a distinction between ‘expression’ and ‘comprehension’ rather than ‘speech’ and ‘language’ © Talking Mats Ltd 2014

Strong face validity – changes made to descriptors of TOM AAC Wording changed to ensure consistency across the descriptors Within the impairment section, distinction was made between ‘expression’ and ‘comprehension’ rather than ‘speech’ and ‘language’. More detailed explanation was also provided for each descriptor to increase the clarity of meaning for each section © Talking Mats Ltd 2014

Inter-rater reliability testing AAC professionals (n = 17) provided with ten case histories to rate using TOM AAC Scores collated and analysed using Intraclass Correlation Coefficient (ICC) © Talking Mats Ltd 2014

Findings Agreement was generally acceptable – highest for physical impairment (0.887), comprehension (0.831) and wellbeing (0.816) – the lowest was for expression (0.486) © Talking Mats Ltd 2014

Inter-rater reliability TOM domainICC Physical Impairment0.887 Cognitive Impairment0.726 Sensory Impairment0.605 Expression0.486 Comprehension0.831 Activity0.622 Participation0.703 Wellbeing0.816 © Talking Mats Ltd

Limitations We may not have provided sufficient information in case histories five and six, as ratings for these case histories gave the largest range between individuals. The domain of ‘expression’ appeared to produce the most variation between raters – need for clearer definition of descriptors Further investigation needed around ‘expression’ © Talking Mats Ltd 2014

With adaptations © Talking Mats Ltd 2014

Greig © Talking Mats Ltd 2014

Example © Talking Mats Ltd 2014

Summary TOM AAC has been tested for face validity and inter-rater reliability TOM AAC has the potential to provide a quick, simple and reliable for all AAC professionals TOM next edition will include AAC © Talking Mats Ltd 2014

References Communication Matters (2012) surement_sept_2012.pdf surement_sept_2012.pdf Framework for Measuring Impact Scottish Government (2012) ‘A Right to Speak’ Lacey, A. & Luff, D. (2001). Trent focus for research and development in primary health care: An introduction to qualitative analysis. London: Trent Focus. Khangura et al 2012 Evidence summaries: the evolution of a rapid review approach TOMS: Enderby P., John A., & Petheram B. (2006) Therapy Outcome Measures for Rehabilitation Professionals: Speech and Language Therapy, Physiotherapy, Occupational Therapy, Rehabilitation Nursing & Hearing Therapists. Second Edition, John Wiley & Sons Ltd. CODES: Shrout, PE, Fleiss, JL. Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin 1979; 86(2): © Talking Mats Ltd 2014