 Elaine Mormer, PhD, University of Pittsburgh  Deborah Moncrieff, PhD University of Pittsburgh  Deborah Dixon, MA, ASHA, Director, School Services  Janet Deppe, MS, ASHA, Director, State Advocacy

 Personnel preparation programs must develop valid and reliable student assessment instrumentation (ACAE 2005 standards; CAA standards 2013)  No standardized approach to clinical skills and knowledge across AuD and SLP programs  Several states have developed performance measures for classroom teachers, but not for Audiologists & Speech-Language Pathologists (Asha, 2012)  Development of student/clinician performance evaluation should follow a systematic approach, using appropriate resources

We applied a systematic approach to design an instrument to assess and track AuD and SLP student knowledge and skills necessary for provision of services to high need children in underserved populations. The steps were as follows: 1.Identification of relevant skills and knowledge (i.e. items to be rated) 2.Application of a reliable and valid rating scale 3.Implementation of a data collection mechanism

 1. Identification of valid skill and knowledge list Identified resource materials e.g. publications, standards, etc.(see next panel) Solicited input from relevant constituents e.g. school personnel, consumers, etc. Peer review and feedback Pilot items with clinical instructors  2. Application of a valid and reliable rating scale Determine scale format e.g. # of points vs. checklist Create anchor values Create clearly defined descriptor for each scale point Considered priorities for psychometric characteristics  3. Implementation of a manageable data collection mechanism Format i.e. paper based vs. online? Availability of support staff for distribution and collection? Use of existing online mechanisms e.g. Typhon AHST, E*Value

Produced by the American Speech ‐ Language ‐ Hearing Association’s (ASHA) Value ‐ Added Project Team member requests, rapidly developing State ‐ level policies regarding accountability measures for school ‐ based speech ‐ language pathologists (SLPs) Objective was to identify a value ‐ added model designed for SLPs or one that specifically accounted for the unique contributions of SLPs Team reviewed literature, attended seminars, and conducted a peer review to obtain input from related professional organizations, members, pertinent stakeholders, and researchers on value ‐ added models and assessments

 Addresses cultural /linguistic variations in screening/assessment activities  Calculate classroom reverberation times  Participate effectively on multi-disciplinary team  Minimize barriers to curriculum access  Assist educational team member in making referrals  Train and supervise support personnel  Write appropriate IEP goals, considering academic, behavioral, and developmental issues  Counsel regarding transition planning

Data collection mechanism is online via the TyphonGroup © EASI survey system

1. Currently piloting with selected clinical instructors 2. Implementation of instrument in Fall term Evaluation of instrument pending responses on first round of implementation 4. Ongoing editing of items and scale as per constituent feedback

 Kogan J, Conforti L, Benabeo E, Iobst W, Holmboe, E(2011). Opening the black box of clinical skills assessment via observation: a conceptual model. Medical Education doi: /j x  American Speech-Language-Hearing Association (2005). Quality indicators for professional service programs in audiology and speech-Language pathology. Available from:  American Speech-Language-Hearing Association (2012). Performance assessment of contributions and effectiveness of speech-language pathologists (PACE). Available from: Assessment-Contributions-Effectiveness.pdfhttp:// Assessment-Contributions-Effectiveness.pdf  American Psychological Association (1999). Standards for Educational and Psychological Testing., Washington, DC: American Educational Research Association  Crossley J, Humphris G, Jolly B. (2002). Assessing Health Professionals. Medical Education 36 :

 Value Added Assessment  Research Findings  Rationale for the development of the PACE  Goals of an assessment system  Components of PACE  Tools and resources

 Value-added assessment a process to accurately and fairly assess a professional’s impact on student performance and overall success of the school community.  A comprehensive, statistical method of analyzing test data that measures teaching and learning.

 Federal grant programs and waivers require states to include VAA/ teacher accountability measures in applications  VAA is viewed as an important accountability measure  Teacher accountability systems are being developed in many states

 Research has primarily focused on implications of use of VAA with classroom teachers.  Notable concerns surfaced, such as difficulty linking student outcomes to one teacher and uncertainty about the accuracy of imputation models for missing student data.

 Evaluating the value that an SLP brings to the school or connecting their value to specific student performance is a challenge when compared to a classroom teacher.  ASHA’s Value-Added Working Team was not able to identify any VAA models that specifically incorporated SLPs.

 Since there was no system specifically developed for SLPs or other support personnel ASHA wanted to ensure that the assessment model for SLPs:  accurately reflects the speech-language pathologist’s (SLP) unique role in contributing to a child’s overall performance.  Demonstrates that the SLP is contributing to the success of the school community.

 ASHA also wanted to make sure that the evaluation system for SLPs was:  Comprehensive  Used multiple measures  Demonstrated valid and reliable findings  Provided data for professional development objectives  Linked to the specific roles and responsibilities of the specific job

- PACE Matrix  Portfolio  Observation chart  Teacher, student, and parent checklist  Self reflection tool  Observations by individual with knowledge of the roles and responsibilities of SLP  The matrix consists of a set of nine objectives by which an SLP should be evaluated.  These objectives are derived from typical roles and responsibilities of a school based SLP  The portfolio is developed to show evidence of mastery of each objective

 The PACE documents can be located at: e-Assessment-of-Contributions-and-Effectiveness/ e-Assessment-of-Contributions-and-Effectiveness/  Additional tools/resources include:  A guide for developing the portfolio for the Matrix  An evaluator’s guide  Observation “Look Fors” and scoring system for the Matrix  A Step-by-Step guide for using the PACE system

 When considering an assessment instrument, what are relevant quality markers necessary to include?  What are the challenges of evaluating related services scholars and graduates?  How can data collected be used to improve program quality?  Can we ensure “calibrated” responses across raters?