The Competency Assessment Toolkit for Professional Psychology Catherine L. Grus, Ph.D., Associate Executive Director for Professional Education and Training Education Directorate American Psychological Association

THANK YOU Many thanks to Nadya Fouad, University of Wisconsin Nadine Kaslow, Emory University Benchmarks Workgroups Assessment “Toolkit” Workgroup

Overview Considerations in the assessment of competence Development of the Toolkit Putting the toolkit into practice

Competence … “ … the habitual and judicious use of communication, knowledge, technical skills, clinical reasoning, emotions, values, and reflection in daily practice for the benefit of the individual and community being served.” From Epstein and Hundert (2002)

Competence … … is developmental depending on stage of professional functioning … context dependent … executed according to ethical principles, guidelines, standards, and values of the profession and … requires public verification

Competencies … are elements of competence that are …. Observable Measurable Containable Practical Derived by experts And flexible are elements of competence that are ….

Assessment of Competence - Considerations Use of both “formative” and “summative” evaluation methods Authentic evaluations Real world tasks, meaningful activities [High] Fidelity: Degree to which the assessment represents actual performance

Employ a range of perspectives (multi-informant) and methods Challenge of evaluating attitudes and values Psychometric issues Feasibility/costs Lifelong self-assessment

What methods are commonly used to assess students/ trainees? In class examinations and course “grades” Faculty/supervisor completes rating scale Letters of recommendation EPPP, state jurisprudence exams

Developing Faculty/Supervisor Competencies Training in “difficult conversations” How to “critique constructively” Develop awareness of potential dual roles in evaluation and how that could influence results Benefits of having other faculty evaluate students’ performance Explicit obligations to make evaluations accurate Grade inflation Letters of recommendation to “market” vs. evaluate Knowledge of a range of assessment tools, access to resources “toolkits”

Competency Assessment for “Toolkit” for Professional Psychology Charge from APA Board of Educational Affairs: Develop a “Toolkit” for professional psychology Purpose: Promote broader implementation of competence assessment and provide information about application of assessment methods to the assessment of competence Coordinated with Benchmarks Work Group

Benchmarks Core Competencies Foundational Competencies: Professionalism Reflective practice Scientific knowledge and methods Relationships Individual and cultural diversity Ethical and legal standards and policy Interdisciplinary systems Functional Competencies: Assessment Intervention Consultation Research and evaluation Supervision Teaching Administration Advocacy

How are the Benchmarks Organized? Core Foundational and Functional Competencies Essential Component: what are the critical elements of/what knowledge/skills/attitudes that make up this competency? Behavioral Anchor: what would it look like if you saw it (essential component)?

Components of Toolkit Background and Introduction Assessment Method Fact Sheets Grid of Assessment Methods and Competencies/Essential Components Best used for

Fact Sheet: Portfolio Review Description A portfolio is a collection of products, gathered by the person being assessed … Application Competency Applies to1 Predominant Use of2 Developmental Level3 Professionalism E F,S 2,3,4 Reflective Practice 1,2,3,4 Implementation Deciding on form (e.g., web-based or hard copy) Psychometric Properties Reliability has not been well established due to the variable content included in a portfolio Strengths Provides a low cost assessment strategy Challenges Requires intense commitment of time and labor intensive for the assessors and the person being assessed Future directions Create and implement training modules for assessors and people being assessed in the portfolio assessment strategy

Grid of Possible Assessment Methods (Toolkit) Annual review Case reviews Process/outcome data Rating Forms Consumer surveys OSCE Portfolio reviews Ratings Performance Record reviews Self-assessment Simulations/role plays Stand. patient interview Stand. oral exams Written exams 360 Evaluations Intervention Knowledge interventions 2 3   1 Intervention planning Skills Implementation Progress evaluation

Assessment Methods in Toolkit 360 evaluation Portfolio OSCE Structured Written & Oral Exams Case Presentation Simulation/Role Play Competence Evaluation Rating Form Self-Assessment Ratings of live or recorded performance Standardized Client Interview Client/Patient Process/Outcome Measure Consumer Satisfaction Survey End of Rotation Performance Review

Putting it all Together … Benchmarks Toolkit Using the Toolkit and Benchmarks to Guide the Assessment of Student Learning Outcomes

Identify expected student learning outcomes using competency model (e.g., Benchmarks) Expected level of competence attainment and training plan developed with trainee Select assessment method(s) Student demonstrates problems with competencies acquisition Remediation plan developed Formative assessment Student demonstrates emerging competencies Training plan continues Summative evaluation to assess readiness for next level of training

Identify expected student learning outcomes using competency model (e.g., Benchmarks)

Program Outcome: Students will demonstrate the following intervention (functional) competencies prior to going on internship. Knowledge of scientific, theoretical, empirical and contextual bases of intervention, including theory, research, and practice Formulates and conceptualizes cases and plan interventions utilizing at least one consistent theoretical orientation Clinical skills Implements evidence-based interventions that take into account empirical support, clinical judgment, and client diversity (e.g., client characteristics, values, and context) Evaluate treatment progress and modify treatment planning as indicated, utilizing established outcome measures

Identify expected student learning outcomes using competency model (e.g., Benchmarks) Select assessment method(s)

Grid of Possible Assessment Methods (Toolkit) Annual review Case reviews Process/outcome data Rating Forms Consumer surveys OSCE Portfolio reviews Ratings Performance Record reviews Self-assessment Simulations/role plays Stand. patient interview Stand. oral exams Written exams 360 Evaluations Intervention Knowledge interventions 2 3   1 Intervention planning Skills Implementation Progress evaluation

Identify expected student learning outcomes using competency model (e.g., Benchmarks) Expected level of competence attainment and training plan developed with trainee Select assessment method(s)

A. Knowledge of Interventions Readiness for Practicum Developmental Level A. Knowledge of Interventions Readiness for Practicum Readiness for Internship   Essential Component: Basic knowledge of scientific, theoretical, and contextual bases of intervention and basic knowledge of the value of evidence-based practice and its role in scientific psychology Behavioral anchor: Articulates the relationship of EBP to the science of psychology Identifies basic strengths and weaknesses of intervention approaches for different problems and populations Knowledge of scientific, theoretical, empirical and contextual bases of intervention, including theory, research, and practice Behavioral Anchor: Demonstrates knowledge of interventions and explanations for their use based on EBP Demonstrates the ability to select interventions for different problems and populations related to the practice setting Investigates existing literature related to problems and client issues Writes a statement of one’s own theoretical perspective regarding intervention strategies

Identify expected student learning outcomes using competency model (e.g., Benchmarks) Expected level of competence attainment and training plan developed with trainee Select assessment method(s) Student demonstrates problems with competencies acquisition Remediation plan developed Formative assessment Student demonstrates emerging competencies Training plan continues

Remediation Plan Competency Domain/ Essential Components Problem Behaviors Expectations for Acceptable Performance Trainee’s Responsibilities/ Actions Supervisors’/ Faculty Responsibilities/ Timeframe for Acceptable Performance Assessment Methods Dates of Evaluation Consequences for Unsuccessful Remediation

Identify expected student learning outcomes using competency model (e.g., Benchmarks) Expected level of competence attainment and training plan developed with trainee Select assessment method(s) Student demonstrates problems with competencies acquisition Remediation plan developed Formative assessment Student demonstrates emerging competencies Training plan continues Summative evaluation to assess readiness for next level of training

For more information: http://www.apa.org/ed/graduate/competency.html Coming soon: Kaslow, N. J., Grus, C. L., Campbell, L. C., Fouad, N. A., Hatcher, R. L., & Rodolfa, E. R. (in press) Competency Assessment Toolkit for Professional Psychology. Training and Education in Professional Psychology.