Navigating Assessment Malissa Martin, Ed.D., ATC, CSCS Director, Athletic Training Education Department of Health Sciences College of Mount St Joseph Cincinnati, OH

Outcomes Review components of a Program Assessment Plan Develop and analyze components of an assessment plan Describe ways evidence is used to enhance student learning……… ”Close the Loop”

Agenda 2:00-2:15 Review Components of Program Assessment Plan 2:15-2:30 Learning Outcomes Indicators 2:30-2:45 Performance Indicators and Measures 2:45-3:00 Benchmarks 3:00-3:15 Closing the Loop 3:15-3:30 Summary

Assessment is Learning……. Students Curriculum (didactic and clinical) Institutional fit Assessment process

Accreditation General Program Assessment Questions How are your LOs appropriate to your mission, program, degree? What evidence do you have that your students are achieving your LOs? In what ways to do you analyze student learning? How do you evaluate and improve the effectiveness of your assessment of student learning? Where do you assess your LOs?

Assessment “Systematic collection, review, and use of information about educational programs undertaken for the purpose of improving student learning and development” Marchese, 1987

Programmatic Assessment Provides information about groups of students Determines student’s ability to integrate individual courses/clinical experiences into a coherent whole “Academic Introspection” Denney, 1996

Characteristics of an Assessment Plan All LOs are addressed and assessed Multiple assessment measures Efficient procedures Describe people, committees, process involved Plan “closes the loop” Ball State College of Business Banta TW & Palomba CA (1999) Assessment Essentials Josey-Bass

Meaning of Assessment What should your graduates know, be able to do, and value? Have graduates acquired learning? What are the contributions of the program to student growth? How can student learning be improved? Hutchings & Marchese, 1990

Common Language See handout…..

Guided Outcome Questions…. Where do students start? What do they encounter along the way? Where do they end up? “The Ideal Graduate”

Components of a Program Assessment Plan 1. Expected Outcomes (learning outcomes) Summative What will students be able to do upon completion of the program? Formative What will students be able to do along the way

Nichols Model 5 Column Assessment Report Model SACS

Bloom’s and Assessment Plan Varying degrees of learning Produce learning vs providing instruction (Barr and Tagg, 1995)

Group Exercise At your table, develop 2 program learning outcomes (LOs). Excluding –Students will successfully pass the BOC examination and gain ATC credential

Components… Where are outcomes addressed? Courses (list) Clinical program Service learning

Components… 2. Criteria to assess outcomes Performance Indicators How will students meet LO -type of product When will students meet LO -formative vs summative Where will students meet LO didacticclinical extra-curricular experience (service learning)

Components… 3. Measures (how will data be collected) PortfolioSimulations PerformanceCase studies Writing samplesPoster presentations Oral presentationsInterviews TestsClinical evaluations Simulations Focus groups Surveys Qualitative and quantitative measures Direct vs Indirect measures

Group Exercise Using your 2 Los, develop –Performance indicators for each How will students meet LO –Product When will students meet LO –Formative vs summative Where in curriculum will students meet LO –Course-Clinical Exp-Post-Grad –Identify measures used to assess the LO

Performance Assessment Authentic assessment Linked to curriculum/courses Real examples of student work Critical and creative thinking Holistic feedback Student motivation Labor intensive

Key Points Performance Assessment Articulate skills to be examined Design appropriate tasks to demonstrate skills Specify criteria for evaluating performance Develop reliable process for rating performance Training raters Evaluating results

Current Trend Course-embedded assessment Ewell, 1991 and Wright, 1997

Components… 4. Program’s expected level of performance on each outcome (benchmarks ) Numerical score on national exam Mastery level on writing sample Masterly level on oral/poster presentation Masterly level a behavioral performance Masterly level on a clinical evaluation Holistic score on ability to solve a case study At what level is student expected to achieve the outcome?

Group Exercise Using your PIs: –Develop criteria or benchmarks for each PI

Assessment Schedule Entry into the program End of a specific semester Completion of a set of courses Completion on a certain number of credits Upon graduation Upon employment Number of years after graduation Should each outcome be assessed every year?

Who collects/analysis/interprets results? Inside EvaluatorsOutside Evaluators Student affairs Alumni Writing center Employer Assessment committee BOC ATEP committee GRE

Components… 5. Results of data for each outcome What percentage (number) of students met the LO/PI at the criteria/benchmark established May want to include sample size

Sharing Results Who will see the results ? Administration Faculty Students Outside accrediting agencies

Components… 6. Closing the Loop Does the assessment plan lead to improvements in academic and co-curricular programs? Revise curricula, sequence of courses…. Design more effective student orientation Revise handbook Increase connection between academic and clinical program Purchase additional therapeutic exercise equipment

Closing the Loop Question “What changes to your program have you made based upon results of your program assessment data?”

Closing the Loop Curricular changes –Course information (competencies) –Instructional delivery –Course sequencing –Instructional resources (textbooks, anatomical models, library holdings) –Technology

Closing the Loop Clinical program –Clinical instruction –Clinical sites –Clinical rotations –Clinical opportunities –Clinical performances

Closing the Loop Space Budget Faculty development Human resources

Group Exercise Closing the Loop With results provided: Develop a “closing the loop” plan

Assessing the Program Plan Follow-up on implemented changes How often? Who?

Trending Data Why? See changes overtime…… Overtime learning evidence Charts student’s chronological achievement Lots of work on front-end!

Trending Data Semester to semester Year to year Each learning outcome (program vs student) Clinical evaluations (student) Clinical evaluations (instructors) Clinical site evaluations Teacher evaluations

Key Points to Faculty Involvement R esponsibility R esources R ewards Palomba and Banta (1999)

Characteristics of a Good Assessment Plan Asks important questions Reflects the institutional mission Reflects program goals and Los Linked to decision making about –Curriculum-budget-resources Involvement of multiple individuals Contains relevant assessment techniques

Characteristics of a Good Assessment Plan Indirect and direct evidence of learning Reflects what is known about student learning Shares information with multiple audiences Leads to reflection of faculty, staff, and students Allows for continuity, flexibility, and improvement in the assessment process

Navigating Assessment Thank You…… Questions ?????????????