Making Assessment Meaningful Marian University August 2012 Susan Hatfield Winona State University Visiting Professor, Higher Learning SHatfield@winona.edu

1. Assessment is about student learning

2. It isn’t going away

3. You are probably already doing it

Evaluation Assessment Used to see if students understand Quizzes Count toward final grade Used to see if students understand Tests Scored and returned Scored, tabulated. returned & discussed; adjustments to syllabus Rubrics Returned to students with grade

Does not meet Meets Exceeds Verbal Delivery x Nonverbal Delivery x Organization x Evidence x Transitions x

Evaluation Assessment Used to see if students understand Quizzes Count toward final grade Used to see if students understand Tests Scored and returned Scored, tabulated. returned & discussed; adjustments to syllabus Returned after being aggregated & analyzed; adjustments to syllabus Rubrics Returned to students with grade

Does not meet Meets Exceeds Verbal Delivery 14% 81% 5% Nonverbal Delivery 22% 74% 14% Organization 14% 79% 7% Evidence 9% 72% 19% Transitions 7% 85% 8% Program Summary

4. Assessment requires exertion and intention

Exertion without Intention

Intention without Exertion

Intention and Exertion

5. There is a difference between assessing Program Effectiveness and assessing Student Learning Outcomes

Assessment of Program Effectiveness What the program will do or achieve Curriculum Retention Graduation Placement Satisfaction (graduate and employer)

Assessment of Student Learning Outcomes What students will do or achieve Direct Methods of Assessment

6. Assessment drives improvement & demonstrates integrity

How Assessment Works Phase 2 Phase 3 Phase 1 O U T C M E Compare Against Benchmarks, Standards, Targets, Past Performance New / Revised learning event New / Revised learning event learning event component BASELINE component component

7. Assessment makes more sense if you focus on what is important to you

The relationship between values and measures

8. The success of any assessment efforts depends on the outcomes

Student Learning Outcomes Students should be able to <<action verb>> <<something>>

Student Learning Outcomes Learner Centered Specific Action oriented Public Distinctive Frame Perceptions Cognitively appropriate

Lower division course outcomes ANALYSIS COMPREHENSION EVALUATION APPLICATION SYNTHESIS KNOWLEDGE Associate Classify Compare Compute Contrast Differentiate Discuss Distinguish Estimate Explain Express Extrapolate Interpolate Locate Predict Report Restate Review Tell Translate Analyze Appraise Calculate Categorize Classify Compare Debate Diagram Differentiate Distinguish Examine Experiment Inspect Inventory Question Separate Summarize Test Arrange Assemble Collect Compose Construct Create Design Formulate Integrate Manage Organize Plan Prepare Prescribe Produce Propose Specify Synthesize Write Appraise Assess Choose Compare Criticize Determine Estimate Evaluate Grade Judge Measure Rank Rate Recommend Revise Score Select Standardize Test Validate Cite Count Define Draw Identify List Name Point Quote Read Recite Record Repeat Select State Tabulate Tell Trace Underline Apply Calculate Classify Demonstrate Determine Dramatize Employ Examine Illustrate Interpret Locate Operate Order Practice Report Restructure Schedule Sketch Solve Translate Use Write Lower division course outcomes

Upper division Course / Program outcomes ANALYSIS COMPREHENSION EVALUATION APPLICATION SYNTHESIS KNOWLEDGE Associate Classify Compare Compute Contrast Differentiate Discuss Distinguish Estimate Explain Express Extrapolate Interpolate Locate Predict Report Restate Review Tell Translate Analyze Appraise Calculate Categorize Classify Compare Debate Diagram Differentiate Distinguish Examine Experiment Inspect Inventory Question Separate Summarize Test Arrange Assemble Collect Compose Construct Create Design Formulate Integrate Manage Organize Plan Prepare Prescribe Produce Propose Specify Synthesize Write Appraise Assess Choose Compare Criticize Determine Estimate Evaluate Grade Judge Measure Rank Rate Recommend Revise Score Select Standardize Test Validate Cite Count Define Draw Identify List Name Point Quote Read Recite Record Repeat Select State Tabulate Tell Trace Underline Apply Calculate Classify Demonstrate Determine Dramatize Employ Examine Illustrate Interpret Locate Operate Order Practice Report Restructure Schedule Sketch Solve Translate Use Write Upper division Course / Program outcomes

9. The outcomes need to be defined

Components Define student learning outcomes Provide a common language for describing student learning Must be outcome specific Must be shared across faculty Number of components will vary by outcome

Communication Write Relate Speak Listen Participate Component

Components Communication Write Relate Speak Listen Participate Process delivery Process Summary content organization

Components Communication Write Relate Speak Listen Participate Letter mechanics Letter style organization

The Reality of Assessing Student Learning Outcomes Why you need common components

teacher4 teacher2 teacher1 teacher3 teacher5 Speaking eye contact gestures volume sources transitions style rate poise examples verbal variety appearance evidence conclusion organization attention getter

1 K A A A A S 2 K A S 3 K K K K 4 5 K S 6 K K A A S 7 S A A S Program Level Student Learning Outcomes 1xx 1xx 2xx 2xx 2xx 3xx 3xx 3xx 4xx Capstone 1 K A A A A S 2 K A S 3 K K K K 4 5 K S 6 K K A A S 7 S A A S K= Knowledge/Comprehension; A= Application / Analysis; S= Synthesis /Evaluation

1 K A A S A S 2 K A A A Build a curriculum 3 K K A S 4 K K S 5 K K A S Program Level Student Learning Outcomes 1xx 1xx 2xx 2xx 2xx 3xx 3xx 3xx 4xx Capstone 1 K A A S A S 2 K A A A Build a curriculum 3 K K A S 4 K K S 5 K K A S S 6 K K K A S 7 S A A S S K= Knowledge/Comprehension; A= Application / Analysis; S= Synthesis /Evaluation

Analyze Curriculum 1 K A A S A S 2 K A A A 3 K K A S Diagnose Learning Program Level Student Learning Outcomes 1xx 1xx 2xx 2xx 2xx 3xx 3xx 3xx 4xx Capstone Analyze Curriculum 1 K A A S A S 2 K A A A 3 K K A S Diagnose Learning 4 K K S 5 K K A S S 6 K K K A S 7 S A A S S K= Knowledge/Comprehension; A= Application / Analysis; S= Synthesis /Evaluation

1 K A A S A S 2 K A A A 3 K K A S Diagnose Learning 4 K K S 5 K K A S Program Level Student Learning Outcomes 1xx 1xx 2xx 2xx 2xx 3xx 3xx 3xx 4xx Capstone 1 K A A S A S 2 K A A A 3 K K A S Diagnose Learning 4 K K S 5 K K A S S 6 K K K A S 7 S A A S S K= Knowledge/Comprehension; A= Application / Analysis; S= Synthesis /Evaluation

11. You don’t have to assess every outcome in every course

Assessment Points Assessment is NOT assessing every student on every outcome in every class by every faculty member every semester

1 K A A S A S 2 K A A A S 3 K A A A A S 4 K A S S 5 K A K A S S 6 K K Program Level Student Learning Outcomes 1xx 1xx 2xx 2xx 2xx 3xx 3xx 3xx 4xx Capstone 1 K A A S A S 2 K A A A S 3 K A A A A S 4 K A S S 5 K A K A S S 6 K K A A S 7 S A A S S S K= Knowledge/Comprehension; A= Application / Analysis; S= Synthesis /Evaluation

12. There are multiple ways to assess student learning outcomes

Assessment Tools Quantitative Data Numbers Percentage Ratios Rates Qualitative Data Descriptions Explanations Perceptions

Types of Assessment Data Objective Data records Databases, incidents, reports Surveys, Questionnaires, Interviews, Focus Groups Subjective Self-Report feelings Logbook, Action Report Behavioral Self-Report actions Subjective Analysis thoughts Reflective Essay Objective Analysis observations Checklist, Rubric, Scale

Learning Objects Standardized Exam, abstract, advertisement, annotated bibliography, biography, briefing, brochure, budget, care plan, case analysis, chart, cognitive map, court brief, debate, definition, description, diagram, dialogue, diary, essay, executive summary, exam, flow chart, group discussion, instruction manual, inventory, lab notes, letter to the editor, matching test, mathematical problem, memo, micro theme, multiple choice test, narrative, news story, notes, oral report, outline, performance review, plan, presentation, process analysis, proposal, regulation, research proposal, review of literature, taxonomy, technical report, term paper, thesis, word problem, work of art. (Walvoord Anderson 1998).

1 2 3 4 Student Learning Outcome Test Questions 1 2 3 4 5 6 7 8 9 10 11 12 4 13 14 15 16 17 18

Student Learning Outcome Assignment 1 1 2 2 3 3 4 4

Student Learning Outcome Assignment 1 1 2 3 2 4

Student Learning Outcome Experience 1 2 3 4

Options for Assessing Learning Test Project Case Study Care Plan Patient History Presentation Portfolio Brochure Internship Number or percentage correct Comparison to national norms

1 2 3 4 Student Learning Outcome Test Questions % correct 1 2 3 4 73% 5 6 7 8 9 82% 3 10 11 12 64% 4 13 14 15 16 17 18 87%

Options for Assessing Learning Test Project Case Study Care Plan Patient History Presentation Portfolio Brochure Internship Number or percentage correct Comparison to national norms Scale Holistic Rubric

1 2 3 4 5 Presentation 1 2 3 4 5 1 2 3 4 5 Student Learning Outcome Components Presentation Content Familiar 1 2 3 4 5 New Delivery Distracts 1 2 3 4 5 Enhances Organization Unclear Focused 1 2 3 4 5

Student Learning Outcome Components Presentation Content 1 2 3 4 5 6 7 Delivery 1 2 3 4 5 6 7 Organization 1 2 3 4 5 6 7

Rubrics for Students Clarify expectations Focus attention Set standards Provide detailed feedback

Rubrics for Faculty Clarify goals of assignment Standardize feedback More detailed feedback Enhance objectivity Prevent evaluation drift Focus discussions with students

Rubrics for Assessment Consistent Feedback across students, course, program allowing for the monitoring of learning outcomes Provide Direction for Course, Class, Program Improvement

Rubrics Analytical Numerous components which you want to assess individually Summative

Exceeds Presentation Student Learning Outcome Components Content Delivery Organization

Rubrics Developmental Expectations for performance rise throughout the class or curriculum

Presentation sophistication Student Learning Outcome Components Content sophistication Delivery Organization

Fourth Assignment Presentation Student Learning Outcome Components Content Delivery Organization

Rubrics Analytical Holistic - generally smaller assignments with fewer criteria, shorter turn around time.

Presentation Proficient Excellent Notable Student Learning Outcome Components Presentation Content Delivery Needs Work Proficient Excellent Unacceptable Notable Organization

F D C B A Presentation Student Learning Outcome Components Content Delivery Organization

Resource www.winona.edu/air/rubrics.htm

13. Look for and analyze patterns of data

Interpreting Data Consistency - over time Consensus - different populations Distinctiveness - different situations/ variables / items

Consistency Examines the same practice of and individual or group over time Key question: Has this person or group acted, felt, or performed this way in the past / over time?

Consistency How well are students performing on the learning outcome? High performance Low performance 05 06 07 08 09 10 11 12

Consensus Comparison to or among groups of students Key questions: Variation between disciplines, gender, other demographic variables Key questions: What is the general feeling, outcome, attitude, behavior? Do other groups of people act, perform or feel this way?

Consensus How well are students performing on the learning outcome? High performance Low performance Females Males Transfers OTA

Distinctiveness Examines individual or cohort perspectives across different situations, categories Key Question: Does a person or group respond differently based upon the situation, item, issue?

Distinctiveness How well are our students achieving our Core Learning Outcomes? High Performance Low Performance

One data point is not a pattern of evidence no matter how soon you have your reaccreditation visit.

14. Assessment results in Action R A

How Assessment Works Year 2 Year 3 Year 1 O U T C M E Compare Against Benchmarks, Standards, Targets, Past Performance New / Revised learning event New / Revised learning event learning event component BASELINE component component

Acting on Results Development Infrastructure Curriculum Faculty, Staff, Student Infrastructure Policy, Process, Planning Curriculum Learning Opportunities

The Seven Principles for Good Practice in Undergraduate Education 1. Student-Faculty Contact 2. Cooperative Learning 3. Active Learning 4. (Prompt) Feedback 5. Time on Task 6. High Expectations 7. Respect for Diverse Talents and Ways of Learning

15. Reporting Out R A

Learning Outcome: students will be able to <<action verb>><<something>> Components Grading criteria, Central traits of successful achievement Courses & Experiences list of courses and experiences supporting achievement of the outcome Assessment Point course, timeframe, experience 1 2 3 4. 5. 6 Learning Object Assignment, test, other student work Benchmark Acceptable performance level on assessment tool Assessment Tool Test score, national norm rubric, scale Assessment Point Benchmark Assessment tool Learning Object Assessment Point Benchmark Assessment tool Learning Object 3

<<object>> What we looked at <<object>> How we assess it <<assessment tool>> What we found <<results>> What it means <<interpretation>> What we’re going to do about it <<action>> What happened Outcome <<feedback>> 6