Youngstown State University September 2011 Susan Hatfield Winona State University Winona, MN Assessing your Assessment Plan

YSU Added note: the first four slides are introduction slides

What is the focus of your assessment efforts?

Accreditation Reporting Number & Amount Quality & Utility Interpreting Learning Internal QuestionsExternal Questions Collecting it Using it ComplianceCommitment

Promoting a Culture of Learning

BEGINNING PROGRESS MATURING

Workshops, Speakers and Consultants on student learning Ongoing dialogue on student learning, development and measurement of learning outcomes Student learning has become central to the institution and student learning, performance, and achievement are celebrated Promoting a Culture of Learning BEGINNING PROGRESS MATURING

Activities Events SystemsProcesses Traditions Promoting a Culture of Learning BEGINNING PROGRESS MATURING

OccasionalUsual ExternalInternal AccreditationImprovement SurfaceEmbedded IndividualStructural IsolatedPervasive Promoting a Culture of Learning BEGINNING PROGRESS MATURING

OccasionalUsual ExternalInternal AccreditationImprovement SurfaceEmbedded IndividualStructural IsolatedPervasive LEADERSHIP KNOWLEDGE RESOURCES COMMUNICATION COMMITMENT Promoting a Culture of Learning BEGINNING PROGRESS MATURING

What are you measuring?

The relationship between values and measures Measures Values

Do you have measurable, meaningful program level student learning outcomes?

Reviewing Student Learning Outcomes A: Student Learning or Program Effectiveness?

What the student / faculty / staff will √ Know √ Do Assessing Assessing Learning

What the course / program /department / area will do or achieve Participation Support Improvement Placement Retention Development Satisfaction. Assessing Program Effectiveness

Satisfaction surveys Exit interviews & focus groups Observations Follow-up studies Retention and transfer studies Length of time to degree Efficiency, accuracy of services performed Number of students served, repeat visits Self report of impact on those served Graduation and transfer rates Placement rates.

Reviewing Student Learning Outcomes A: Student Learning or Program Effectiveness? B: Reasonable Number?

Exertion without Intention

Exhaustion

Intention without Exertion

Intention and Exertion

Outcomes can overlap

Outcome Common Core

Outcome Program A

Outcome Program B

Outcome Program C

Outcome Program D

Reviewing Student Learning Outcomes A: Student Learning or Program Effectiveness? B: Reasonable Number? C: Tied to mission / goals?

Mission General Education Academic Programs Vision Student Affairs External Agencies

Reviewing Student Learning Outcomes A: Student Learning or Program Effectiveness? B: Reasonable Number? C: Tied to mission / goals? D: Appropriate format?

Format for learning outcomes: Students / Faculty / Staff will be able to > >

Students / Faculty / Staff will be able to,, and,,, and.

Reviewing Student Learning Outcomes A: Student Learning or Program Effectiveness? B: Reasonable Number? C: Tied to mission / goals? D: Appropriate format? E: Cognitively appropriate?

KNOWLEDGE COMPREHENSION APPLICATION ANALYSIS SYNTHESIS EVALUATION Cite Count Define Draw Identify List Name Point Quote Read Recite Record Repeat Select State Tabulate Tell Trace Underline Associate Classify Compare Compute Contrast Differentiate Discuss Distinguish Estimate Explain Express Extrapolate Interpolate Locate Predict Report Restate Review Tell Translate Apply Calculate Classify Demonstrate Determine Dramatize Employ Examine Illustrate Interpret Locate Operate Order Practice Report Restructure Schedule Sketch Solve Translate Use Write Analyze Appraise Calculate Categorize Classify Compare Debate Diagram Differentiate Distinguish Examine Experiment Inspect Inventory Question Separate Summarize Test Arrange Assemble Collect Compose Construct Create Design Formulate Integrate Manage Organize Plan Prepare Prescribe Produce Propose Specify Synthesize Write Appraise Assess Choose Compare Criticize Determine Estimate Evaluate Grade Judge Measure Rank Rate Recommend Revise Score Select Standardize Test Validate

KNOWLEDGE COMPREHENSION APPLICATION ANALYSIS SYNTHESIS EVALUATION Cite Count Define Draw Identify List Name Point Quote Read Recite Record Repeat Select State Tabulate Tell Trace Underline Associate Classify Compare Compute Contrast Differentiate Discuss Distinguish Estimate Explain Express Extrapolate Interpolate Locate Predict Report Restate Review Tell Translate Apply Calculate Classify Demonstrate Determine Dramatize Employ Examine Illustrate Interpret Locate Operate Order Practice Report Restructure Schedule Sketch Solve Translate Use Write Analyze Appraise Calculate Categorize Classify Compare Debate Diagram Differentiate Distinguish Examine Experiment Inspect Inventory Question Separate Su rize Test Arrange Assemble Collect Compose Construct Create Design Formulate Integrate Manage Organize Plan Prepare Prescribe Produce Propose Specify Synthesize Write Appraise Assess Choose Compare Criticize Determine Estimate Evaluate Grade Judge Measure Rank Rate Recommend Revise Score Select Standardize Test Validate Lower level course outcomes

KNOWLEDGE COMPREHENSION APPLICATION ANALYSIS SYNTHESIS EVALUATION Cite Count Define Draw Identify List Name Point Quote Read Recite Record Repeat Select State Tabulate Tell Trace Underline Associate Classify Compare Compute Contrast Differentiate Discuss Distinguish Estimate Explain Express Extrapolate Interpolate Locate Predict Report Restate Review Tell Translate Apply Calculate Classify Demonstrate Determine Dramatize Employ Examine Illustrate Interpret Locate Operate Order Practice Report Restructure Schedule Sketch Solve Translate Use Write Analyze Appraise Calculate Categorize Classify Compare Debate Diagram Differentiate Distinguish Examine Experiment Inspect Inventory Question Separate Summarize Test Arrange Assemble Collect Compose Construct Create Design Formulate Integrate Manage Organize Plan Prepare Prescribe Produce Propose Specify Synthesize Write Appraise Assess Choose Compare Criticize Determine Estimate Evaluate Grade Judge Measure Rank Rate Recommend Revise Score Select Standardize Test Validate Advanced Course / Program outcomes

Student Learning Outcomes Students will have a basic knowledge of human anatomy.

Student Learning Outcomes Interpret a financial statement

Student Learning Outcomes Question inconsistent results

Student Learning Outcomes Demonstrate the ability to solve problems, to work independently and to work as members of a team.

Student Learning Outcomes Be recognized for excellent preparation and will be vigorously recruited by employers.

Student Learning Outcomes Students will be able to identify, define, and evaluate……

Student Learning Outcomes Students will be able to identify, define, and evaluate……

Student Learning Outcomes Students will be able to select, describe, and interpret…..

Reviewing Student Learning Outcomes A: Student Learning or Program Effectiveness? B: Reasonable Number? C: Tied to mission / goals? D: Appropriate format? E: Cognitively appropriate? F: Do faculty agree on the definition of the outcome? (components)

Components What are you looking for in student work to be able to tell if they “get it” Grading criteria

Goal

Outcome

Communication RelateSpeakListenParticipateWrite

Component Communication RelateSpeakListenParticipateWrite

Component Communication RelateSpeakListenParticipateWrite

Component Communication RelateSpeakListenParticipateWrite

Component Communication RelateSpeakListenParticipateWrite

Component Communication RelateSpeakListenParticipateWrite

The Reality of Assessing Learning Outcomes: Why you need common components.

Speaking

volume poise conclusion Speaking teacher1

volume poise conclusion Speaking teacher2 teacher1 gestures rate evidence

volume poise conclusion Speaking teacher2 teacher1teacher3 gestures rate evidence sources examples organization

volume poise conclusion Speaking teacher4teacher2teacher1teacher3 eye contact style appearance gestures rate evidence sources examples organization

volume poise conclusion Speaking teacher5 eye contact style appearance gestures rate evidence sources examples organization transitions verbal variety attention getter teacher4teacher2teacher1teacher3

volume poise conclusion eye contact style appearance gestures rate evidence sources examples organization transitions verbal variety attention getter Can our students deliver an effective Public Speech?

volume poise conclusion eye contact style appearance gestures rate evidence sources examples organization transitions verbal variety attention getter Can our students deliver an effective Public Speech?

volume poise conclusion eye contact style appearance gestures rate evidence sources examples organization transitions verbal variety attention getter Can our students deliver an effective Public Speech?

Are your outcomes differentiated by degree?

Calculate X Calculate X Calculate X Calculate X Apply C Apply C Apply C Apply C Describe Y Describe Y Describe Y Describe y Evaluate W Evaluate W Evaluate W Evaluate W Diagnose Z Diagnose Z Diagnose Z Diagnose Z CertificateAssociatesBachelorsMasters

Define X Name C Identify Y Recite W List Z Describe X Translate C Compare Y Report W Explain Z Apply X Produce C Calculate Y Evaluate W Manage Z Evaluate X Validate C Synthesize y Create W Diagnose Z CertificateAssociatesBachelorsMasters

Calculate X Calculate X Calculate X Calculate X Apply C Apply C Apply C Apply C Describe Y Describe Y Describe Y Describe y Evaluate W Evaluate W EvaluateSEvaluateS Evaluate W Diagnose Z Diagnose Z Diagnose Z Diagnose Z Major 1Major 2Major 3Major 4

Calculate X Calculate X Calculate X Calculate X Describe Y Describe Y Describe Y Describe y Diagnose Z Diagnose Z Diagnose Z Diagnose Z Major 1Major 2Major 3Major 4

Are your outcomes supported by the curriculum?

Program Level Student Learning Outcomes

1xx Program Level Student Learning Outcomes 1xx 2xx 3xx 4xx Capstone

1xx Program Level Student Learning Outcomes 1xx 2xx 3xx 4xx Capstone

1xx X X X X X X X Program Level Student Learning Outcomes 1xx X X X X X X X 2xx X X X X X X X X X X X X X X X X X X X X X 3xx X X X X X X X X X X X X X X X X X X X X X 4xx X X X X X X X Capstone X X X X X X X

1xx X X X Program Level Student Learning Outcomes 1xx X X X 2xx X X X X X 3xx X X X X X X 4xx X X X X Capstone X X

1xx K K K Program Level Student Learning Outcomes K= Knowledge/Comprehension; A= Application / Analysis; S= Synthesis /Evaluation 1xx S K K 2xx A A S A A K 3xx A K A A K A 4xx S A K S Capstone S S

1xx K K K Program Level Student Learning Outcomes K= Knowledge/Comprehension; A= Application / Analysis; S= Synthesis /Evaluation 1xx S K K 2xx A A S A A K 3xx A K A A K A 4xx S A K S Capstone S S

1xx K K K Program Level Student Learning Outcomes K= Knowledge/Comprehension; A= Application / Analysis; S= Synthesis /Evaluation 1xx S K K 2xx A A S A A K 3xx A K A A K A 4xx S A K S Capstone S S

1xx K K K Program Level Student Learning Outcomes K= Knowledge/Comprehension; A= Application / Analysis; S= Synthesis /Evaluation 1xx S K K 2xx A A S A A K 3xx A K A A K A 4xx S A K S Capstone S S

1xx K K K Program Level Student Learning Outcomes K= Knowledge/Comprehension; A= Application / Analysis; S= Synthesis /Evaluation 1xx S K K 2xx A A S A A K 3xx A K A A K A 4xx S A K S Capstone S S

1xx K K K Program Level Student Learning Outcomes K= Knowledge/Comprehension; A= Application / Analysis; S= Synthesis /Evaluation 1xx S K K 2xx A A S A A K 3xx A K A A K A 4xx S A K S Capstone S S

1xx K K K Program Level Student Learning Outcomes K= Knowledge/Comprehension; A= Application / Analysis; S= Synthesis /Evaluation 1xx S K K 2xx A A S A A K 3xx A K A A K A 4xx S A K S Capstone S S

1xx K K K Program Level Student Learning Outcomes K= Knowledge/Comprehension; A= Application / Analysis; S= Synthesis /Evaluation 1xx S K K K 2xx A A A A A K K 3xx A K K K A A S S A S S A 4xx S A S S Capstone S S option cluster

1xx K K K Program Level Student Learning Outcomes K= Knowledge/Comprehension; A= Application / Analysis; S= Synthesis /Evaluation 1xx S K K K 2xx A A A A A K K 3xx A K K K A A S S A S S A 4xx S A S S Capstone S S

1xx K K K Program Level Student Learning Outcomes 1xx S K K K 2xx A A A A A K K 3xx A K K K A A S S A S S A 4xx S A S S Capstone S S Build a curriculum K= Knowledge/Comprehension; A= Application / Analysis; S= Synthesis /Evaluation

1xx K K K Program Level Student Learning Outcomes 1xx S K K K 2xx A A A A A K K 3xx A K K K A A S S A S S A 4xx S A S S Capstone S S Diagnose Learning Analyze Curriculum K= Knowledge/Comprehension; A= Application / Analysis; S= Synthesis /Evaluation

1xx K K K Program Level Student Learning Outcomes 1xx S K K K 2xx A A A A A K K 3xx A K K K A A S S A S S A 4xx S A S S Capstone S S Diagnose Learning K= Knowledge/Comprehension; A= Application / Analysis; S= Synthesis /Evaluation

Do students know the program has outcomes?

Program Level Outcomes In the catalog On the website In department related publications On the syllabi

Do you have a plan for assessing the outcomes?

Assessment Plan Assessment does NOT mean Every outcome for Every student in Every course Every semester by Every faculty member

1xx K K K Program Level Student Learning Outcomes K= Knowledge/Comprehension; A= Application / Analysis; S= Synthesis /Evaluation 1xx S K K K 2xx A A A A A A K 3xx A K K A A A S S A S S A 4xx S A S S Capstone S S S

1xx K K K Program Level Student Learning Outcomes K= Knowledge/Comprehension; A= Application / Analysis; S= Synthesis /Evaluation 1xx S K K K 2xx A A A A A A K 3xx A K K A A A S S A S S A 4xx S A S S Capstone S S S

1xx K K K Program Level Student Learning Outcomes K= Knowledge/Comprehension; A= Application / Analysis; S= Synthesis /Evaluation 1xx S K K K 2xx A A A A A A K 3xx A K K A A A S S A S S A 4xx S A S S Capstone S S S

1xx K K K Program Level Student Learning Outcomes K= Knowledge/Comprehension; A= Application / Analysis; S= Synthesis /Evaluation 1xx S K K K 2xx A A A A A K K 3xx A K K K A A S S A S S A 4xx S A S S Capstone S S

1xx K K K Program Level Student Learning Outcomes K= Knowledge/Comprehension; A= Application / Analysis; S= Synthesis /Evaluation 1xx S K K K 2xx A A A A A K K 3xx A K K K A A S S A S S A 4xx S A S S Capstone S S

1xx K K K Program Level Student Learning Outcomes K= Knowledge/Comprehension; A= Application / Analysis; S= Synthesis /Evaluation 1xx S K K K 2xx A A A A A K K 3xx A K K K A A S S A S S A 4xx S A S S Capstone S S

How will know if your students have learned?

Student Learning Outcome

Student Learning Outcome Test Questions

Student Learning Outcome Assignment

Student Learning Outcome Assignment

Student Learning Outcome Experience

Matching Outcomes to Objects Logical decision ? Individual vs. group assignment ? Amount of feedback provided in preparation of the assignment ? Timing / program / semester

Student Learning Outcome

Student Learning Outcome Test Questions

Student Learning Outcome Assignment

Student Learning Outcome Experience

Are you studying patterns of evidence?

Interpreting Data Consistency - over time Consensus - different populations Distinctiveness - different situations/ variables / items

Interpreting Data Consistency - over time Consensus - different populations Distinctiveness - different situations/ variables / items

Consistency Examines the same practice of and individual or group over time Key question: Has this person or group acted, felt, or performed this way in the past / over time?

Consistency How well are students performing on the learning outcome? High performance Low performance

Consistency How many students utilize our services? High performance Low performance

Consistency High performance Low performance How many students utilize our services?

Consensus Comparison to or among groups of students Variation between disciplines, gender, other demographic variables Key questions: What is the general feeling, outcome, attitude, behavior? Do other groups of people act, perform or feel this way?

Consensus How well are students performing on the learning outcome? FemalesMalesTransfersOTA High performance Low performance

Consensus How effectively are we meeting the needs of our students ? First GenInt’lNon residentialDistance High performance Low performance

Distinctiveness Examines individual or cohort perspectives across different situations, categories Key Question : Does a person or group respond differently based upon the situation, item, issue?

Distinctiveness How well are our students achieving our Learning Outcomes? High Performance Low Performance SLO3SLO4SLO2SLO5SLO1SLO6

Distinctiveness How satisfied are our student with the services we provide? High Performance Low Performance S3S4S2S5S1S6

Are we measuring achievement of the outcomes?

Low performance High performance SLO3SLO4SLO2SLO5SLO1SLO6 Have our students achieved our learning outcomes?

SLO3SLO4SLO2SLO5SLO1SLO6 Low performance High performance Have our students achieved our learning outcomes?

Have our students achieved our learning outcomes? SLO3SLO4SLO2SLO5SLO1SLO6 Low performance High performance

What are you doing to close the loop?

Closing the Loop

Development Faculty, Staff, Student Infrastructure Policy, Process, Planning Curriculum Learning Opportunities

Closing the Loop Development Faculty, Staff, Student Infrastructure Policy, Process, Planning Curriculum Learning Opportunities

Closing the Loop Development Faculty, Staff, Student Infrastructure Policy, Process, Planning Curriculum Learning Opportunities

Closing the Loop Development Faculty, Staff, Student Infrastructure Policy, Process, Planning Curriculum Learning Opportunities

The Seven Principles for Good Practice in Undergraduate Education 1. Student-Faculty Contact 2. Cooperative Learning 3. Active Learning 4. (Prompt) Feedback 5. Time on Task 6. High Expectations 7. Respect for Diverse Talents and Ways of Learning Zelda Gamson and Arthur Chickering Wingspread Journal 1987

Increase Reduce Improve Determine Maintain

Increase Reduce Improve Determine Maintain the Number Amount Percentage Ratio Level Rate

Increase Reduce Improve Determine Maintain Number Amount Percentage Ratio Level Rate Someone Students Faculty Staff Community members Employer of the

Increase Reduce Improve Determine Maintain Number Amount Percentage Ratio Level Rate Someone Students Faculty Staff Community members Employer Something applications grants Interviews request workshops reports satisfaction engagement of the

Increase Reduce Improve Determine Maintain Number Amount Percentage Ratio Level Rate Someone Students Faculty Staff Community members Employer Something applications grants Interviews request workshops reports satisfaction engagement > Something of who that with the

Increase Reduce Improve Determine Maintain Number Amount Percentage Ratio Level Rate Something applications grants Interviews request workshops reports satisfaction engagement Contribute to the annual campaign of who that with Someone Students Faculty Staff Community members Employers > Something the

Increase Reduce Improve Determine Maintain Number Amount Percentage Ratio Level Rate Someone Students Faculty Staff Community members Employer Something applications grants Interviews request workshops reports satisfaction engagement Submit Grant Applications of who that with > Something the

Increase Reduce Improve Determine Maintain Number Amount Percentage Ratio Level Rate Paper, Glass and Plastic Are not recycled of who that with Someone Students Faculty Staff Community members Employers > Something the

Increase Reduce Determine Maintain Number Amount Percentage Ratio Level Rate of who that with Someone Seniors Faculty Staff Community members Employers > Something Conduct Independent Research Something the

Increase Reduce Determine Maintain Number Amount Percentage Ratio Level Rate of who that with Someone Seniors Faculty Staff Community members Employers > Something Pass the licensing exam on the first try Something the

Increase Reduce Determine Maintain Number Amount Percentage Ratio Level Rate of who that with Someone Seniors Faculty Staff Community members Employers > Something Pass the licensing exam on the first try Something theby Action

Educating….. Assisting ….. Explaining ….. Assisting ….. Developing ….. Consulting ….. Delivering ….. Facilitating ….. Creating ….. Nurturing ….. Negotiating ….. Sponsoring ….. Training ….. Translating ….. Validating ….. Updating ….. Enforcing ….. Testing ….. Assessing ….. Reporting ….. Issuing ….. Communicating ….. Integrating ….. Responding ….. Arranging ….. Coordinating ….. Providing ….. Organizing ….. Producing ….. Training ….. Maintaining ….. Supporting ….. Directing ….. Designing ….. Collecting ….. Developing ….. Responding ….. Reviewing ….. Studying ….. Distributing ….. Executing ….. Recruiting ….. Generating ….. Implementing ….. Investigating ….. Designing ….. Collaborating ….. Revising ….. Scheduling ….. Sharing ….. Increase Reduce Improve Determine Maintain Number Amount Percentage Ratio Level Rate Something applications grants Interviews request workshops reports satisfaction engagement by of who that with Someone Students Faculty Staff Community members Employers > Something the

Increase Reduce Determine Maintain Number Amount Percentage Ratio Level Rate of who that with Someone Seniors Faculty Staff Community members Employers > Something Pass the licensing exam on the first try Something theby Action

Increase Reduce Determine Maintain Number Amount Percentage Ratio Level Rate of who that with Someone Seniors Faculty Staff Community members Employers > Something Pass the licensing exam on the first try Something theby Action

Youngstown State University September 2011 Susan Hatfield Winona State University Winona, MN Assessing your Assessment Plan