Outcomes assessment Basics Dan Stroud May 25, 2018 Image taken from: http://www.everyday-thai.com/learn_thai_online/speak/intermediate_lessons.html

Word Cloud image take from http://www.assessment.uconn.edu/

A Vision for Assessment Commitment to the Outcome To provide sufficient support and guidance to help you realize the dividends for the time/effort invested Enhanced learning Improved programs/degrees Greater communication about teaching/learning among faculty To create a culture of learning, where striving to enrich our students’ learning is ALWAYS what is PROJECTED

Some Guiding Assumptions… Teaching and learning can be improved through systematic inquiry Assessment is always a work in progress, and it’s ok if things don’t go perfectly Assessment is about lessons learned in the efforts to enhance learning/teaching Goal of the Assessment Annual Report = To demonstrate concerted effort on the part of faculty to examine student outcomes and make appropriate adjustments to improve program

FIVE “Big Picture” questions to ask When conducting assessment How do you define a successful student in your program? What have you learned about your students’ learning? What Evidence do you have that proves their skill level? How are you using that evidence to improve Student understanding in your program? After implementing a plan of action, how did it change the resulting skill level? Slide taken from Debbie Smith’s 2010 presentation at UMKC, “I Survived Assessment”

Assessing Our University’s (& Your Department’s) Assessment Efforts Compliance Commitment External Questions Internal Questions Number & Amount Quality & Utility Reporting Interpreting Slide taken from Susan Hatfield’s Presentation on “Coaching Assessment: Student Learning Outcomes” Collecting it Using it Accreditation Learning

Assessment Components for Academic Degree Programs Mission statement Learning Outcomes (usually 4-6) Remember: SMART Specific Measurable Attainable Relevant/Results- Oriented Time-bound Specific -Clear and definite terms describing the abilities, knowledge, values, attitudes, and performance Measurable -It is feasible to get data: data are accurate and reliable; it can be assessed in more than one way Aggressive and Attainable -The outcome has the potential to move the program or unit forward Results – oriented -Describe what standards are expected from students or the functional area being assessed Time-bound -Describe a specified time period for accomplishing the outcome

Measurements Complete Measurements Process What instrument? why? formative and summative assessment? direct and indirect measure? When possible, Multiple measures can enhance assessment and make it more meaningful to program How conduct measurement? which students? when measured? where? how administered? by whom? often good to use smaller samples of students; capstone courses How collect and store data? Who analyzes data? how? when?

Achievement Targets What kind of performance do you expect from your students Learning? What should the Outcomes be? What is the desirable level of performance for your students ? Rubrics can clarify this (see the next slides) What percentage of students do you expect to achieve this?

Using Rubrics A rubric is: “a set of criteria and a scoring scale that is used to assess and evaluate students’ work” (Cambell, Melenyzer, Nettles, & Wyman, 2000). Addresses performance standards in a clear and concise manner (which students appreciate!) Clearly articulates to students the areas of improvement needed to meet these standards To find examples, Google rubrics for your discipline Cambell, D. M., Melenyzer, B. J., Nettles, D. H., & Wyman, R. M. (2000). Portfolio and performance assessment in teacher educations. Boston: Allyn and Bacon.

Example of a Rubric Foreign Languages and Literatures Assessment Tool for Oral Proficiency Interview adapted from “Interpersonal Mode Rubric Pre-Advanced Learner” 2003 ACTFL Category Exceeds Expectations Meets Expectations Does Not Meet Expectations Comprehensibility Who can understand this person’s meaning? How sympathetic must the listener be? Does it need to be the teacher or could a native speaker understand the speaker? How independent of teaching situation is the conversation? Easily understood by native speakers, even those unaccustomed to interacting with language learners. Clear evidence of culturally appropriate language, Although there may be some confusion about the message, generally understood by those unaccustomed to interacting with language learners. Generally understood by those accustomed to interacting with language learners. Language Control Accuracy, form, appropriate vocabulary, degree of fluency High degree of accuracy in present, past and future time. Accuracy may decrease when attempting to handle abstract topics Most accurate with connected discourse in present time. Accuracy decreases when narrating and describing in time frames other than present.   Most accurate with connected sentence-level discourse in present time. Accuracy decreases as language becomes complex.

How to build a rubric Answer the following questions: Given your broad course goals, what determines the extent of student understanding? What criterion counts as EVIDENCE of student learning? What specific characteristics in student responses, products or performances should be examined as evidence of student learning? Slide from Molly Mead and Erica Schurter presentation entitled, “Using Rubrics to Make Assessment for Efficient”

For example: Can our students deliver an effective Public Speech? Developing a rubric helps you to clarify the characteristics/components of your Learning Outcomes: For example: Can our students deliver an effective Public Speech? Slide taken from Susan Hatfield’s Presentation on “Coaching Assessment: Student Learning Outcomes” eye contact gestures volume sources transitions style rate poise examples verbal variety appearance evidence conclusion organization attention getter

More rubric help AACU Rubrics http://www.aacu.org/value/rubrics Rubrics from Susan Hatfield (Accreditation Mentor) www.winona.edu/air/rubrics.htm Rubistar http://rubistar.4teachers.org/

Findings Part I: specific findings What do the data tell you? Compare new data to achievement targets Did students meet or deviate from expectations? Important: Include specific numbers/percentages when possible Do not use course grades or pass rates.

Findings (cont.) Part II: general findings Conversations what do the data tell you? Part II: general findings What lessons did your faculty learn from this evidence about your students? What broader implications do you draw about your program? Ex: curriculum, admissions, administration, policies, requirement, pedagogy, assessment procedures, and so on Conversations The more people involved, the better!

Action Plans Concrete Steps for Change list of specific innovations that you would like to introduce in AY 2013-14 to address lessons learned in AY 2012-13. Again, in curriculum, admissions, administration, policies, requirement, pedagogy, assessment procedures, and so on Resources? Time Period? Point Person? It is best to have documentation of the changes made through these Action Plans (e.g., in syllabi, the course catalogue, meeting minutes)

Don’t Forget the Assessment Annual Report Part I: Detailed Assessment Report “Assessment Narrative All items (mission -> action plans) submitted in Assessment Management System folder Assessment Deadline December 1 Annually

Assessment Plan Narrative Part II: Timeline/Account of Activities “Assessment Plan Narrative” In 1-2 pages, tell the story of all the work and careful consideration you and your colleagues accomplished in your assessment work this year (Ex.: meetings, mentoring, experiments, setbacks, lessons learned) Plug into your Assessment Folder for that year Please follow the four outlined questions (see next slide)

Four Questions for the Assessment Narrative Process: Please describe the specific activities and efforts used to design, implement, and analyze your assessment plan during this academic year. This narrative might be organized chronologically, listing meetings, mentoring sessions, and experiments at each stage of the developmental process including the names of people involved in various capacities, with each event given one paragraph. Positives: Please describe what was most useful about the assessment process, or what went well.  What did you learn about your faculty, students, or program through this experience?  Challenges: Please describe the challenges you encountered in terms of the development or implementation of your assessment procedures, as well as the lessons you learned from this experience and your efforts or plans for overcoming them. This section might be organized topically. Support: Please describe your program’s experience during the past year with the support and administrative structures in place at MSU for Assessment:  the Provost’s Office, the University Assessment Committee, the Institutional Research and Assessment office, and so on.

Submission: December 1st No edits allowed after 1st of December After this date, the Peer Review process will commence with feedback returned by approximately March 1st to the programs. Also After this date, information gathered and reported can be loaded into the next year’s assessment folder. The previous year’s Narrative should be held in the new folder until end of year reporting is complete for that cycle.

After December 1st Assessment entries for AY 2013-14 begin Assessment Cycle runs from Approximately August 15 (End of Summer II for a 12 month cycle. Necessary to implement the Action discussed in the Narrative and recorded in the previous year’s assessment plan. Update mission statements, goals, learning outcomes, and measurements based on feedback from UAC. Items in the new assessment folder will carry over from last year unless changed, with the exception of previous year’s documentation. Begin Entering new findings and formulating new action plans.

Questions? Image taken from http://expeditedselling.com/guiding-tips/questions-to-ask-before-using-crm/

Contact Information For assistance with assessment, please contact Dr. Dan Stroud, Assessment Specialist at 397-4742 or daniel.stroud@mwsu.edu.