Making It Meaningful: Authentic Assessment for Intentional Education David W. Marshall, PhD Joanna M. Oxendine, MEd

Overview  Two Cultures: A Contrast in Emphasis  Defining Outcomes  Principles of Meaningful Assessment  Making Meaningful Change

Two Cultures: A Contrast in Emphasis

CCSF’s Assessment Cycle

The Assessment Cycle Write Outcomes Identify Assessment Measures Gather Evidence Analyze Results Strategize Program Improvement

Perception of the Assessment Cycle Write Outcomes Identify Assessment Measures Gather Evidence PACKAGE RESULTS SUBMIT REPORTS

The Culture of Compliance  Sees accreditation as an end in itself  Seeks information on what accreditors want to see  Worries about whether what they have matches accreditors’ expectations Students become unimportant elements of the assessment process

Another View of the Assessment Cycle Articulate Intended Outcomes Work Toward Intended Outcomes Gather Evidence Determine Extent to Which Outcomes Were Met Plan for Intentional Improvements

The Culture of Intentionality  Is student-centered  Seeks information about how well students are learning and/or how well various areas of the college are supporting the student experience  Reflects on what we teach or do and how we teach or do it  Accepts (some) responsibility for student learning and the student experience  Experiments with new strategies for student success Students become the primary focus of the assessment process

Evidenced-Based Change in the Context of a Culture of Intentionality

What is Evidence-Based Change? Evidence-based change is: An approach to organizational activities and decision-making that uses information about the effectiveness of those activities and decisions to implement strategic and targeted revisions towards increased impact of the organization’s key goals.

What is Evidence-Based Change? Evidence-based change is: An approach to organizational activities and decision-making that uses information about the effectiveness of those activities and decisions to implement strategic and targeted revisions towards increased impact of the organization’s key goals.

What is Evidence-Based Change? Evidence-based change is: An approach to organizational activities and decision-making that uses information about the effectiveness of those activities and decisions to implement strategic and targeted revisions towards increased impact of the organization’s key goals.

What is Evidence-Based Change? Evidence-based change is: An approach to organizational activities and decision-making that uses information about the effectiveness of those activities and decisions to implement strategic and targeted revisions towards increased impact of the organization’s key goals.

The Organization’s Key Goals?? Articulate Intended Outcomes Work Toward Intended Outcomes Gather Evidence Determine Extent to Which Outcomes Were Met Plan for Intentional Improvements

Defining Shared Purpose An approach to organizational activities and decision-making that uses information about the effectiveness of those activities and decisions to implement strategic and targeted revisions towards increased impact of the organization’s key goals.

An Analogy Those who:  Pray/Teach  Fight/Defend  Farm/Provide Food Mutuality

Taking the Analogy Too Far

Defining Outcomes

Two Types of Outcomes  Student Learning Outcomes  Statements that describe the knowledge, skills, attitudes, and habits of mind that students take with them from a learning experience (adapted from Suskie, 2009)  Service Area Outcomes  Statements that describe the intended results of specific programmatic, operational, or administrative activities  Two types at CCSF  Student Service Outcomes: statements that identify what a student will be able to know and/or do after receiving a service and/or participating in student support programs  Administrative Unit Outcomes: statements directly related to the college’s operations and non-academic services that support the learning environment Outcomes indicate means by which success in achieving goals might be defined.

Two Types of Outcomes Intentionality and Outcomes From the student perspective, outcome articulation:  Makes explicit for students what is expected of them so they know what to expect.  Removes the need for students to intuit what learning is expected. From the college’s perspective, outcome articulation:  Directs efforts to evaluate how successful activities have been.  Helps to identify what kind of information to collect. Outcomes indicate means by which success in achieving goals might be defined.

Outcomes and the Two Cultures CULTURE OF COMPLIANCE  Rarely communicates outcomes to students  Files outcomes with the appropriate office or administrative oversight entity  Sticks with what has always been done  Works on outcomes assessment for an accreditation cycle CULTURE OF INTENTIONALITY  Makes outcomes visible to students  Incorporates outcomes into practice  Assesses outcomes appropriately  Uses outcomes for ongoing conversations about effectiveness and continual improvement

Outcomes and the Two Cultures CULTURE OF COMPLIANCE  Rarely communicates outcomes to students  Files outcomes with the appropriate office or administrative oversight entity  Sticks with what has always been done  Works on outcomes assessment for an accreditation cycle CULTURE OF INTENTIONALITY  Makes outcomes visible to students  Incorporates outcomes into practice  Assesses outcomes appropriately  Uses outcomes for ongoing conversations about effectiveness and continual improvement

Principles of Meaningful Assessment

A Process of Questions 1. What do we want students to know, understand, and be able to do? What are the intended results of our programmatic, operational, or administrative activities? 2. Where do students learn what we expect them to learn? How do we accomplish what we set out to do? 3. How well did students learn what we expected them to learn? How well did we do what we intended to do? 4. How do we know how well they learned what we expected them to learn? How do we know how well we did what we expected to do? 5. What does our evidence tell us? How can we use what our evidence tells us to make appropriate decisions? The Culture of Intentionality opens a clear approach to assessment.

A Process of Questions 1. What do we want students to know, understand, and be able to do? What are the intended results of our programmatic, operational, or administrative activities? 2. Where do students learn what we expect them to learn? How do we accomplish what we set out to do? 3. How well did students learn what we expected them to learn? How well did we do what we intended to do? 4. How do we know how well they learned what we expected them to learn? How do we know how well we did what we expected to do? 5. What does our evidence tell us? How can we use what our evidence tells us to make appropriate decisions? The Culture of Intentionality opens a clear approach to assessment. Outcomes

A Process of Questions 1. What do we want students to know, understand, and be able to do? What are the intended results of our programmatic, operational, or administrative activities? 2. Where do students learn what we expect them to learn? How do we accomplish what we set out to do? 3. How well did students learn what we expected them to learn? How well did we do what we intended to do? 4. How do we know how well they learned what we expected them to learn? How do we know how well we did what we expected to do? 5. What does our evidence tell us? How can we use what our evidence tells us to make appropriate decisions? The Culture of Intentionality opens a clear approach to assessment. Outcomes Mapping

A Process of Questions 1. What do we want students to know, understand, and be able to do? What are the intended results of our programmatic, operational, or administrative activities? 2. Where do students learn what we expect them to learn? How do we accomplish what we set out to do? 3. How well did students learn what we expected them to learn? How well did we do what we intended to do? 4. How do we know how well they learned what we expected them to learn? How do we know how well we did what we expected to do? 5. What does our evidence tell us? How can we use what our evidence tells us to make appropriate decisions? The Culture of Intentionality opens a clear approach to assessment. Outcomes Mapping Alignment of Measures

A Process of Questions 1. What do we want students to know, understand, and be able to do? What are the intended results of our programmatic, operational, or administrative activities? 2. Where do students learn what we expect them to learn? How do we accomplish what we set out to do? 3. How well did students learn what we expected them to learn? How well did we do what we intended to do? 4. How do we know how well they learned what we expected them to learn? How do we know how well we did what we expected to do? 5. What does our evidence tell us? How can we use what our evidence tells us to make appropriate decisions? The Culture of Intentionality opens a clear approach to assessment. Outcomes Mapping Alignment of Measures Evidence

A Process of Questions 1. What do we want students to know, understand, and be able to do? What are the intended results of our programmatic, operational, or administrative activities? 2. Where do students learn what we expect them to learn? How do we accomplish what we set out to do? 3. How well did students learn what we expected them to learn? How well did we do what we intended to do? 4. How do we know how well they learned what we expected them to learn? How do we know how well we did what we expected to do? 5. What does our evidence tell us? How can we use what our evidence tells us to make appropriate decisions? The Culture of Intentionality opens a clear approach to assessment. Outcomes Mapping Alignment of Measures Evidence Closing the Loop

Making Meaningful Change

Responding to the Results If an outcome meets expectations at the standard:  Take no action  Identify this area as a program strength  Consider increasing expectations outlined in the outcome  Raise the standard of attainment  Consider surveying students about their experience of the program or service area Assessment may find that student learning or a program meets expectations at the determined standard for some outcomes.

Responding to the Results If results for an outcome do not meet expectations:  ASK QUESTIONS!!  Are our outcomes aligned with our expectations?  What aspect of our program or activities might be in need of revision?  Are our teaching and learning methods in need of revision?  With whom might we need to work to strengthen our outcomes?  What crazy idea might we come up with to tackle the problem? Assessment may find that student learning or a program does not meet expectations at the determined standard for some outcomes.

TOTAL ACROSS ALL SECTIONS Meets the outcome: 287 (74%) Developing the outcome: 74 (20%) No evidence: 21 (6%) CRN: Daphne: Met Jerome: Developing Janet: Not assessed Ben: Met Su-Lin: Developing Ahmed: Not assessed CRN: Daphne: Met Jerome: Developing Janet: Not assessed Ben: Met Su-Lin: Developing Ahmed: Not assessed CRN: Daphne: Met Jerome: Developing Janet: Not assessed Ben: Met Su-Lin: Developing Ahmed: Not assessed CRN: Daphne: Met Jerome: Developing Janet: Not assessed Ben: Met Su-Lin: Developing Ahmed: Not assessed CRN: Daphne: Met Jerome: Developing Janet: Not assessed Ben: Met Su-Lin: Developing Ahmed: Not assessed CRN: Daphne: Met Jerome: Developing Janet: Not assessed Ben: Met Su-Lin: Developing Ahmed: Not assessed MEETS THE SLO: Online: 45 (85%) Face-to-Face: 242 (74%) MEETS THE SLO: BOG Waiver: 192 (78%) No BOG Waiver: 65 (72%) MEETS THE SLO: Morning: 146 (78%) Afternoon: 96 (68%) PROGRAM IMPROVEMENTS! CCSF Progress on NEW ACCJC STANDARD for Disaggregated SLO Data MEETS THE SLO: Male: 161 (77%) Female: 123 (73%)

BIO 9 CurricUNET Data Processed by Office of Research (ARGOS) SLO: Critique scientific information in the media for accuracy and reliability. 287 Meets SLO (74%) Male: 161 (77%) Female: 123 (73%) Morning: 146 (78%) Afternoon: 96 (68%) BOG Waiver: 192 (78%) No BOG Waiver: 65 (72%) Online: 45 (85%) Face-to-face: 242 (74%) 74 Developing SLO (20%) Male: 37 (22%) Female: 36 (17%) Morning: 34 (18%) Afternoon: 53 (23%) BOG Waiver: 57 (21%) No BOG Waiver: 15 (18%) Online: 5 (9%) Face-to-face: 67 (20%) 21 No evidence of SLO (6%) Male: 8 (5%) Female: 13 (6%) Morning: 10 (5%) Afternoon: 12 (8%) BOG Waiver: 18(5%) No BOG Waiver: 3 (4%) Online: 3 (6%) Face-to-face: 22 (7%)

For meaningful change to occur… Faculty and staff must be the driving force and they must work collectively!  Faculty and staff:  know the curriculum or systems in which they work.  are responsible for seeing that outcomes are met.  will be the ones who innovate practice.  The administration:  creates space for the work to occur.  fosters the culture of intentionality.

Responding to the Results Students benefit from an institution’s thoughtful response to an honestly undertaken attempt to determine a program’s strengths and weaknesses. Articulate Intended Outcomes Work Toward Intended Outcomes Gather Evidence Determine Extent to Which Outcomes Were Met Plan for Intentional Improvements

To Conclude With an approach that strives for student-centered intentionality, compliance becomes a matter of simply completing forms and writing narratives, thereby accreditation takes care of itself. Articulate Intended Outcomes Work Toward Intended Outcomes Gather Evidence Determine Extent to Which Outcomes Were Met Plan for Intentional Improvements