Improving Undergraduate Education through the Assessment of Student Learning SACS-COC Institute on Quality Enhancement and Accreditation July 30, 2006 Peter T. Ewell National Center for Higher Education Management Systems

Looking Back: Origins of the Assessment Movement Two Decades Ago Undergraduate Reform Reports of 1985-86 Internal Stimulus: Call for More Coherent Teaching/Learning Approaches and Information for Improvement External Stimulus: Stakeholder Demands for Information on “Return on Investment” Tensions in Motive and Message Ever Since

Why Didn’t Assessment Go Away? Pressure to Produce Evidence of Student Learning Outcomes Never Let Up By Early 1990s, Accreditors Replace States as Primary External Stimulus to Get Started Intermittent Federal Interest in Assessment as an Element of National Accountability But Resulting Faculty Ambivalence About a Process Seen as “External” and “Administrative”

Looking Back: What’s Been Accomplished? Assessment Is for the Most Part Perceived as Inevitable and Legitimate Vast Majority of Institutions Have Statements of Learning Outcomes (General and Programmatic) A “Semi-Profession” of Folks Involved in Assessment Steadily Growing Sophistication with Respect to Methods of Gathering Evidence

Looking Back: What Hasn’t Happened? Authentic Integration of Assessment into Faculty Cultures and Behaviors Assessment Activities Still Largely “Added On” to the Curriculum Instead of Being Embedded In It Systematic and Widespread Use of Assessment Results for Institutional and Curricular Improvement Proactive and Sincere Institutional Engagement with Accrediting Organizations Around Topics of Assessment

Assessment as a “Perpetual Movement?” Most Social Movements… Fade as Fads, or Go Away Because Core Ideas are Mainstreamed The “Assessment Movement” Has Done Neither... External Requirements Keep it Alive But External Pressures also Constrain Faculty Buy-In and Meaningful Institutional Use Moving Beyond a “Perpetual Movement” Will Demand... Making Assessment Real to Faculty by Connecting It to the Actual Practice of Teaching and Learning Re-Focusing Accountability on the Authentic Student Abilities that Society Says it Needs

Some Prominent Changes in Higher Education’s Operating Environment Doing More with Less Need for Curricula that are Effective and Efficient Need Information About Curricular Functioning to Enable Effective Action Changes in Instructional Delivery Competencies and “Deep Learning” Student Engagement and Role of Technology New Views of Accountability “Quality”— From Inputs/Processes to Results Stakeholder Voices — Students and Employers

Challenges to the Academy: The Internal Dimension Changing Paradigm of Teaching and Learning Resulting Changes in Academic Roles, Behaviors and Structures Nature and Role of “Faculty” in Instruction Patterns of Student Enrollment [Technology as a “Wild Card”]

A Changing Paradigm of Teaching and Learning From “Faculty Teaching” to “Students Learning” Students “Make Their Own Paths” through Multiple Learning Opportunities Explicit Designs for Learning Based on Research

Changing Paradigm: Possible Lines of Response Demonstrated Achievement Becomes Paramount, not “Seat Time” Students “Teach” One Another Individual (and Asynchronous) Paths Technology Seen as Opportunity to Re- Think [not as a “Solution”]

Some Implications for Student Assessment Processes Assessments Reinforce Common Standards for Learning Across Curricula and Classes “Seamless” Assessments Become an Integral Part of Curriculum and Pedagogy Assessments Emphasize Connections and Longitudinal Development, not Just Attainment

Assessment Approaches: A Resulting Shift in Emphasis Accountability-Based: Assessments Added onto Instruction to “Check Up” on the System in the Aggregate Scholarship and Improvement: Assessments Built into the System to Simultaneously Assure Standards and Provide Feedback on Collective Performance

Kinds of Information Needed Alignment of Key Learning Outcomes Across Units, Sequences, and Courses Match Between Curricular Design, Delivery, and Student Experience Match Between Instruction and Needs of Diverse “Student Bodies” Effectiveness of Particular Innovations and Interventions

Changing Nature of the Faculty Role Dis-aggregation of Instructional Roles More Things for Faculty to Do New Potential Career Patterns and Paths

New Faculty Roles: Possible Lines of Response Emphasize Peer Support and Collaboration in Faculty Development Recognize and Regularize Alternative Career Paths Recognize “Mentorship” as the One Thing You Can’t Responsibly Outsource

Kinds of Information Needed Capturing and Re-Aggregating Data About Discrete Instructional Functions Accounting Technology-Based Costs Tracking Faculty “Assets”

Changing Patterns of Student Attendance Increased Levels of Multi-Institutional Attendance Increased Complexity in Course-Taking Behavior within Institutions Greater (and Unpredictable) Time Lapses Between Instructional Encounters

Changing Patterns of Attendance: Possible Lines of Response Coherence Based on Common Practices [and Outcomes], not Common Content Stress Ways for Students to Exploit and Reflect on Their Own Experiences Establish Clear Transition Points at which to Assess Student Mastery of Key Concepts

Kinds of Information Needed Relationships Between Particular Institutional Experiences and Particular Outcomes Tracking Student Learning Styles and Individual Paths of Development “Episode-based” [as opposed to time-based] Data Structures

Challenges to the Academy: The External Dimension Increased Accountability Changing Expectations Regarding What Students Know and Can Do The “New Competition”

What Are States Doing? Forces Influencing State Approaches Decreased Agency Capacity Due to Funding Cuts Momentum of K-12 Reform (NCLB) Political Uncertainty and Instability Types of State Policy Responses P-16 Alignment Performance Measures and “Report Cards” Achievement Testing (Driven by K-12)

What’s Shaping Accreditation? Forces Influencing Accreditors Pressure for Specific Performance from Federal Government Demands from Institutions to “Add Value” New Models from Other Sectors and Abroad “New Looks” in Accreditation Focus on Outcomes and Effectiveness Presenting Evidence [e.g. “Portfolios”] Review Approaches [e.g. “Academic Audits”] Connection to Institutional Planning [e.g. QEP]

Learning Outcomes: What Employers Expect Higher-Order “Literacies” as Well as Specific Skills Framed in Terms of “Practice” [not “Knowledge”] As Much About Attitudes as Academics [“Soft Skills”]

External Forces: Some Resulting Influences on the Academy Increased Emphasis on Credentialing Modularity and Acceleration to Increase Accessibility Pressure to Respond to “Students as Customers” Accountability “Superstructures” that Divert Attention and Information Resources

Kinds of Information Needed Outcomes and Performance Measures Data on What Experiences/Services Students Can Expect Peer Comparisons [Increasingly Outside the Academy] and Comparative Performance Needs and Satisfaction of External Stakeholders

Attributes of a Meaningful “Culture of Evidence” Shared Recognition That Many (But Not All) Things Are Knowable An Accessible Store of Information About Organizational Condition and Performance An Attitude Toward Problem-Solving that Minimizes “Finger-Pointing” Clear Follow-Through On Decisions Made and Why They Were Taken

Cultures of Evidence: Success Factors Visible Metaphor of Scholarship Beginning with Real Problems and Processes, not with “Method” Consistent Messages from Leadership Periodically “Re-Socializing” the Community “Closing the Loop” with Action Turn now to what we think we have learned about some of the practicalities of making assessment processes work…”tactics” of assessment in any setting.. - Begin with factors inducing and inhibiting success.. - Then move to the particularly delicate topic of what the relationship ought to be between quality assurance agencies and institutions (or internally between assessment offices and units/faculties) in order to get real engagement and improvement.. [SLIDE]

Cultures of Evidence: Inhibiting Factors Either Excessive or Non-Existent Consequences Alien Language and “Management Culture” Excessive Complexity Burnout and “Committee Fatigue” Changing the Rules Noted lots of things wrong as well …and note particularly that our experience suggests that institutions (like learners) will not always follow a linear path of development here--may fall back or forget things …need to be constantly vigilant to be effective [SLIDE]

Meaningful Assessment is More About Mindset than Method Questions About Learning are not Just Matters of Opinion What Information Might We Collect? What Might We Expect to Find? What Difference Would Finding Out Make? Assessment is About Improving Practice What are We Trying to Fix? How Good is Good Enough? What Changes are Implied?

The Bottom Line For Internal Management, “Seat of the Pants” Decision-making is no Longer Sufficient Information Used Openly, Consistently, and Continuously to Inform Academic Decisions For External Constituencies, “Trust Me” is no Longer Sufficient Need Clear, Understandable Evidence of Student Academic Attainment Be Vigilant about the Information You Choose and the Signals it Sends Make Sure that What You Measure is What You Value Harness the Accreditation Process to Make it Happen!