Creating Academic Cultures of Evidence: A Perspective on Learning Outcomes Peter T. Ewell National Center for Higher Education Management Systems (NCHEMS) Simon Fraser University 28 September, 2011

Some Topics We Will Discuss Today Conceptual Basis of an Outcomes Approach: Strengths, Challenges, Core Concepts Forces Shaping Outcomes and Assessment Building Authentic “Cultures of Evidence” Using Assessment Information to Improve Curriculum and Instruction

A Taxonomy of Terms Commonly Used in Connection with the “Assessment” of Student Learning Outcomes Units of Analysis Institution Program Student Ways of Looking at Performance Efficiency Productivity Effectiveness Output Productivity Outcome Ways of Looking at Outcomes Satisfaction Learning  Knowledge  Skill  Ability  Attitude/Disposition Attainment Development Ways to Review Performance Evaluation Measurement Indicator Assessment Evidence of Achievement  Examinations  Performances  Student Work Behaviors  Employment  Further Education  Career Mobility  Income

Advantages of an Outcomes Approach Clarity of Goals and Objectives Flexibility Across Different Modes of Provision Allows Comparisons of Performance Across Populations, Treatments, and Settings Supports Establishment of Portable Credentials Across Institutions and Jurisdictions

Pitfalls of an Outcomes Approach Definitions of Outcomes [Conceptual and Operational] Legitimacy Among Traditional Academics Fractionation: Losing a Sense of the Whole with Respect to Complex Abilities Loss of Serendipity Because Not All Important Outcomes May be Specifiable in Advance

Assessing Student Learning Outcomes: The Basics “Assessment” Means Gathering Systematic Information About What [Undergraduate] Students Know and Can Do as a Result of Attending College Generally Driven by Formal Statements of Intended “Outcomes” (e.g. Oral Communications, Quantitative Reasoning, or Problem-Solving) Developed by the Faculty Usually Collected in the Aggregate (that is, to Inform Judgments About Programs, not People)

A Taxonomy of Approaches to Assessment Learning/Teaching (Formative) Instruction Individual Tests Portfolios Placement Diagnostic Tests Advanced Placement Tests Vocational Tests Accountability (Summative) “Gate-keeping” Rising Junior Exams Comprehensive Exams Certification Exams Capstone Performances Program Enhancement Individual assessment results may be aggregated to serve program evaluation purposes Campus and Program Evaluation Productivity Reviews Performance Indicators Individual Group Assessment Level Assessment Focus or Purpose

Continuous Improvement Continuous Improvement Accountability Accountability Strategic dimensions PurposeFormative (improvement)Summative (judgment) OrientationInternalExternal MotivationEngagementCompliance Implementation InstrumentationMultiple/triangulationStandardized Nature of evidenceQuantitative and qualitativeQuantitative Reference points Over time, comparative, established goal Comparative or fixed standard Communication of results Multiple internal channels Public communication, media Use of resultsMultiple feedback loopsReporting Two Paradigms of Assessment Ewell, Peter T. (2007). Assessment and Accountability in America Today: Background and Context. In Assessing and Accounting for Student Learning: Beyond the Spellings Commission. Victor M. H. Borden and Gary R. Pike, Eds. Jossey-Bass: San Francisco.

Possible Sources or Models for Learning Outcomes Statements AAC&U Liberal Education for America’s Promise (LEAP) Statements and Rubrics Lumina Foundation Degree Qualifications Profile (DQP) and Qualifications Frameworks from Other Nations UK Quality Assurance Agency Subject Area Benchmarks Examples on NILOA Website Course Syllabi at Your Own Institution

AACU LEAP Statements and Rubrics Generic and Successively Refined Statements of What Students Should Know and Be Able to Do Address Broad and Integrated Content Knowledge, Intellectual Skills, Personal and Social Responsibility, and Applied/Integrative Learning Developed Through Employer Feedback as Well as the Higher Education Community Valid Assessment of Learning in Undergraduate Education (VALUE) Rubrics Being Tested Nationwide

Lumina Degree Profile  Three Degree Levels: Associate, Bachelor’s, and Master’s  Five Learning Areas: Specialized Knowledge, Broad/Integrative Knowledge, Intellectual Skills, Applied Learning, and Civic Learning  Framed as Successively Inclusive Hierarchies of “Action Verbs” to Describe Outcomes at Each Degree Level  Intended as a “Beta” Version, for Testing, Experimentation, and Further Development Beginning this Year

Characteristics of Assessment Evidence Direct vs. Indirect Assessment Forced-Choice vs. “Authentic” Assessments Standardized vs. Non-Standardized Results Specially-Constructed vs. Naturally- Occurring Assessments

Types of Assessment Evidence Examinations Tasks and Demonstrations Student Work Behavioral Outcomes Self-Reports (Surveys/Interviews)

A “Revolution” in Teaching and Learning  Insights from Cognitive Science Reinforce the Findings of Decades of Action Research on Teaching and Learning  A “New Majority” of Alternative Instructional Approaches is Emerging  So What Is the Appropriate Role of Learning Assessment in This New Environment?

Learning for Deep Understanding  Learners “Make Meaning” by Constantly Forging New Connections  Learners Actively Manipulate Their Environments via “Presenting Problems”  Learners Learn through Collaboration and Service  Learning Takes Place in a Supportive Culture and Context

Central Features of a Curriculum to Foster “Deep Learning”  Learning Outcomes are Known by Both Faculty and Students  Curricular Designs are Deliberate and Intentional Based on Research and Known “Best Practices” “Mapped” to Address Learning Outcomes  Curricular Designs are Actually Implemented as Planned  Teaching and Learning Yields Intended Learning Outcomes Demonstrably and Consistently  Faculty Talk About the Evidence for Learning to Make Appropriate Changes in What they Collectively Do Curricula are Optimized for Learning When:

What Assessment Characteristics Respond to These Conditions? Assessments “Add Value” at the Same Time as They Examine “Value Added” “Seamless” Assessments Become an Integral Part of Curriculum and Pedagogy Mechanisms are in Place to “Roll Up” the Results of Key Assignments and Exercises to the Course, Discipline, and Program Levels Assessments Emphasize Connections and Longitudinal Development, not Just Attainment

…And What Topics Should Assessment Particularly Address? Alignment of Key Learning Outcomes Across Topics, Sequences, and Courses Match Between Curricular Design, Delivery, and Student Experience Match Between Teaching/Learning Practices and the Needs of Diverse “Student Bodies” Effectiveness of Particular Innovations and Intervention

Attributes of a Meaningful “Culture of Evidence” for Learning Outcomes Shared Recognition That Many (But Not All) Things Are Knowable A Comprehensive Framework for Thinking About Learning Outcomes An Accessible Store of Information About Student Learning and What Produces It An Attitude Toward Problem-Solving that Minimizes “Finger-Pointing” at Institutions and Individuals Clear Follow-Through On Decisions Made and the Evidence Used to Make Them Willingness to Stop Doing Things When They Don’t Work

Implementation Strategies for Building a Culture of Evidence Start Small with Some “Easy Wins” Emphasize Positive and Collective Incentives Adjust the Approach to Fit Important Differences in Context Share Approaches Across Institutions and Faculties Involve “Middle Managers” and Hold Them Accountable

Strategies for Addressing Key Difficulties In Outcomes-Based Approaches Keep the Outcomes Framework Simple Think Constantly and Explicitly About Performance and What It Looks Like Use Multiple Forms of Assessment Articulate Assessment Into the Curriculum and the Everyday “Rhythms” of Teaching and Learning

Some “Points of Attack” in Building Cultures of Evidence Expectations Exercises What Do You Expect to Find? At What Point Do We Have a Problem? Use Results to Inform and Initiate Discussions, Rather than “Give Answers” Focus on Why Results Occurred Create “Data Dialogues” Initiate Visible Opportunities to Consider Results and What They Mean

Some “Points of Attack” in Building Cultures of Evidence (Continued) Present Results as Discrepancies Between Perceptions and Reality Among Different Constituencies or Groups Disaggregate Results to Show Variations Across Different Student Populations Across Different Dimensions of Ability Package Results Around Problems or Embed Them in Regular Decision Processes

Some Final Reminders: Building an Effective Culture of Evidence Means….  Not “Measuring Everything that Moves”  Not Just “Checking up After the Fact”  Not Searching for “Final” Answers  Not Always Being as “Precise” as Possible  Not Ever Expecting to be Done