U.S. Survey Reveals Small Impact of Outcomes Assessment on Student Learning Presented at the 31 st EAIR Forum Vilnius, Lithuania 24 August, 2009 By Trudy W. Banta Professor of Higher Education and Senior Advisor to the Chancellor for Academic Planning and Evaluation Indiana University-Purdue University Indianapolis 355 N. Lansing St., AO 140 Indianapolis, Indiana iupui.edu

Designing Effective Assessment: Principles & Profiles of Good Practice Trudy W. Banta Elizabeth A. Jones Karen E. Black Jossey-Bass (Wiley) 2009 © TWBANTA-IUPUI

Profiles  Invited over 1000  Received 146  Selected 49 for use in full  Categorized all 146 and published Web sites © TWBANTA-IUPUI

Outline for Profiles  Background and Purpose  Methods over ? Years  Resources Required  Findings  Use of Findings  Impact of Using Findings  Success Factors  Web sites © TWBANTA-IUPUI

Plan Implement Evaluate Improve Culture of Evidence

~ Organization ~ of Principles & Profiles  Planning  Implementing  Improving & Sustaining - Building a Scholarship of Assessment Banta & Associates Jossey-Bass 2002 © TWBANTA-IUPUI

Types of Institutions Represented  Doctoral and Research Universities Private 7 Public16………..47%  Master’s13………..26%  Baccalaureate 7…..……14%  Associate 5………..10%  ACE 1……..…..2% © TWBANTA-IUPUI

Planning Profiles  Brigham Young University Campus Wiki for degree learning outcomes  USMA at West Point Interdisciplinary teams assess 10 mission-related goals for learners  Kennesaw State University 2008 CHEA Award for linking assessment with planning, program review, faculty development © TWBANTA-IUPUI

West Point 6 Developmental Domains 1. Intellectual 2. Physical 3. Military 4. Social 5. Moral-ethical 6. Human spirit © TWBANTA-IUPUI

West Point Intellectual Domain 10 Goals (write, speak, think; engineering, math, info tech) A. Stated learner outcomes 1. Standards a. Rubrics developed by faculty © TWBANTA-IUPUI

West Point Interdisciplinary Goal Teams use Curriculum-embedded direct measures of learning Student surveys (fr., sr.) Graduate survey (3 years after) Employer surveys Employer focus groups © TWBANTA-IUPUI

West Point Use of Assessment Findings  Review of core curriculum  Changes in warranted areas: History English Engineering Information Technology © TWBANTA-IUPUI

As a catalyst for outcomes assessment accreditation is powerful  regional  disciplinary © TWBANTA-IUPUI

Engaging faculty in taking a second look at student work for outcomes assessment purposes may require extra pay © TWBANTA-IUPUI

Standardized tests of generic skills (e.g., writing, critical thinking)  used by just 8%  always supplemented © TWBANTA-IUPUI

At the University of Tennessee CAAP Academic Profile (now MAPP) COMP (like CLA and withdrawn by 1990) College BASE

© TWBANTA-IUPUI In TN We Learned 1) No test measured 30% of gen ed skills 2) Tests of generic skills measure primarily prior learning 3) Reliability of value added =.1 4) Test scores give few clues to guide improvement actions

© TWBANTA-IUPUI An Inconvenient Truth.9 = the correlation between SAT and CLA scores of institutions thus 81% of the variance in institutions’ scores is due to prior learning

More use of RUBRICS  locally developed  VALUE from AAC&U © TWBANTA-IUPUI

Better Ways to Demonstrate Accountability Measures of Learning  Standardized tests in major fields  Internship performance  Senior projects  Electronic portfolios  External examiners © TWBANTA-IUPUI

Student Electronic Portfolio Students take responsibility for demonstrating core skills Unique individual skills and achievements can be emphasized Multi-media opportunities extend possibilities Metacognitive thinking is enhanced through reflection on contents - Sharon J. Hamilton IUPUI

Accountability Report  85% achieve Outstanding ratings in writing as defined...  78% are Outstanding in applying knowledge and skills in internships  75% are Outstanding in delivering an oral presentation © TWBANTA-IUPUI

For External Credibility  Collaborate on rubrics  Use employers as examiners  Conduct process audits © TWBANTA-IUPUI

Build Assessment into Valued Processes 1. Assessment of learning 2. Curriculum review and revision 3. Survey research 4. Program review 5. Scholarship of Teaching & Learning 6. Evaluation of initiatives 7. Faculty development 8. Promotion & tenure 9. Rewards and recognition

Impact of Using Assessment Findings NOT measured in student learning gains ?WHY? © TWBANTA-IUPUI

Group Assessment Has Failed to Demonstrate Institutional Accountability 42% less than 2 years old Focus on improvement at unit level Rare aggregation of data centrally Too few faculty involved HE scholars focused on K-12 assessment