George D. Kuh Rankings and the Visibility of Quality Outcomes in the EHEA Dublin, Ireland January 31, 2013 Measuring What Matters: Forging the Right Tools to Assess Student Learning Outcomes

Overview Overview  The U.S. context  A word about NILOA  Assessment: Purposes and approaches  The kind of learning we need today  The measurement tools we need  Concluding thoughts

The U.S. Context Unprecedented numbers of increasingly diverse students matriculating Unprecedented numbers of increasingly diverse students matriculating Many underprepared students Many underprepared students Rising college costs Rising college costs Continuing shift of cost from government to students Continuing shift of cost from government to students Increasing numbers of part-time instructors Increasing numbers of part-time instructors Worries about university quality, global competitiveness Worries about university quality, global competitiveness

NILOA NILOA’s mission is to discover and disseminate effective use of assessment data to strengthen undergraduate education and support institutions in their assessment efforts. D EGREE Q UALIFICATIONS P ROFILE S URVEYS ● W EB S CANS ● C ASE S TUDIES ● F OCUS G ROUPS ● O CCASIONAL P APERS ● W EBSITE ● R ESOURCES ● N EWSLETTER ● L ISTSERV ● P RESENTATIONS ● T RANSPARENCY F RAMEWORK ● F EATURED W EBSITES ● A CCREDITATION R ESOURCES ● A SSESSMENT E VENT C ALENDAR ● A SSESSMENT N EWS ● M EASURING Q UALITY I NVENTORY ● P OLICY A NALYSIS ● E NVIRONMENTAL S CAN ● D EGREE Q UALIFICATIONS P ROFILE

Assessment 2013 Assessment 2013 Greater emphasis on student learning outcomes and evidence that student performance measures up

Assessment 2013 Assessment 2013 Greater emphasis on student learning outcomes and evidence that student performance measures up Demands for comparative measures Increased calls for transparency --- public disclosure of student and institutional performance Assessment “technology” has improved markedly, but still is insufficient to document learning outcomes most institutions claim

Measuring Quality in Higher Education ( Vic Borden & Brandi Kernel, 2010) Web-based inventory hosted by AIR of assessment resources. Key words can be used to search the four categories:  instruments (examinations, surveys, questionnaires, etc.);  software tools and platforms;  benchmarking systems and data resources;  projects, initiatives and services.

Assessment Purposes  Improvement  Accountability

Continuous Improvement Continuous Improvement Accountability Accountability Strategic dimensions PurposeFormative (improvement)Summative (judgment) OrientationInternalExternal MotivationEngagementCompliance Implementation InstrumentationMultiple/triangulationStandardized Nature of evidenceQuantitative and qualitativeQuantitative Reference points Over time, comparative, established goal Comparative or fixed standard Communication of results Multiple internal channels Public communication, media Use of resultsMultiple feedback loopsReporting Two Paradigms of Assessment Ewell, Peter T. (2007). Assessment and Accountability in America Today: Background and Context. In Assessing and Accounting for Student Learning: Beyond the Spellings Commission. Victor M. H. Borden and Gary R. Pike, Eds. Jossey-Bass: San Francisco.

Assessment Tools  Direct measures  Direct (outcomes) measures -- Evidence of what students have learned or can do  Indirect measures  Indirect (process) measures -- Evidence of effective educational activity by students and institutions -- Evidence of effective educational activity by students and institutions

Direct Measures  ETS Proficiency Profile & Major Field Tests  ACT Collegiate Assessment of Academic Proficiency (CAAP)  Collegiate Learning Assessment (CLA) – the AHELO measure of general skills  Competency tests (e.g., nursing, education)  Portfolios (authentic student work such as writing samples)  Performances, demonstrations

Indirect Measures  National Surveys of Student Engagement (NSSE/CCSSE/AUSSE/SASSE)  Beginning College Survey of Student Engagement (BCSSE)  Faculty Survey of Student Engagement (FSSE)  Cooperative Institutional Research Program (CIRP)  Your First College Year (YFCY)  College Student Experiences Questionnaire (CSEQ)  Noel Levitz Student Satisfaction Inventory

Institution-level assessments of learning outcomes for all institutions

Program-level assessments of learning outcomes for all institutions

Student Engagement What Really Matters in University: Student Engagement Because individual effort and involvement are the critical determinants of college impact, institutions should focus on the ways they can shape their academic, interpersonal, and extracurricular offerings to encourage student engagement. Pascarella & Terenzini, 2005, p. 602

The quality of student experiences varies more within than between institutions.

% of Variance Between Institutions

Simple Simple Generally Applicable Generally Applicable Accurate Accurate Commensurate Complexity Commensurate Complexity (Thorngate, 1976; Weick, 1985)

US Economy Defined by Greater Workplace Challenges and Dynamism  More than 1/3 of the entire US labor force changes jobs ANNUALLY.  Today's students will have jobs by age 38.  Half of workers have been with their company less than 5 years.  Every year, more than 30 million Americans are working in jobs that did not exist in the previous year. DOL-BLS

The World Wants More From Us and Our Graduates  …more college-educated workers.  …more educated workers with higher levels of learning and knowledge.

Raising The Bar – October/November 2009 – Hart Research for 23 Employer expectations of employees have increased % who agree with each statement Our company is asking employees to take on more responsibilities and to use a broader set of skills than in the past Employees are expected to work harder to coordinate with other departments than in the past The challenges employees face within our company are more complex today than they were in the past To succeed in our company, employees need higher levels of learning and knowledge today than they did in the past

Key Capabilities Open the Door for Career Success and Earnings “Irrespective of college major or institutional selectivity, what matters to career success is students’ development of a broad set of cross-cutting capacities…” Anthony Carnevale, Georgetown University Center on Education and the Workforce

Narrow Learning is Not Enough: The Essential Learning Outcomes  Knowledge of Human Cultures and the Physical & Natural World  Intellectual and Practical Skills  Personal and Social Responsibility  “Deep” Integrative Learning

Deep, Integrative Learning Deep, Integrative Learning  Attend to the underlying meaning of information as well as content  Integrate and synthesize different ideas, sources of information  Discern patterns in evidence or phenomena  Apply knowledge in different situations  View issues from multiple perspectives

Do we measure what we value? Do we value what we measure? Do we measure what we value? or Do we value what we measure? Wise decisions are needed about what and how to measure the proficiencies demanded by the 21 st century

We need – and are poised for – a “sea change” in what counts as meaningful evidence of student progress and accomplishment.

Employers On Accountability Challenge – December 2007 – Hart Research for Supervised internship/community-based project 83% 79% 60% 56% 32% Senior project (e.g., thesis, project) Essay tests Electronic portfolio & faculty assessments Multiple-choice tests Evidence of College Graduates Skills and Knowledge Very effectiveFairly effective

To Get the Right Kind of Evidence…  We need assessment approaches sensitive to a wide variety of knowledge, abilities, proficiencies, and dispositions

Employers On Accountability Challenge – December 2007 – Hart Research for

Promising Experiments

Shift the national conversation from what is taught to what is learned by providing HEIs with a template of widely agreed-upon competencies required for the award of degrees.

3 levels: Associate, Bachelor, Masters 5 areas: Broad, Integrative Knowledge Specialized Knowledge Intellectual Skills Applied Learning Civic Learning

Occasional Paper #16 The Degree Qualifications Profile: Implications for Assessment Peter T. Ewell & Carol Geary Schneider This paper offers guidance for how to gather evidence about the extent to which the competencies described in the DQP are mastered at the levels claimed. The challenges associated with assessing DQP proficiencies are outlined.

Promising Experiments A dozen two- and four-year HEIs are using the VALUE Rubrics in A dozen two- and four-year HEIs are using the VALUE Rubrics in a “proof of concept at scale” with an eye toward building a national vehicle for using common rubrics

Valid Assessment of Learning in Undergraduate Education (VALUE) Rubrics  Inquiry and analysis  Critical thinking  Creative thinking  Written communication  Oral communication  Reading  Quantitative literacy  Information literacy  Teamwork  Problem solving  Civic knowledge and engagement  Intercultural knowledge and competence  Ethical reasoning and action  Foundations and skills for lifelong learning  Integrative learning

AAC&U VALUE Project – 15 Rubrics

Promising Experiments Massachusetts effort led by the Commissioner for Higher Education is enlisting additional states to use assessment of authentic student learning work to compare performance and monitor progress

Moving Quality Assurance Forward  Shift the motivation for assessment work from compliance mentality to institutional responsibility  Experiment with ways to “roll up” program level results into meaningful institution-level profiles of student accomplishment  Reconcile or ameliorate the tensions between the accountability and improvement purposes and uses of assessment

Moving Quality Assurance Forward  Show how assessment results are being used to modify curriculum and teaching and learning approaches and enhance student learning

Transparency

Voluntary System of Accountability (APLU/AASCU)

VSA Student Learning Outcomes Pilot  Four-year experiment  Value-added approach  Institutional level evidence  Administer and publicly post results: –Collegiate Assessment of Academic Proficiency –Collegiate Learning Assessment –ETS Proficiency Profile

Templates Voluntary System of Accountability (APLU/AASCU) U-CAN /Building Blocks for 2020 NAICU) U-CAN /Building Blocks for 2020 (NAICU) College Navigator College Navigator (NCES) Transparency by Design/College Choices for Adults Transparency by Design/College Choices for Adults (WCET) Voluntary Framework of Accountability Voluntary Framework of Accountability (AACC) ransparency Framework Transparency Framework (NILOA)

Transparency Reports

TransparencyFramework Providing Evidence of Student Learning: A Framework for Transparency Based on an examination of about 1000 institutional websites, the Transparency Framework provides guidance to institutions for effectively presenting learning outcomes assessment information on their websites.

The things we have to learn before we do them, we learn by doing them. Aristotle, Nicomachean Ethics

What Matters in University: A Data-Based Narrative… We need a campaign led by university leaders, staff, students, and quality assurance agencies that features students’ best work, buttressed by evidence that all students meet established proficiencies in response to staff-designed assignments that require students to show they can use and apply what they are learning to concrete situations on and off the campus.

Questions&Discussion