The Assessment of Higher Education Learning Outcomes (AHELO) OECD’s Study of the Scientific and Practical Feasibility of Assessing Baccalaureate-Level Student Learning Outcomes Across Nations and Institutions Lock Haven University, Pennsylvania February 2, 2012 Charlie Lenth, SHEEO National Project Manager

Introduction and Orientation Lots of Topics—Limited Time 1.AHELO background, design and development 2.U.S., SHEEO, state and institutional involvement 3.Current status, immediate tasks and timelines 4.Field manuals—institutional coordination, sampling, test administration 5.Discussion of next steps

AHELO Background, Context and Development OECD background OECD’s education studies, surveys and analyses –Programme for International Student Assessment (PISA) –Institutional Management in Higher Education (IMHE) Changing Education Needs, Methods and Objectives 2006 Council of Education Ministers’ call for postsecondary assessment, commissioned papers and “expert” discussion to develop project design

Basic Project Objectives and Design Feasibility study—to determine scientific and practical feasibility of assessing college learning across nations Assess learning across and within the disciplines or program areas at degree completion Minimum five diverse nations with non-scientific sample of 10 institutions each Small, random sample of students with institutions as primary unit of analysis Student, faculty, institutional surveys and data to allow meaningful interpretation and analysis

U.S. Involvement in AHELO Research, design and development (CAE, Peter Ewell, ETS, CHEA and ACE, others) No federal, state or institutional involvement until 2009 Invitation from OECD in 2010, and from SHEEO to state members and institutions Meeting participation: Washington, New York with CAE for Generic Skills, OECD for GNE and NPM

U.S. Financial Support for AHELO Participation Lumina Foundation is largest foundation donor Hewlett Foundation for initial U.S. participation; other smaller grants later U.S. Department of Education –Payment of national participation fees –Encouragement for FIPSE competitive program application in 2010 –Discretionary grant provided September 2011 –OECD Education Policy Committee

Country Participation in AHELO Generic Skills Economics Engineering Columbia Belgium Australia Egypt Egypt Canada Finland Italy Columbia Korea Mexico Egypt Kuwait Netherlands Japan Mexico Russia Mexico Norway Slovakia Russia Slovakia United States Total countries: 16 plus observers and others asking to join

Overview--U.S. Participants’ Roles National level –US Dept of Ed provides financial support and member of OECD Education Governing Board –SHEEO as National Project Manager (NPM) and national representative in Group of National Experts (GNE) –Foundations, stakeholders (CHEA), other interested parties State level –SHEEO agency provides project leadership, coordination, budget oversight in Connecticut, Missouri and Pennsylvania Institutional level –AHELO Institutional Coordinator and implementation “team” –Sample 200 students and administer test and surveys –Sample 40 faculty to complete 20-minute survey –Complete institutional survey by IR and other relevant offices –Access to data and determine its institutional uses

U.S. Higher Education Institutions Participating in AHELO Feasibility Study Connecticut Board of Regents for Higher Education –Southern Connecticut State University (public regional university; 9,000 undergraduates) Missouri Department of Higher Education –Central Methodist University (private university; 3,500 students) –Missouri State University (public institution; 16,000 undergraduates) –Truman State University (public institution; 6,000 undergraduates) –University of Central Missouri (public regional; 12,000 students) –Webster University (private, not-for-profit; 4,000 undergraduates ) Pennsylvania State System of Higher Education (PASSHE) –Cheyney University of Pennsylvania (public historically black university; 1,300 undergraduates) –Clarion University (public institution; 5,100 undergraduates) –Lock Haven University (public institution; 5,000 undergraduates) –Millersville University (public institution; 7,200 undergraduates)

Institutional Sampling Plan AHELO Sampling Manual provides international guidelines for both student and faculty samples Basic components: 1.Institutions provide census of “in scope” students and faculty to NPM (SPSS.sav or Excel.xls/.xlsx preferred) 2.NPM will draw equal probability samples using prescribed routines and submit for ACER review 3.Flexibility for stratified or “augmented” sample design if justified and documented by institution (e.g., strata for discipline/program areas, student characteristics, special accommodations, etc.) 4.Review and approval by ACER Sampling Team (led by Statistic Canada—Jean Dumais)

Sampling Steps and Timelines 1.Institutions use Sampling Manual to prepare student and faculty sampling plans/census files 2.Coding guidelines and CIP crosswalk available at crosswalk.xlsx crosswalk.xlsx 3.State Coordinators and NPM review and submit all U.S. plans to ACER/Statistic Canada 4.Review by and negotiations with ACER Sampling Team (allow at least one week) 5.Approved (or clarification) of institutional sampling frameworks by Statistics Canada (allow two prior to initial test administration)

Institutional Coordination Manual Manual outlines institutional guidelines applicable to diverse international contexts Institutional Coordinators (ICs) undertake training, sampling, scheduling, system testing, beginning 1-2 months before testing dates ICs, State Coordinators and NPM set and approve test dates and other scheduling parameters ICs recruit and train Test Administrators (TAs) (Minimum one TA per room of 40 students) ICs monitor test sessions, response rates, address needs as they emerge, work with State Coordinators and NPM to meet project objectives

Test Administration Procedures International TA Manual is non-secure document for distribution as necessary Outlines testing conditions and restrictions (e.g., no electronic devices other than computers provided, access to other websites blocked) Access to testing platform, complete instructions, student identifiers and testing passwords, etc. Testing session parameters: –Welcome and necessary instructions – 10 minutes –Performance tasks – 90 minutes –Multiple choice questions – 30 minutes –Student contextual survey – 10 minutes

Questions and Challenges NPM and state coordinators will seek and share consistent answers to technical questions (e.g., file layout, IDs, fields, international codes) Institutions can work together to plan student incentives and follow-up procedures (e.g., invitation, incentives, rescheduling missed testing dates) Institutional and state coordinators set schedules and provide project timelines consistent with international guidelines and deadlines

Subsequent Steps U.S. lead scorer will be trained on international guidelines, then recruit and train 5-7 U.S scorers National scoring done in May on secure platform using prescribed procedures and common rubrics Scored assessment results analyzed and reported by ACER collaborators and researchers Review by Technical Advisory Group and OECD Preparation, review and release of Feasibility Study findings and recommendations Final project conference December 2012

Please Take Note OECD and ACER are acutely aware that the entire feasibility study rests on voluntary participation by nations, institutions, students and faculty Motivation may be a challenge for students, faculty and even institutions, but potential and international importance are high Resulting data, analyses and uses are expected to include institutional access and some “comparative benchmarks,” depending upon reliability, validity, and other criteria

Context and Consequences AHELO is part of a larger postsecondary education agenda with implications for students, institutions, nations and the global environment. Much has already been learned about both scientific and practical feasibility for assessment within an international setting. OECD and other major organizations will shape future decisions and financial commitments on the results of this study.

Thank you. Charles S. Lenth State Higher Education Executive Officers