Director of Policy Analysis and Research A State-Level Student Learning Outcomes Assessment Process That Really Looks at Outcomes Michael J. Keller Director of Policy Analysis and Research Maryland Higher Education Commission Presentation at SHEEO/NCES Network Conference Washington, DC-March 31, 2004

+ Within the next few years, greater attention is likely to be given to the results of assessment activities. Key stakeholders are inquiring about the quality of learning that is taking place in college.

Examples Accreditation organizations are asking campuses to provide information about the outcomes of assessment efforts. The Middle States Commission on Higher Education has revised its accreditation standards as they relate to the establishment of learning goals and the assessment of student achievement. Student outcomes, including the measurement of student learning, are being discussed as part of the Higher Education Act reauthorization in Congress. The National Forum on College-Level Learning, sponsored by the Pew Charitable Trusts, has begun to explore ways to make state-by-state comparisons of the knowledge gained in higher education. At the 2002 Governor’s Conference on Higher Education in Maryland, there was a consensus that assessment of student learning is an activity that deserves attention by campus leaders.

Background on State-Level Assessment In Maryland As part of the State’s performance accountability process prior to 1996, Maryland’s public colleges and universities had to develop a plan for the assessment of undergraduate student learning outcomes and submit annual progress reports to the Higher Education Commission. When the Commission adopted a system of benchmarked indicators for accountability in 1996, the campuses assumed responsibility for monitoring student learning outcomes. The campuses provide the Commission with a report every three years about their progress in improving student learning, instructional effectiveness, and curricula.

The 1998 and 2001 Assessment Reports Provided a good overview of the processes in which the public campuses have engaged in improving student learning. Shared only a limited amount of information about the impact that these efforts were having on undergraduates.

Maryland Is Not Unusual Measuring Up 2000 and 2002 gave every state a grade of “Incomplete” in the area of student learning. This grade highlights a gap in the nation’s ability to measure consistently and meaningfully what students learn in college. The assessment of student learning at the collegiate level, in Maryland and nationwide, has lagged behind efforts in elementary and secondary schools.

A Change in Direction in Maryland When the Commission received the 2001 assessment report, it asked the Secretary of Higher Education to convene a workgroup consisting of representatives of the State’s public two-and four-year campuses for the purpose of identifying ways of measuring the progress made in the actual educational outcomes of students. Composition of the workgroup Faculty and staff from the public colleges and universities with considerable experience in the area of undergraduate learning outcomes assessment. A representative from Middle States Commission staff

Consensus of the Workgroup Agreement emerged to tie the 2004 reports closely to the information which each campus prepares for its Middle State accreditation review. The reports will focus on the five competencies related to general education and essential skills that are used in Middle States’ accreditation process: Written and oral communications Scientific and quantitative reasoning Critical analyses and reasoning Technological competency Information literacy Campuses will be asked to address five questions in their reports for each of these competencies.

What is the definition for this competency at your institution?

What direct or indirect measures, methods, instruments and/or analyses are used to do assessment in this competency?

At what level(s) does assessment for this competency occur-courses, programs, and/or institutions?

Are results available for one or more of the assessment activities related to this competency.

Have the results of each assessment activity related to the competency been used to enhance teaching and learning as well as academic and strategic planning?

If a Campus Has No Current Activities in One or More of the Competencies They must acknowledge it in their reports. They must discuss whether plans have been or will be developed to do assessment in these areas in the future.

Flexibility in Assessment Approaches It is recognized that the selection of assessment activities and tools for general education competencies and the schedule for implementation will vary across campuses. Numerous approaches are legitimate. Campuses are provided with examples of widely-recognized and accepted measures for assessment in general education but not restricted to these techniques.

Direct Measures- Those that provide clear and compelling evidence of what students are learning. Course-embedded assessments, including written work and presentations scored using a rubric. Scores on locally designed tests and competency exams accompanied by test “blueprints” describing what is being assessed. Score gains between entry and exit on tests, competency exams and writing samples. Ratings of student skills in the context of class activities, projects and discussions. Portfolios of student work. Scores on nationally-normed instruments

Indirect Measures-Those that provide signs that students are probably learning, but it is less clear exactly what they are learning. Grades in assignments in general education courses not accompanied by a rubric. Student grades or passing rates in general education courses. Student evaluations and ratings of the knowledge and skills they have gained in general education courses. Student or graduate satisfaction with their learning in general education competencies. Results of nationally-normed surveys.

Flexibility in Reporting It is recognized that campuses are on different timetables with respect to Middle States’ reviews. The reporting of assessment results to the Higher Education Commission may not coincide with campus schedules for Middle States accreditation. This will impact the ability of institutions to provide information about the results of assessment activities.

State-Level Use of the Assessment Reports The Commission will publish a consolidated report about the student learning outcomes assessment activities and results at each campus. A summary of these activities and results, unedited by the Commission. Commission analysis about the consequences of these activities for student educational outcomes, as measured by the information provided by the campuses. The report will be disseminated and will appear on a permanent basis on the Commission’s web site.