What could we learn from learning outcomes assessment programs in the U.S public research universities? Samuel S. Peng Center for Educational Research and Evaluation National Taiwan Normal University

Main purpose Looking for effective models for the assessment of college student learning outcomes in Taiwan because Taiwan will include student learning outcomes as a major dimension of the next wave of institutional and academic program evaluation. Models that will provide comprehensive information about students’ knowledge, skills and attitudes as results of college education, and at the same time provide an informed basis for the improvement of teaching.

Procedures Selected a dozen public research universities in the U.S. and reviewed their assessment programs with a focus on – what has been assessed and – how the assessment has been done.

Major findings

1. Culture of Assessment Decisions on quality assurance and program improvement are based on hard evidence. There is a culture of assessment on student learning. All universities gather a lot of institutional data as well as faculty and student data for decision-making purpose. Data on student learning outcomes have become important recently, because they not only reveal the quality of students as a result of receiving college education, but also provide valuable data for administrators and faculty members to identify areas in the instructional program for improvement.

As stated by the University of Maryland at College Park, “Assessment of student learning is an important means for improving teaching and learning in higher education. Student learning outcomes assessment has become a focus of national efforts to improve higher education (such as the Spellings Commission) as well as a key concern for higher education accrediting organizations (such as the Middle States Commission on Higher Education). Within institutions of higher education there is a movement toward a “culture of assessment” that focuses attention on student learning and the improvement of teaching.”

2. Role of Accrediting Organizations Accrediting organizations in recent years have played a critical role in fostering or enhancing assessment programs by requiring institutions to report the assessment programs and results of student learning outcomes. Topics required by the report generally include:

In the management area, the issues include: --Does it have an administrative office for assessing student learning outcomes? -- Who is in charge? Does it have a committee to plan and set guidelines for assessment? -- Does each academic unit have assessment plan and annual report? -- Are there any regulations or law provisions for the evaluation?

In the area of implementation and operation, the issues include: – Does each academic unit have instructional objectives? – What learning outcomes are being evaluated and how are they being evaluated? – Have instructional goals been achieved? – How are evaluation results being used?

In the area of dissemination and services, the issues include: – What services are provided to faculty members to improve their evaluation skills? – How are evaluation results being disseminated? Those topical areas provide an excellent framework for evaluation. To comply with the requirements, colleges and universities usually design their evaluation programs to provide data accordingly to comply with the requirements as summarized below.

3. Assessment Programs (1 of 6) 1) Administration of assessment All universities have an administrative unit or a committee to direct and implement assessment. At the University of Maryland, for example, the Provost's Commission on Learning Outcomes Assessment provides the leadership and organizational procedures for engagement in such assessment. … The Commission's charge is to work with all academic units in developing student learning outcomes and ongoing assessments. The CommissionThe Commission is made up of four interacting groups of faculty and administrators: 1) A Steering Committee of Deans; 2) College Coordinators, named by each College to serve as a liaison for their respective colleges; 3) A Planning Team consisting of colleagues from Undergraduate Studies, Institutional Research, Planning and Assessment, and the Graduate School; and 4) Faculty Working Groups, named by the Steering Committee, made up of teaching and research faculty for specific assessment needs.” Steering CommitteeCollege CoordinatorsPlanning TeamFaculty Working Groups

3. Assessment Programs (2 of 6) 2) General assessment practices (1 of 6) (1) Assessment starts at the academic program level. Academic units are required to develop an assessment plan. Take the University of North Carolina at Chapel Hill as an example, “All academic and administrative units on campus are expected to maintain an assessment plan which articulates its intended student learning outcomes or administrative outcomes as appropriate. The plan must contain the mission of the program or unit, intended outcomes, methods to be used to measure the extent to which the outcomes are met, and an appropriate schedule for conducting the assessments. Each academic year, the unit is responsible for submitting to its Dean or Vice Chancellor’s office a report describing the assessments conducted, a summary of the results, and how the results have been used to improve student learning or administrative expectations.”

3. Assessment Programs (2 of 4) 2) General assessment practices (2 of 6) (2) The assessment program focuses on institutional and academic unit efficiency, not individual students’ performance. While data are gathered from students, the focus, however, is not how each individual student performs, but on how each academic program or administration unit performs.

3. Assessment Programs (2 of 4) 2) General assessment practices (3 of 6) (3) Professional knowledge and core ability assessments are most commonly done at the academic unit level The assessment is done through the use of direct assessments such as the capstone course performance and core abilities testing. Graduating seniors will be required to take the capstone course in which students are systematically assessed through examinations and/or a theme-based research report to determine what they have learned and how they could use their knowledge in their specific academic field. Some aspects of core abilities such as writing, critical thinking and problem solving abilities are also imbedded in those examinations and/or a theme- based research report.

3. Assessment Programs (2 of 4) 2) General assessment practices (4 of 6) (4) Core abilities such as writing and critical thinking could be assessed independently through the use of standardized tests in some universities. Frequently mentioned tests are: Measures of Academic Proficiency and Progress by Educational Testing Service; College Basic Academic Subjects Examination by the University of Missouri-Columbia Assessment Resource Center; College Outcome Measures Program by American College Testing; and Collegiate Learning Assessment by the Council for Aid to Education.

3. Assessment Programs (2 of 4) 2) General assessment practices (5 of 6) (5) Institutions also conduct indirect assessments of learning outcomes, e.g., National Survey of Student Engagement to determine students’ learning opportunities and learning efforts, and graduates follow-up survey to assess performance of their graduates and impacts of education on their career development. These surveys help to identify areas for improvement.

3. Assessment Programs (2 of 4) 2) General assessment practices (6 of 6) (6) Progress in performance is measured by performance trends and peer comparisons (e.g., UNC).

3. Assessment Programs (3 of 4) 3) Application of assessment results There are many uses of the assessment results for improving instruction and increasing learning. For example, improvement actions may result in: – Revised/redesigned curriculum – Added new course(s) – Revised course(s) – Changes of course requirements – Improved outcomes assessments – Improved monitoring of student progress – Modified qualifying exam procedures – Developed new concentration or minor

3. Assessment Programs (4 of 4) 4) Resources and services provided to faculty members (1of 3) (1) Website to store and disseminate information Each institution generally has its special Website that contains a large number of online resources related to assessment methodology, examples of assessment systems and plans from other institutions, and sample templates for reporting and analyzing results.

3. Assessment Programs (4 of 4) 4) Resources and services provided to faculty members (2 of 3) (2) Workshops Each institution also provides various seminars or workshops for faculty members to gain further understanding of the overall assessment program, the resources available to them, as well as assessment methods that can be applied to their teaching.

3. Assessment Programs (4 of 4) 4) Resources and services provided to faculty members (3 of 3) (3) Small research grants Some universities such as Western Michigan University provide small grants to encourage faculty members to develop effective strategies for assessing learning outcomes and using assessment results for improving instruction.

Conclusion The above findings provide an excellent basis for designing an appropriate assessment of student learning in Taiwan. We could at least follow a few general principles, as advocated by the University of Virginia, that – Assessments should be useful, feasible, and minimize the burden on faculty and students. – Each academic program should define its own most important outcomes. – Assessment methods should be kept as simple as possible and connected to what faculty and students are already doing. – The unit of analysis for learning outcomes assessments should be at the academic program level. – Faculty involvement is essential. – Results should be actively used to make program improvements.

Conclusion We believe that a clear vision and effective system of student learning outcomes assessment will help higher education in Taiwan move a giant step forward in its pursuance of excellence and prominence.