بسم الله الرحمن الرحيم

Dr. Mohammad S. Al-Homoud Outcome-Based Program Assessment An Overview Dr. Mohammad S. Al-Homoud Director Program Assessment Center Deanship of Academic Development King Fahd University of Petroleum & Minerals Dhahran 31261 assess@kfupm.edu.sa

What is Assessment? Assessment is a systematic ongoing, iterative process of monitoring (recording and analyzing) student learning outcomes and processes in order to determine what we are doing well and what we must improve. “Processes that identify, collect, use and prepare data for evaluation of the achievements of program outcomes, educational objectives, or for program improvement” (ABET)

What is the aim of program assessment? The aim of program assessment is to understand how educational programs are working and to determine whether they are contributing to student growth and development. It focuses on programs rather than on individual students. It provides information on whether the curriculum as a whole provides students with the knowledge, skills and values that graduates should possess in accordance with its mission and set goals and learning objectives.

What is Evaluation? “Processes for the interpretation of the data and evidence obtained from assessment practices that a) determine the extent to which program outcomes or educational objectives are being achieved; or b) result in decisions and actions to improve the program.” (ABET) Evaluation is an event with common checkpoints by an evaluator(s) that involves determining the quality of past efforts and provides a reward or punishment (accountability). It constitutes any effort to use assessment evidence or findings to apply judgments that result in a grade, score or value.

Assessment or Evaluation ? Assessment is meant for quality assurance and involves: Analyzing and interpreting information Activities aimed at improvement Strong connection between assessment results and instructional plans.

Assessment or Evaluation ? Evaluation is meant for accountability and involves: Applying judgments to assessment efforts Activities aimed at demonstration of accountability Little connection with instructional plans Measurement involves: Gathering and quantifying information

What is Accreditation? A procedure by which an authoritative body gives formal recognition that a body (organization) or a person is competent to carry out specific tasks. (ISO/IEC guide 2:96)

New Philosophy Traditional emphasis What we give students? Inputs Faculty teaching Curriculum Educational activities as an end More quantitative (how much) Prescriptive based New emphasis What they can/are able to do? Outcomes Student learning Education Educational activities as means to an end More qualitative (how well) Outcome-based

Traditional Educational Process Standards Input Process Output (Anticipated Outcomes) Traditional Educational Process (Prescriptive, teaching focused)

Outcome-Based Assessment Intended Outcomes Input Process Output (Actual outcomes) Measurement/ Comparison Feedback Outcome-Based Assessment (Outcome-based, learning focused, continuous feedback/improvement)

Continuous Quality Improvement Characteristics of Continuous Quality Improvement: Systematic Dynamic Continuous Motivating Collaborative Integrative not in isolation Focus on education not on curriculum Satisfy the needs of constituents

Assessment Key Questions Program assessment system should answer the following questions: Who are our constituents/suppliers? What are we trying to do (objectives/ outcomes)? Do our constituents/suppliers understand our objectives? How well we are achieving what we are doing? What is the process of doing it?

Assessment Key Questions What qualities and level of qualities that we want? How are we going to measure these qualities? How do we utilize the results to continuously improve? Are we achieving our objectives? Are we satisfying the needs of our constituents? Does it work?

Assessment Driving Forces Students Faculty Employers/ industry Push/need for program assessment Accrediting Agencies Administrators Government authorities Assessment Driving Forces

Assessment Benefits Making the institution more self-conscious about what its programs are accomplishing Providing information for recruitment Providing information for planning Providing information for accreditation Improvements in teaching and learning Improving institutional image Enhancing funding opportunities Attracting better students

Approaches to Assessment What to assess (educational outcomes)? Cognitive (knowledge-based) : what should students know? Affective (attitudes and values) : what should students value? Psychomotor (skills) : what should students do or be able to do?

Approaches to Assessment Whom to assess (objects)? Students Faculty Course Program Why assess (purpose)? Formative: on going (e.g. to ascertain progress or development) Summative: at the end (e.g. to ascertain mastery level of achievement)

Assessment Steps Recognition/commitment Define program constituents Define program /institution mission statements Define educational objectives that meet program / institution constituents Define program outcomes Select appropriate assessment tools Collect necessary data Emphasis on outcomes (graduates preparation for professional practice)

Assessment Steps Demonstrate how criteria and program educational objectives/outcomes are being met Seek external review Implement actions for improvement Refine and repeat the process

Assessment Steps Mission Goals Objectives Outcomes Assessment tools Assessment activities Actions for improvement

Goals and Objectives Goals are used to express intended results in general terms (broad learning concepts such as clear communication, problem solving, ethical awareness, etc.) Objectives are used to express intended results in precise terms (specific behaviors students should exhibit). Objectives tell us more specifically what needs to be assessed, and thus a more accurate guide to suitable assessment tools.

Goals and Objectives Both terms are used interchangeably Both terms describe the intended results of educational activities Both terms provide direction for assessment

Goals and Objectives Program Educational Objectives Statements that describe the expected accomplishments of graduates during their first few years of career after graduation as a result of their educational preparation. Types of program educational objectives: Objectives that all graduates are expected to accomplish; Objectives that some graduates are expected to accomplish.

Goals and Objective Audiences for objective statements: Constituents external to the institution including: Prospective students Parents Student sponsors, donors, and contributors Employers Transfer institutions, graduate programs Industry advisors Accrediting agencies Government organizations

Goals and Objectives Features of Good Statements of Program Educational Objectives Relevant to the mission of the institution; Each addresses one or more needs of one or more of the program constituents; Understandable by the program constituents; Clear, concise, and consistent; Realistic/achievable; Measurable; Forward looking (not short term); Manageable list (limited number of statements); Focus on student learning.

Program Outcomes Statements that describe what students are expected to know and be able to do by the time of graduation. These relate to the knowledge, skills and behaviors that students acquire in the program. Intended outcomes (from described statements of expected learning outcomes)

Program Outcomes Actual outcomes (form the results of assessment activities) Achievements of all program outcomes should indicate that the graduate is prepared to achieve the program educational objectives and should be verified before certification for graduation.

Program Outcomes: Features of Good Statements of Intended Outcomes: Relative to the missions of the program and institution; Describe the expected learning outcomes for students from individual courses/the program as a whole; Reflect what the students are expected to know and be able to do rather than what the course/program intends to cover; Each describes an area of knowledge and/or skill that a person is expected to possess;

Program Outcomes: Supportive of one or more educational objectives; Describe an observable behavior using action verbs (i.e. apply, identify, solve, construct, etc.); Avoid vague words that might be open to interpretation (i.e. understand, appreciate, etc.); Assessable (outcomes can be demonstrated in terms of performance).

Assessment Methods I. Direct Assessment Methods: Directly determine whether students have mastered the content of their academic programs. Require students to display their knowledge and skills as they respond to the instrument itself (i.e. objective tests, essays, presentation, and classroom assignments).

Assessment Methods Direct assessment methods include: Standardized exams Locally developed exams Oral exams Portfolios (work collected over time) Performance appraisal

Assessment Methods Direct assessment methods include (Cont.): Oral presentations Projects, demonstrations, case studies, simulations Capstone experience (embodied in capstone courses) Juried activities Evaluation of field work. Behavioral observations

Assessment Methods II. Indirect Assessment Methods: Ask students to reflect on their learning, what they have learned and experienced, rather than to demonstrate it (i.e. surveys and interviews). Details about instructional or curricular strengths that can not be provided by direct methods alone.

Assessment Methods Indirect assessment methods include: Written surveys and questionnaires Entering students Current students Graduating seniors Faculty Alumni Employers Parents Exit interviews Focus groups External examiner Archival records

Program Outcomes Influencing Factors Curriculum: Major Courses GE Pre-College Experience Surrounding Environment: Campus life Social life Out-of-Class Experiences: Co-op Internship Educational Outcomes Educational Objectives Classroom Experience: Pedagogy Faculty Facilities “Primary” Educational Outcomes Assessment Zone Educational Objectives Assessment Zone Pre-College Educational Experience College Educational Experience Post-College Experience

Assessment Key Success Factors Assessment Key Success Factors Include: Commitment Participation (involvement) of all constituents Listening to voices Actions for improvement Feedback Relating assessment to the world of work Continuity and flexibility Consistency of reporting format over time History of results Responsibility Resources Rewards

THANK YOU