1. بسم الله الرحمن الرحيم

2. Outcome-Based Program Assessment An Overview Dr. Mohammad S. Al-Homoud Director Program Assessment Center Deanship of Academic Development King Fahd University of Petroleum & Minerals Dhahran 31261 assess@kfupm.edu.sa

3. What is Assessment? Assessment is a systematic ongoing, iterative process of monitoring (recording and analyzing) student learning outcomes and processes in order to determine what we are doing well and what we must improve. “Processes that identify, collect, use and prepare data for evaluation of the achievements of program outcomes, educational objectives, or for program improvement” (ABET)

4. What is the aim of program assessment? The aim of program assessment is to understand how educational programs are working and to determine whether they are contributing to student growth and development. It focuses on programs rather than on individual students. It provides information on whether the curriculum as a whole provides students with the knowledge, skills and values that graduates should possess in accordance with its mission and set goals and learning objectives.

5. What is Evaluation? “Processes for the interpretation of the data and evidence obtained from assessment practices that a) determine the extent to which program outcomes or educational objectives are being achieved; or b) result in decisions and actions to improve the program.” (ABET) Evaluation is an event with common checkpoints by an evaluator(s) that involves determining the quality of past efforts and provides a reward or punishment (accountability). It constitutes any effort to use assessment evidence or findings to apply judgments that result in a grade, score or value.

6. Assessment or Evaluation ? Assessment is meant for quality assurance and involves: Analyzing and interpreting information Activities aimed at improvement Strong connection between assessment results and instructional plans.

7. Assessment or Evaluation ? Evaluation is meant for accountability and involves: Applying judgments to assessment efforts Activities aimed at demonstration of accountability Little connection with instructional plans Measurement involves: Gathering and quantifying information

8. What is Accreditation? A procedure by which an authoritative body gives formal recognition that a body (organization) or a person is competent to carry out specific tasks. (ISO/IEC guide 2:96)

9. Traditional emphasis What we give students? Inputs Faculty teaching Curriculum Educational activities as an end More quantitative (how much) Prescriptive based New emphasis What they can/are able to do? Outcomes Student learning Education Educational activities as means to an end More qualitative (how well) Outcome-based New Philosophy

10. ProcessOutput (Anticipated Outcomes) Input Standards Traditional Educational Process (Prescriptive, teaching focused) Traditional Educational Process

11. InputProcess Output ( Actual outcomes ) Measurement / Comparison Intended Outcomes Feedback Outcome-Based Assessment (Outcome-based, learning focused, continuous feedback/improvement) Outcome-Based Assessment

12. Characteristics of Continuous Quality Improvement: Systematic Dynamic Continuous Motivating Collaborative Integrative not in isolation Focus on education not on curriculum Satisfy the needs of constituents Continuous Quality Improvement

13. Assessment Key Questions Program assessment system should answer the following questions: Who are our constituents/suppliers? What are we trying to do (objectives/ outcomes)? Do our constituents/suppliers understand our objectives? How well we are achieving what we are doing? What is the process of doing it?

14. Assessment Key Questions What qualities and level of qualities that we want? How are we going to measure these qualities? How do we utilize the results to continuously improve? Are we achieving our objectives? Are we satisfying the needs of our constituents? Does it work?

15. Push/need for program assessment Employers / industry Accreditin g Agencies Governmen t authorities Administra tors Faculty Students Assessment Driving Forces

16. Assessment Benefits Making the institution more self-conscious about what its programs are accomplishing Providing information for recruitment Providing information for planning Providing information for accreditation Improvements in teaching and learning Improving institutional image Enhancing funding opportunities Attracting better students

17. Approaches to Assessment What to assess (educational outcomes)? Cognitive (knowledge-based) : what should students know? Affective (attitudes and values) : what should students value? Psychomotor (skills) : what should students do or be able to do?

18. Approaches to Assessment Whom to assess (objects)? Students Faculty Course Program Why assess (purpose)? Formative: on going (e.g. to ascertain progress or development) Summative: at the end (e.g. to ascertain mastery level of achievement)

19. Assessment Steps Recognition/commitment Define program constituents Define program /institution mission statements Define educational objectives that meet program / institution constituents Define program outcomes Select appropriate assessment tools Collect necessary data Emphasis on outcomes (graduates preparation for professional practice)

20. Assessment Steps Demonstrate how criteria and program educational objectives/outcomes are being met Seek external review Implement actions for improvement Refine and repeat the process

21. Goals Assessment Steps Mission Actions for improvement ObjectivesOutcomes Assessment tools Assessment activities

22. Goals and Objectives Goals are used to express intended results in general terms (broad learning concepts such as clear communication, problem solving, ethical awareness, etc.) Objectives are used to express intended results in precise terms (specific behaviors students should exhibit). Objectives tell us more specifically what needs to be assessed, and thus a more accurate guide to suitable assessment tools.

23. Goals and Objectives Both terms are used interchangeably Both terms describe the intended results of educational activities Both terms provide direction for assessment

24. Program Educational Objectives Statements that describe the expected accomplishments of graduates during their first few years of career after graduation as a result of their educational preparation. Types of program educational objectives: Objectives that all graduates are expected to accomplish; Objectives that some graduates are expected to accomplish. Goals and Objectives

25. Audiences for objective statements: Constituents external to the institution including: Prospective students Parents Student sponsors, donors, and contributors Employers Transfer institutions, graduate programs Industry advisors Accrediting agencies Government organizations Goals and Objective

26. Features of Good Statements of Program Educational Objectives Relevant to the mission of the institution; Each addresses one or more needs of one or more of the program constituents; Understandable by the program constituents; Clear, concise, and consistent; Realistic/achievable; Measurable; Forward looking (not short term); Manageable list (limited number of statements); Focus on student learning. Goals and Objectives

27. Program Outcomes Statements that describe what students are expected to know and be able to do by the time of graduation. These relate to the knowledge, skills and behaviors that students acquire in the program. Intended outcomes (from described statements of expected learning outcomes)

28. Program Outcomes Actual outcomes (form the results of assessment activities) Achievements of all program outcomes should indicate that the graduate is prepared to achieve the program educational objectives and should be verified before certification for graduation.

29. Program Outcomes: Features of Good Statements of Intended Outcomes: Relative to the missions of the program and institution; Describe the expected learning outcomes for students from individual courses/the program as a whole; Reflect what the students are expected to know and be able to do rather than what the course/program intends to cover; Each describes an area of knowledge and/or skill that a person is expected to possess;

30. Supportive of one or more educational objectives; Describe an observable behavior using action verbs (i.e. apply, identify, solve, construct, etc.); Avoid vague words that might be open to interpretation (i.e. understand, appreciate, etc.); Assessable (outcomes can be demonstrated in terms of performance). Program Outcomes:

31. Assessment Methods I. Direct Assessment Methods: Directly determine whether students have mastered the content of their academic programs. Require students to display their knowledge and skills as they respond to the instrument itself (i.e. objective tests, essays, presentation, and classroom assignments).

32. Assessment Methods Direct assessment methods include:  Standardized exams  Locally developed exams  Oral exams  Portfolios (work collected over time)  Performance appraisal

33. Assessment Methods Direct assessment methods include (Cont.):  Oral presentations  Projects, demonstrations, case studies, simulations  Capstone experience (embodied in capstone courses)  Juried activities  Evaluation of field work.  Behavioral observations

34. Assessment Methods II. Indirect Assessment Methods: Ask students to reflect on their learning, what they have learned and experienced, rather than to demonstrate it (i.e. surveys and interviews). Details about instructional or curricular strengths that can not be provided by direct methods alone.

35. Assessment Methods Indirect assessment methods include:  Written surveys and questionnaires oEntering students oCurrent students oGraduating seniors oFaculty oAlumni oEmployers oParents  Exit interviews  Focus groups  External examiner  Archival records

36. Pre-College Experience Curriculum:  Major  Courses  GE Educational Outcomes Educational Objectives Surrounding Environment:  Campus life  Social life Classroom Experience:  Pedagogy  Faculty  Facilities Out-of-Class Experiences:  Co-op  Internship College Educational ExperiencePost-College Experience Pre-College Educational Experience “Primary” Educational Outcomes Assessment Zone Educational Objectives Assessment Zone Program Outcomes Influencing Factors

37. Assessment Key Success Factors Assessment Key Success Factors Include: Commitment Participation (involvement) of all constituents Listening to voices Actions for improvement Feedback Relating assessment to the world of work Continuity and flexibility Consistency of reporting format over time History of results Responsibility Resources Rewards

38. THANK YOU