Program Assessment Process for Continuous Quality Improvement (CQI) Prepared by: Abul Fazal M. Arif Member, ABET Steering Committee November 16, 2008.

ABET Steering CommitteeProgram Assessment Process 2 Contents ABET General Criteria, Cycle Assessment Process – key components Tasks to be Done Closing the Loop Action Plan

ABET Steering CommitteeProgram Assessment Process 3 ABET GENERAL CRITERIA “Renumbered” Criteria:( Accreditation Cycle) Criterion 1. Students Criterion 2. Program Educational Objectives Criterion 3. Program Outcomes Criterion 4. Continuous Improvement Criterion 5. Curriculum Criterion 6. Faculty Criterion 7. Facilities Criterion 8. Support Criterion 9. Program Criteria

ABET Steering CommitteeProgram Assessment Process 4 Assessment Process for Quality Assurance Assessment is an ongoing process of setting high expectations for student learning, measuring progress toward established learning outcomes, and providing a basis for reflection, discussion and feedback to improve University academic programs. It is a systematic and cyclic process that makes expectations and standards explicit and public.

ABET Steering CommitteeProgram Assessment Process 5 Key Components of Assessment Process Performance Criteria Assessment Methods Evaluation Curriculum Mapping Closing the Loop Action Taken / To be Taken Program Learning Outcomes Program Educational Objectives

ABET Steering CommitteeProgram Assessment Process 6 Performance Criteria Assessment Methods Evaluation Curriculum Mapping Closing the Loop Action Taken / To be Taken Program Learning Outcomes Program Educational Objectives Educational objectives are statements that describe the expected accomplishments of graduates during the first few years after graduation—usually 3-5 years.

ABET Steering CommitteeProgram Assessment Process 7 Performance Criteria Assessment Methods Evaluation Curriculum Mapping Closing the Loop Action Taken / To be Taken Program Learning Outcomes Program Educational Objectives Assessment Process for Quality Assurance These are statements related to skills, knowledge and behaviors that describe what students are expected to know and/or be able to do by the time of graduation.  If students have achieved these outcomes, it is anticipated that they will be able to achieve the educational objectives after graduation.

ABET Steering CommitteeProgram Assessment Process 8 Assessment Process (cont.) At the heart of assessment are the Program Objectives and Outcomes. Purpose: To assess the extent a program is achieving its objectives and outcomes. To be conducted continuously at different stages  continuous improvement. The assessment process itself should be evaluated and refined in light of emerging insights. All accrediting bodies, including ABET, require evidence of continuous quality improvement of both, Process and Program.

ABET Steering CommitteeProgram Assessment Process 9 Tasks to be done Task 1: Review Program Educational Objectives (PEO) Task 2: Review Program Learning Outcomes Task 3: Align Learning outcomes with PEO Task 4: Present to Constituents Task 5: Get Approval & Publish Task 6: Mapping Course Outcomes to Program Outcomes Task 7: Develop Performance Indicators Task 8: Identify Assessment Tools/Methods Task 9: Develop Assessment Plan Task 10: Data Collection and Evaluation Task 11: Closing the Loop

ABET Steering CommitteeProgram Assessment Process 10 TASK 1: Review Program Educational Objectives (PEO) Review process will be initiated by Department ABET Committee or PAC Guidelines for review 1.PEO should be consistent with proper definition. 2.PEO should be aligned with mission of university/College/Department. 3.Numbers of PEO should be manageable. 4.PEO should be measurable.

ABET Steering CommitteeProgram Assessment Process 11 STEP 1: Revise Program Educational Objectives Examples of PEO  Prepare graduates who will be successful in mechanical engineering-related careers and other diverse career paths  Prepare graduates to be leaders  Prepare graduates to be good citizens of the society  Prepare graduates to pursue advanced degree, if so desired  Prepare graduates to be life long learners  Prepare graduates to be entrepreneurs

ABET Steering CommitteeProgram Assessment Process 12 TASK 1: Review Program Educational Objectives Examples of PEO - Combined  Prepare graduates who will have successful careers, and become leaders, in industry and the public sector  Prepare graduates who will appropriately apply acquired knowledge, work well in team, effectively communicate ideas and technical information, and continue to gain knowledge  Prepare graduates who will continue professional development and will pursue continuing education opportunities relevant to their careers Note: For more examples of PEO, refer to Aggarwal’s workshop notes.more examples of PEO

ABET Steering CommitteeProgram Assessment Process 13 TASK 2: Review Program Learning Outcomes Task 2.1: Engineering programs must demonstrate that their students attain the following (a) to (k) outcomes: (a) an ability to apply knowledge of mathematics, science, and engineering (b) an ability to design and conduct experiments, as well as to analyze and interpret data (c) an ability to design a system, component, or process to meet desired needs within realistic constraints such as economic, environmental, social, political, ethical, health and safety, manufacturability, and sustainability (d) an ability to function on multidisciplinary teams

ABET Steering CommitteeProgram Assessment Process 14 Task 2: Review Program Learning Outcomes (cont.) (e) an ability to identify, formulate, and solve engineering problems (f) an understanding of professional and ethical responsibility (g) an ability to communicate effectively (h) the broad education necessary to understand the impact of engineering solutions in a global, economic, environmental, and societal context (i) a recognition of the need for, and an ability to engage in life-long learning (j) a knowledge of contemporary issues (k) an ability to use the techniques, skills, and modern engineering tools necessary for engineering practice

ABET Steering CommitteeProgram Assessment Process 15 Task 2: Review Program Learning Outcomes (cont.)  Task 2.2: Check if any additional outcomes that may be articulated by the program criteria.  Check Criterion 9 on Program Criteria from ABET , (For Mechanical Engineering) Mechanical Engineering program criteria include the requirement that the program must demonstrate that graduates have the ability to: apply principles of engineering, basic science, and mathematics (including multivariate calculus and differential equations) to model, analyze, design, and realize physical systems, components or processes; and work professionally in both thermal and mechanical systems areas.

ABET Steering CommitteeProgram Assessment Process 16 TASK 3: Align Learning outcomes with Program Objectives. Program outcomes must foster attainment of program educational objectives. Link outcomes with PEO All Learning Outcomes may or may not link to all/any PEO Program Learning Outcomes PEO(a)(b)(c)(d)(e)(f)(g)(h)(i)(j)(k) 1XXX 2XXX 3XXX 4XXX 5XXX

ABET Steering CommitteeProgram Assessment Process 17 TASK 4: Present to Constituents Main Stakeholders/Constituents  Faculty through Department Council  Employers through Industrial Advisory Committee  Students through various Professional Groups or Student Societies etc. Others:  Administration  Alumni  Other Graduate Schools  Recruiters  Employers Note: Mostly faculty define Program Learning Outcomes

ABET Steering CommitteeProgram Assessment Process 18 TASK 5: Get Approval & Publish Get Approval  Final form of the Program Educational Objectives and Learning Outcomes should be approved by the Department Council  Keep record in “ABET File” Publish at:  Department Website  General Catalog  Bulletin Boards etc Keep minutes of meeting of constituent’s involvement and Council approval as evidence in record.

ABET Steering CommitteeProgram Assessment Process 19 Performance Criteria Assessment Methods Evaluation Curriculum Mapping Closing the Loop Action Taken / To be Taken Program Learning Outcomes Program Educational Objectives Curriculum Mapping is the alignment of the program learning outcomes with the curriculum and, therefore, it needs development of performance indicators based on the curriculum.

ABET Steering CommitteeProgram Assessment Process 20 Performance Criteria Assessment Methods Evaluation Curriculum Mapping Closing the Loop Action Taken / To be Taken Program Learning Outcomes Program Educational Objectives TASK 6: Mapping of Course Learning Outcomes to Program Learning Outcomes  Include only Core (required) courses  Select courses for Outcome Assessment

ABET Steering CommitteeProgram Assessment Process 21 Mapping: Course Learning Outcomes and Program Outcomes

ABET Steering CommitteeProgram Assessment Process 22 Mapping: Core Courses & Program Outcomes

ABET Steering CommitteeProgram Assessment Process 23 Performance Criteria Assessment Methods Evaluation Curriculum Mapping Closing the Loop Action Taken / To be Taken Program Learning Outcomes Program Educational Objectives TASK 7: Develop Performance Indicators for each Program Outcome 

ABET Steering CommitteeProgram Assessment Process 24 Performance Criteria Assessment Methods Evaluation Curriculum Mapping Closing the Loop Action Taken / To be Taken Program Learning Outcomes Program Educational Objectives Performance Criteria (indicators, PIs) are defined as the specific, measurable statements identifying the specific knowledge, skills, attitudes and/or behavior students must demonstrate as indicators of achieving the outcomes.

ABET Steering CommitteeProgram Assessment Process 25 Examples of Performance Indicators Outcome (a): Ability to apply knowledge of mathematics……… 1. Applies mathematics principles to obtain solutions. 2. Uses principles of sciences and engineering in solving engineering problems. 3. Combines scientific and engineering principles to formulate models of processes and systems. 4. Combines mathematics principles to formulate models of processes and systems.

ABET Steering CommitteeProgram Assessment Process 26 Examples of Performance Indicators Note: For more examples of PI, refer to Sub-Committee Report.more examples Generally, it is recommended to limit performance criteria to three-four for each outcome.

ABET Steering CommitteeProgram Assessment Process 27 Performance Criteria Assessment Methods Evaluation Curriculum Mapping Closing the Loop Action Taken / To be Taken Program Learning Outcomes Program Educational Objectives Assessment is one or more processes that identify, collect, and prepare data to evaluate the achievement of program outcomes and program educational objectives.

ABET Steering CommitteeProgram Assessment Process 28 TASK 8: Identify Assessment Tools/Methods. Assessment tools are of two types: Direct measures are based on facts (it also includes opinion based on facts). Indirect measures are based on opinion only. Multi-source or multi-method approach is generally recommended for data collection and assessment. Indirect measures are necessary but not sufficient. They should be complemented with at least one direct measure. Sometimes it may not be possible to use direct methods to measure some of the outcomes (for example team work). In those cases, we can rely on indirect methods.

ABET Steering CommitteeProgram Assessment Process 29 Task 8: Identify Assessment Tools (cont.) Examples of Direct measures 1. Student course portfolios and senior capstone course projects using overall score of learning outcome rubric 2. External review or performance appraisal through direct observation on internship 3. Locally developed exams. 4. Oral exams 5. Standardized exams such as FE, GRE, GMAT 6. Special student achievements (awards, student publications and presentations at various meetings) 7. External examiners 8. Graduate schools

ABET Steering CommitteeProgram Assessment Process 30 Learning Outcome Rubric Performance Criteria Unsatisfactory 1 Developing 2 Satisfactory 3 Exemplary 4 Score Contribute Research & gather informati on. Does not collect any information that relates to the topic. Collects very little information – some relates to the topic. Collects some basic information – most relates to the topic Collects a great deal of information – all relates to the topic Take Responsibility Fulfill team role’s duties. Does not perform any duties of assigned team role. Perform very little duties. Perform nearly all duties. Perform all duties of assigned team role. Shear equally Always relies on others to do the work. Rarely does the assigned work – often needs reminding. Usually does the assigned work – rarely needs reminding. Always does the assigned work – without having to be reminded. Value Other’s Viewpoints Listen to other teammate s. Is always talking – never allows anyone else to speak. Usually doing most of the talking – rarely allows others to speak. Listens, but sometimes talk too much. Listen and speaks a fair amount Average Rubrics are developed from performance indicators and consist of sets of categories that define the process towards meeting the goal.

ABET Steering CommitteeProgram Assessment Process 31 Example of Rubric on Teamwork Performance Criteria Unsatisfactory 1 Developing 2 Satisfactory 3 Exemplary 4 Score Contribute Research & gather informati on. Does not collect any information that relates to the topic. Collects very little information – some relates to the topic. Collects some basic information – most relates to the topic Collects a great deal of information – all relates to the topic Take Responsibility Fulfill team role’s duties. Does not perform any duties of assigned team role. Perform very little duties. Perform nearly all duties. Perform all duties of assigned team role. Shear equally Always relies on others to do the work. Rarely does the assigned work – often needs reminding. Usually does the assigned work – rarely needs reminding. Always does the assigned work – without having to be reminded. Value Other’s Viewpoints Listen to other teammate s. Is always talking – never allows anyone else to speak. Usually doing most of the talking – rarely allows others to speak. Listens, but sometimes talk too much. Listen and speaks a fair amount Average

ABET Steering CommitteeProgram Assessment Process 32 Task 8: Identify Assessment Tools (cont.) Examples of Indirect measures Questionnaires and written surveys (Alumni and employer surveys) Senior exit surveys and interviews Student course evaluations Grades ** Focus groups conducted by trained moderator Students internship evaluations (if based on opinion) ** Refer to Sub-Committee Report or Aggarwal Workshop Notes.

ABET Steering CommitteeProgram Assessment Process 33 TASK 9: Development Assessment Plan Selection of Courses Data Collection Cycle Identify Data To be Collected

ABET Steering CommitteeProgram Assessment Process 34 Mapping: Core Courses & Program Outcomes

ABET Steering CommitteeProgram Assessment Process 35 Selection of Courses for Outcome Assessment

ABET Steering CommitteeProgram Assessment Process 36 Selected Courses for Outcome Assessment

ABET Steering CommitteeProgram Assessment Process 37 Data Collection Cycle OutcomeYear 1Year 2Year 3Year 4Year 5Year 6 (a)XX (b)XX (c)XX (d)XX (e)XX (f) (g)XX (h)XX (i)XX (j)XX (k)XX

ABET Steering CommitteeProgram Assessment Process 38 Data Collection Cycle OutcomeYear 1Year 2Year 3Year 4Year 5Year 6 (a)XX (b)XX (c)XX (d)XX (e)XX (f) (g)XX (h)XX (i)XX (j)XX (k)XX Learning Outcomes are measured on a continuous basis (3 years cycle is acceptable). You do not have to measure every student for every outcome. For PEOs, typically data is collected every three years from all constituents for every objective.

ABET Steering CommitteeProgram Assessment Process 39 Data to be collected For Program Learning Outcomes Students Course Evaluation using rubric Student Course Portfolios Grades Performance Appraisal Senior exit surveys and interviews Locally developed exams For Program Educational Objectives Alumni Survey Employers Survey Industrial Advisory Board’s Feedback Placement Data

ABET Steering CommitteeProgram Assessment Process 40 TASK 10: Collection of Data & Evaluation

ABET Steering CommitteeProgram Assessment Process 41 Performance Criteria Assessment Methods Evaluation Curriculum Mapping Closing the Loop Action Taken / To be Taken Program Learning Outcomes Program Educational Objectives Evaluation is one or more processes for interpreting the data and evidence accumulated through assessment practices. Evaluation determines the extent to which program outcomes or program educational objectives are being achieved, and results in decisions and actions to improve the program.

ABET Steering CommitteeProgram Assessment Process 42 Example of Evaluation  Evaluation using EXCEL fileEvaluation

ABET Steering CommitteeProgram Assessment Process 43 TASK 11: Closing the LOOP

ABET Steering CommitteeProgram Assessment Process 44 Performance Criteria Assessment Methods Evaluation Curriculum Mapping Closing the Loop Action Taken / To be Taken Program Learning Outcomes Program Educational Objectives The feedback process is critical to creating and maintaining a systematic quality assurance system. When successfully implemented, all elements of the quality assurance process interact with one another.

ABET Steering CommitteeProgram Assessment Process 45 Performance Criteria Assessment Methods Evaluation Curriculum Mapping Closing the Loop Action Taken / To be Taken Program Learning Outcomes Program Educational Objectives EXAMPLEEXAMPLE of Assessment Plan, Report & Rubrics

ABET Steering CommitteeProgram Assessment Process 46 ABET Accreditation: key Dates TaskResponsibilityDate 1.Complete the form for Programs Evaluation Academic Departments Send Request for Programs Evaluation To ABET HQ Dean Of Engineering Office Draft Self-Study Reports Ready Academic Departments Forward Draft Self-Study Reports to Quality Assurance Committee & English Editing Dean Of Engineering Office Receipt of Reviewed and Edited Reports Quality Assurance Committee & ELC Final Self Study Report ReadyAcademic Departments Preparation of the final ReportsDean Of Engineering Office Dispatch The Final Reports to ABET HQ Dean Of Engineering Office

ABET Steering CommitteeProgram Assessment Process 47 Tasks to be done Task 1: Review Program Educational Objectives (PEO) Task 2: Review Program Learning Outcomes Task 3: Align Learning outcomes with PEO Task 4: Present to Constituents Task 5: Get Approval & Publish Task 6: Mapping Course Outcomes to Program Outcomes Task 7: Develop Performance Indicators Task 8: Identify Assessment Tools/Methods Task 9: Develop Assessment Plan Task 10: Data Collection and Evaluation Task 11: Closing the Loop

ABET Steering CommitteeProgram Assessment Process 48 Task 3: Draft Self-Study Reports Ready By: Academic Departments Submission date: Task Nov 1-15 Dec Dec 1-15 Jan Jan 1-15 Feb Feb 1-15 Mar Mar

ABET Steering CommitteeProgram Assessment Process 49 Sources of Information Aggarwal’s workshop # 1 lectures Aggarwal’s workshop # 2 lectures ABET Sub-Committee Report

ABET Steering CommitteeProgram Assessment Process 50 THANK YOU