1. ABET Accreditation Criterion 4: Continuous Improvement Direct Assessment of Learning Outcomes Dr. Abdel-Rahman Al-Qawasmi Associate Professor EE Department 11/10/2015Criterion 4: Improvement Process Quality Vice-Deanship in the college of Engineering Majmaah University

2. ABET? ABET, incorporated as the Accreditation Board for Engineering and Technology. ABET is the recognized U.S. accreditor of college and university programs in applied science, computing, engineering, and technology 11/10/2015Criterion 4: Improvement Process

3. Goal of ABET To promote Continuous Quality Improvement in Applied Sciences, Computing, Engineering, and Technology education through faculty guidance and initiative 11/10/2015Criterion 4: Improvement Process

4. ABET Criteria 11/10/2015Criterion 4: Improvement Process Students (1) Students (1) Program Educational Objectives (2) Student Outcomes (3) Continuous Improvement (4) Facilities (7) Institutional Support (8) Curriculum (5) Faculty (6)

5. Criteria for Accrediting Engineering Programs Program Educational Objectives Student Outcomes Assessment Evaluation 11/10/2015Criterion 4: Improvement Process

6. Continuous Improvement (4) 11/10/2015Criterion 4: Improvement Process Important : Should document your processes for regularly assessing and evaluating the extent to which the student outcomes are being attained [Exams, student portfolios, senior project presentations] Assessment: is defined as one or more processes that identify, collect, and prepare the data necessary for evaluation Evaluation: is defined as one or more processes for interpreting the data acquired though the assessment processes in order to determine how well the student outcomes are being attained

7. 11/10/2015Criterion 4: Improvement Process Assessment for Quality Assurance © Learning Outcomes Constituents Assessment: Assessment: Collection, Analysis of Evidence Evaluation: Evaluation: Interpretation of Evidence Feedback for Continuous Improvement Measurable Performance Criteria Educational Practices/Strategies Mission Educational Objectives Assess/ Evaluate

8. 11/10/2015Criterion 4: Improvement Process ABET TermsDefinition Objectives Broad statements that describe the career and professional accomplishments that the program is preparing graduates to achieve. Outcomes Statements that describe what students are expected to know and able to do by the time of graduation. Performance Criteria Specific, measurable statements identifying the performance(s) required to meet the outcome; confirmable through evidence. Assessment Processes that identify, collect, use and prepare data that can be used to evaluate achievement. Evaluation Process of reviewing the results of data collection and analysis and making a determination of the value of findings and action to be taken.

9. Why we need LOs? evidence for accountability LOs build evidence for accountability, accreditation, and for continuous improvement. Know what Instructor is doing and whyKnow what students are learning as a result Make improvement changes based on results (research based improvements) 11/10/2015Criterion 4: Improvement Process

10. Step-by-Step Assessment 11/10/2015Criterion 4: Improvement Process Choose your one of the Assigned to your course ABET student Outcomes Students outcomes are assigned to your course and Approved Choose the assessment method Exams, Quiz, Project Analyze the results of assessment KPIs and Rubrics Final Result is below the Target for Performance Yes recommendation and action plan The internal Target for Performance is 75% Your comments and observations

11. ABET student Outcomes a-k (a) an ability to apply knowledge of mathematics, science, and engineering (b) an ability to design and conduct experiments, as well as to analyze and interpret data (c) an ability to design a system, component, or process to meet desired needs within realistic constraints such as economic, environmental, social, political, ethical, health and safety, manufacturability, and sustainability (d) an ability to function on multidisciplinary teams (e) an ability to identify, formulate, and solve engineering problems (f) an understanding of professional and ethical responsibility (g) an ability to communicate effectively (h) the broad education necessary to understand the impact of engineering solutions in a global, economic, environmental, and societal context (i) a recognition of the need for, and an ability to engage in life-long learning (j) a knowledge of contemporary issues (k) an ability to use the techniques, skills, and modern engineering tools necessary for engineering practice. 11/10/2015Criterion 4: Improvement Process

12. 11/10/2015Criterion 4: Improvement Process Data do not assist in driving improvement. To drive any improvement, one needs information - a detailed presentation of the results.

13. SLO, KPIs and Rubrics ABET learning outcomes, Key performance Indicators (KPIs) and Rubrics are ready and approved for every program in the college of Engineering. Refer to the program coordinator to get all information about your course All instructors should be involved in the evaluation process in order to evaluate the program 11/10/2015Criterion 4: Improvement Process

14. Information for Driving Improvement 11/10/2015Criterion 4: Improvement Process Learning Outcome Assessment Area1>B Area 2<B Area 3>B Area 5>B Area 4>B Sub area 1>B Sub area 2<B Sub area 3>B Sub area 4<B The recommended actions for improvement for the student outcome are related to Sub area 2 and Sub area 4

15. 11/10/2015Criterion 4: Improvement Process What is a rubric, anyway????? A rubric is a set of categories which define and describe the important components of the work being completed, critiqued, or assessed. Each category contains a gradation of levels of completion or competence with a score assigned to each level and a clear description of what performance need to be met to attain the score at each level.

16. The recommended actions for improvement Actions Related to the Course Objectives Actions Related to Student Outcomes Actions Related to teaching Strategies Actions Related to Assessment Strategies Actions Related to Other aspects 11/10/2015Criterion 4: Improvement Process The recommended actions should be written by the end of semester in the course Report

17. 11/10/2015Criterion 4: Improvement Process Assessment Methods Written surveys and questionnaires Exit and other interviews Standardized exams Locally developed exams Archival records Focus groups Portfolios Simulations Performance Appraisal External examiner Oral exams Behavioral observations

18. 11/10/2015Criterion 4: Improvement Process Direct Measures Direct measures provide for the direct examination or observation of student knowledge or skills against measurable learning outcomes

19. 11/10/2015Criterion 4: Improvement Process Indirect Measures Indirect measures of student learning that ascertain the opinion or self-report of the extent or value of learning experiences

20. 11/10/2015Criterion 4: Improvement Process Direct Indirect Exit and other interviews Standardized exams Locally developed exams Portfolios Simulations Performance Appraisal External examiner Oral exams Behavioral observations Written surveys and questionnaires Exit and other interviews Archival records Focus groups

21. Assessment Methods Student Learning Outcomes (code) Reports, discussions and presentationsa1 Exams and presentationsa2 Standardized exams, Oral exams, Micro projectsb1 Reports and presentationsb2 Standardized exams, Oral exams, Micro projectsb3 Standardized exams, Oral exams, Micro projectsb4 Behavior observation and reportsb5 Behavior observation and presentationsc1 Discussionsc2 Reports, discussions and presentationsc3 Standardized exams, Oral exams, Micro projectsd1 Reports, discussions and presentationsd2 Exams, quizzes and reportsd3 Standardized exams, Oral exams, Micro projectsd4 11/10/2015Criterion 4: Improvement Process

22. Rubric Satisfactory Developing Unsatisfactory Formulates an experimental plan of data gathering to attain a stated objective (develop correlation, test a model, ascertain performance of equipment, etc.) Develops a simplistic experimental plan of data gathering, does not recognize entire scope of study (e.g. not all parameters affecting the results are investigated) No systematic plan of data gathering; experimental data collection is disorganized, even random, and incomplete experimental plan of data gathering (2) (9)KPIs Develops and implements logical experimental procedures Experimental procedures most often followed, but occasional oversight leads to loss of experimental efficiency and/or loss of data Does not follow experimental procedure Development and implementatio n of logical experimental procedures (4) Analyzes and interprets data carefully using appropriate theory; if required, translates theory into practice or applies to process model(s) Applies appropriate theory to data when prompted to do so, but misinterprets physical significance of theory or variable involved; makes errors in unit conversions Makes no attempt to relate data to theory The analysis and interpretations of data using appropriate theory (7) LO: An ability to design and conduct experiments, as well as to analyze and interpret data 11/10/2015Criterion 4: Improvement Process

23. Example: 11/10/2015Criterion 4: Improvement Process Question1: Experiment Lab (measuring current) SLO: An ability to design and conduct experiments, as well as to analyze and interpret data Maximum grade :15 Number of students: 14 NoGradesNoGrades 1785 2896 310 7 43111 5 1214 6121312 781415 Rubrics Recommendations Write your recommendations based on your observations and the result of your used assessment method Recommendations Write your recommendations based on your observations and the result of your used assessment method KPI (2): Experimental plan of data gathering KPI (2): Experimental plan of data gathering Unsatisfactory Range(0-9) (1) No systematic plan of data gathering; experimental data collection is disorganized, even random, and incomplete Developing Range(9-12) (2) Develops a simplistic experimental plan of data gathering, does not recognize entire scope of study (e.g. not all parameters affecting the results are investigated) Satisfactory Range (12-15) (3) Formulates an experimental plan of data gathering to attain a stated objective (develop correlation, test a model, ascertain performance of equipment, etc.) 8 8 4 4 2 2 Final Result (1X8 + 4X2 + 2X3)/(3X14) =57.1% (weight) Or (4+2)/14= 42% (Student percentage) Final Result (1X8 + 4X2 + 2X3)/(3X14) =57.1% (weight) Or (4+2)/14= 42% (Student percentage)

24. Results Analysis Strength points: 1- The students average performance is good. 2- The plan of the experimental is well defined. Weaknesses points: 1- There is an average problem in applying the procedure. 2- There is a problem in Identifying and collecting data in organizing way Recommended actions: 1- Data collection methods and Statistical analysis should be give more attention 2- Including a statistical analysis and data collection method in the course description or in other low level course. 11/10/2015Criterion 4: Improvement Process

25. Make it Simple! 11/10/2015Criterion 4: Improvement Process Learning Outcome Assessment MethodQuestion (a)First ExamQuestion 1 (e)Second ExamQuestion 3 (h)Mini-ProjectPresentation (h) and (k)Final ExamQuestion 3 and Question 5 Your Plan for outcomes assessment:

26. Course Report The required course Report by ABET at the end of the semester is simple. Collect all results from all assessment methods you used during the semester to fill your report for every learning outcome assigned and approved for your course 11/10/2015Criterion 4: Improvement Process ABET LOabcdefjhijk EE XXX√√√√ (%)77678076

27. EE program SOs 11/10/2015Criterion 4: Improvement Process SLO Course (EE4)(EE3)(EE2)(EE1)(k) (j) (i)(h)(g)(f)(e)(d)(c) (b) (a) X X XEE 101 X X X XEE 111 X XX X XEE 202 XX X EE 205 XX X X XEE 206 X XXXEE 207 X X X X XEE 208 X XXXEE 212 XX X X XEE 221 X X X XEE 234 X XX XEE 270 XX XXEE 271 X X X XEE 288 X XX X EE 307 XX XX EE 308 XX X X EE 322 X X XX EE 323 X X XXX EE 341 X XX X XEE 360 X XX EE 361 Table 4-1-a: Student outcomes mapping for general courses

28. 11/10/2015Criterion 4: Improvement Process

29. Automation of Assessment Process 11/10/2015Criterion 4: Improvement Process

30. Thank you for Listening 11/10/2015Criterion 4: Improvement Process