Assessment and Reporting Graduate Attributes Assessment and Reporting December 2016

Accreditation Requirements Assessment of “Graduate Attributes” (GAs) Use of the above results as part of a “Continual Improvement” (CI) process

Graduate Attributes - CEAB “Generic characteristics, specified by the Accreditation Board, expected to be exhibited by graduates of accredited Canadian engineering programs at the time of graduation.” 1. KB: Knowledge base for engineering 2. PA: Problem analysis 3. IN: Investigation 4. DE: Design 5. ET: Use of Engineering tools 6. IT: Individual and teamwork 7. CS: Communication skills 8. PR: Professionalism 9. IE: Impact of engineering on society and the environment 10. EE: Ethics and equity 11. EP: Economics and project management 12. LL: Life-long learning The graduate attributes broadly define the required outcomes, but they are difficult to measure directly.

Indicators Note: We are required to “measure” indicators, not GAs To measure graduate attributes, all universities must define their own measurable descriptors called “indicators”: “Descriptors of what students must do to be considered competent in the attribute; the measurable and pre-determined standards used to evaluate learning (i.e. measureable characteristics of attributes or components of attributes).” Note: We are required to “measure” indicators, not GAs

Indicators (cont.) The Faculty has developed a “universal” set of indicators See: https://www.mcgill.ca/engineering/faculty-staff-resources/accreditation/accreditation-resources or just Google “McGill Engineering Accreditation”

Our CI Process

Continual Improvement Process Step 1- Program Evaluation Step 2-Curriculum Mapping Step 3- Data Collection of Student Learning Step 4- Data Analysis and Interpretation Step 5- Data-informed Curriculum Improvement Step 5 – 1: Program Evaluation Feedback Loop   Step 5 – 2: Curriculum Map Feedback Loop

(Current) Procedure Quantitatively assess all GAs: at all levels: “I” (Introduced), “D” (Developed) or “A” (Applied), in all courses, in all years, for all students, by correlating performance in “assessment tools” with that in GAs / indicators However, to get satisfactory data for all GAs in a given program, departments must also qualitatively assess all GAs. use other approaches to evaluate “soft skills” GAs (e.g., IT, CS, PR, IE, EE, EP, LL) employ other approaches to evaluate GAs other than just correlating GAs with graded work use rubrics to evaluate GAs (either the Faculty’s universal rubric, or a customized one) A special emphasis on obtaining data for the “soft skills” GAs is undertaken in FACC 100 and FACC 400 (which are common to all programs). A new on-line course (FACC 250: Responsibilities of the Professional Engineer) has been introduced by the Faculty to give students more exposure to PR, IE, and EE

(Current) Procedure (cont.) The Faculty is providing general support to assist departments in meeting the necessary requirements to obtain, analyze and present their GA data. However, every course, program and department is different, so the process may not satisfy all situations Adaptation and customization will be required by each department CEAB guidelines and expectations are very limited Although the faculty can provide guidance, professors and departments will sometimes need to use their judgment, as many questions do not have clear answers!

GA/CI Organizational Structure Execution & Operation Level Management Level Leadership Level Associate Dean-Academic Programs F-CPEI Committee D-CPEI Committee Instructors/faculty members Administrative staff members Accreditation Assessment Specialist Pedagogical Coordinator

F-CPEI & D-CPEI In 2013, the Faculty-level Continuous Program Evaluation and Improvement (F-CPEI) committee was created. Chaired by the Associate Dean-Academic Programs Manages/monitors the implementation and development of the processes and procedures for outcome-based assessment for all Engineering programs. The management of outcome-based assessment for each program is carried out by the Department-level Continuous Program Evaluation and Improvement (D-CPEI) committee. Responsible for curriculum assessment and improvement implementation for undergraduate courses. All faculty members are engaged in the outcomes-based assessment process.

D-CPEI Membership Typical timetable for and purpose of D-CPEI meetings Leadership: Chair of the department Academic committee representative Faculty CPEI representative Faculty members External member(s) (i.e. members from other departments, universities and/or industry) Undergraduate student (VP (Academic) – Departmental student organization) Administrative Assistant Typical timetable for and purpose of D-CPEI meetings Smaller meetings in the Fall and Winter semesters Major annual meeting in the Summer to: Interpret departmental GA analysis Identify short-term actions Monitor long-term actions Approve Continual Improvement actions being forwarded to F-CPEI Keep minutes of these meetings!

Evaluating GAs

Overview Programs (i.e. correlating graded assessments with individual GAs) Infer GAs from assessment tools in courses General approach (most courses, Faculty does the analysis) Using myCourses (analysis is the responsibility of the prof. / dept.) Using Excel Alternate approach To ensure good coverage of GAs is obtained Used to evaluate GAs by other means

General Approach Use myCourses for the majority of courses to correlate individual student performance in certain GAs with the various assessment tools used by the instructor The gradebook setup is done by the instructors or TAs One-on-one consultation with professors is available Group TA training will be offered at the beginning of each semester Professors (only) to associate graduate attribute indicators with each graded item Professors (or TAs) are responsible for entering data (grades) in myCourses at a level that is sufficiently refined to distinguish between different graduate attribute indicators Analysis of the data will be done centrally

Gradebook

Alternate Approaches Departments will need to add to the general approach to: Acquire sufficient quality data for the “soft skills” graduate attributes Use alternate GA evaluation approaches to satisfy CEAB requirements regarding types and variety of measurement tools Maria Orjuela-Laverde (Academic Associate, Pedagogical Coordinator) is available for pedagogical assistance to develop alternate GA evaluation methods

Additional approaches (cont.) http://www.engineerscanada.ca/sites/default/files/intro_team_chair_presentation_english-sep19.pptx

Additional approaches (cont.) http://www.engineerscanada.ca/sites/default/files/intro_team_chair_presentation_english-sep19.pptx

Recommendations Recommendations to departments: Examine your curriculum maps to look for “holes” in the data Start by focussing on “Intro to XXX Engineering” course (or its closest approximation) and capstone design courses to complement your general approach for GA data acquisition List GA indicator codes (e.g. “KB.1,” “PA.2,” etc.) for each question/exam sheet for all graded items (assignments, labs, exams) List learning outcomes on all course outlines (a requirement for the revised course information sheets!) Remember to collect appropriate sample materials for your courses.

Faculty Assistance Analyzing and processing the data acquired via myCourses Providing support (Nasim and Maria) to both professors and TAs Providing reports on the Faculty-wide survey results for: Capstone design survey Alumni survey Obtaining GA data in common FACC courses Establish a faculty-wide “program improvement” procedure based on GA data

Misc. comments Some courses are not on myCourses For a “standard” course, the process has been designed so that the additional work required by a professor is small (as low as 2 hours per course) Subsequent years will require less work (e.g. myCourses gradebook template can be imported from previous years)

Course Information Sheet Learning Outcomes Course Information Sheet

http://www. engineerscanada http://www.engineerscanada.ca/sites/default/files/intro_team_chair_presentation_english-sep19.pptx

Course Learning Outcomes “What the student is expected to have learned - These should be analogous to indicators and may have been used for GA assessment purposes. A small number of outcomes (general measurable skills acquired) are expected - not a list of content.” http://www.mcgill.ca/tls/teaching/course-design/outline

Resources Website: Video clips Instructions TA Workshops (https://www.mcgill.ca/engineering/faculty-staff/accreditation) Short/Detailed list of graduate attributes and indicators Graduate Attribute Indicator Rubrics Video clips How to setup your gradebook Instructions Gradebook setup in myCourses Copying gradebook from one course to another Importing grades from Excel to myCourses TA Workshops One session is offered per semester

Thank you!