Cyclical Program Review John Shepherd Vice-Provost and Associate Vice-President (Academic) Office of Quality Assurance

The Context – QA in Ontario  Ontario Universities’ Council on Quality Assurance (Quality Council) was established July 1, 2010  No more OCGS appraisals; No more UPR  The new regime covers the approval and review of all new and existing undergraduate and graduate programs.

The Context – QA in Ontario  Each university in Ontario has been required to develop its own Institutional Quality Assurance Process (IQAP) for ratification by the Quality Council.  Carleton’s Institutional Quality Assurance Process (IQAP): │Approved by Senate June, 2010 │Ratified by Quality Council March, 2011 │Revised to include DUC – Approved by Senate February, 2012 │Ratified by Quality Council May, 2012 │Joint Carleton-University of Ottawa IQAP approved by Senate January, 2012 │Ratified by Quality Council May, 2012

– What’s the point? To constantly improve programs that are already of good quality To have units feel the exercise was worthwhile and beneficial To constantly strengthen Carleton’s academic planning processes To help position Carleton advantageously in the changing Provincial context

The Carleton Process: Who Administers It?  Carleton’s Office of Quality Assurance (Academic Programs) (OQA): |Vice-Provost |Manager |Quality Assurance Officer |Program Review Co-ordinator  The Carleton University Committee on Quality Assurance (CUCQA): |Vice-Provost (Chair) |Provost (ex officio) |Associate Dean (Programs and Awards), FGPA (ex officio) |Associate University Librarian |7 senior faculty members |1 Dean

OQA and CUCQA – What’s Our Objective?  Quality assurance can have two purposes: |Program improvement |Accountability  Our objective is program improvement  We work with programs to support them  We are not the QA police!! Not just an exam to be passed and forgotten!

Cyclical Program Review Cyclical Program Review should not be approached as a hurdle to be overcome. Rather, it should be seen as an opportunity to strengthen programs that are currently of good quality or can achieve good quality as part of an overall process of continuous improvement.

Cyclical Program Review  Occurs on an 8 year cycle  Simultaneous review of graduate and undergraduate programs  Self-assessment of a program’s strengths and areas for improvement  Consultative – informed by input from faculty, staff, students, associate dean(s), dean(s), and external reviewers  20-month process

Cyclical Program Review - Steps  Self-Study (3 volumes)  Site Visit  Reviewers’ Report  Final Assessment Report  Action Plan

OQA – Here to Help Our objective is program improvement – we are here to help  Customized template for program review  Coordination of data, reports and other materials to inform program review  Compilation of data into the required tables. Customized data and reports may be available upon request.  Electronic document sharing site - cuCollab  Meetings with Review Team members  Review full and partial drafts of the self-study Require other assistance? Just ask!

Volume I: The Self-Study Critical, self-reflective, and program-centric  Program history – response from the last review  Program structure and delivery  Degree Level Expectations and Learning Outcomes and Learning Objectives  Students: Enrolment, retention, satisfaction  Faculty, staff resources  Program improvements

Volume I: Template  Each review team will have an electronic workspace on cuCollab for document sharing |OQA will post the template, data, reports, etc. on this site. |Review teams will also be able to post documents  Each Review Team will be provided with a customized template for Cyclical Program Review.  Model tables are included in the template |Data from OIRP and CURO will be provided. Tables will be populated by OQA. |Customized tables can also be generated.

How to Write a Bad Self-Study  Ineffective self-studies are: |Descriptive rather than reflective, analytical, self-critical, and evaluative |Defensive or aimed at justifying the status quo |Focused on the academic unit rather than the program(s) |Does not address or only superficially addresses Degree Level Expectations and Learning Outcomes |Raw data are attached as appendices, or only used in a descriptive manner |Written by a single faculty member without evidence of buy-in of faculty and students Adapted from: Quality Assurance Framework: Creating an Effective Self-Study for Program Reviews

Writing an Effective Self-Study  Effective Self-Studies are: |Reflective, analytical, self-critical, and evaluative |Aimed at quality improvement. Self-appraisal includes an analysis of strengths and weaknesses, and outlines how improvements can be made |Focused on the programs under review |Curriculum is fully examined, with an eye to Degree Level Expectations, Learning Outcomes, and to change and improvement. |Expresses Degree Level Expectations and learning objectives that operationally drive admission requirements, curriculum content, modes of delivery, bases of evaluation of student performance and commitment of resources. |Data are analyzed and contribute to the evaluation of strengths and weaknesses of the program. Adapted from: Quality Assurance Framework: Creating an Effective Self-Study for Program Reviews

Participant Experiences Dr. Stephen Godfrey Department of Physics

Degree Level Expectations and Learning Outcomes Program Learning Outcomes Departmental vision, characteristics of ideal graduate Course Learning Objectives Course design

Degree Level Expectations ➊ Depth and breadth of knowledge ➋ Undergraduate: Knowledge of methodologies Graduate: Research and scholarship ➌ Application of knowledge ➍ Communications skills ➎ Awareness of the limits of knowledge ➏ Autonomy and professional capacity COU Report: Ensuring the Value of University Degrees in Ontario - November 2011

Learning Objectives and Outcomes  Learning Objectives |Learning Objectives are the instructional goals of the program. They outline the program inputs or learning opportunities students will have over the course of the program  Learning Outcomes │ Learning Outcomes are the measurable skills, knowledge, competencies and/or behaviours that students will have as a result of successfully completing the program

Learning Objectives and Outcomes  For QA, program-level learning outcomes apply to programs not to individual courses.  We need to look at the individual courses to get a “picture” of how the learning outcomes are addressed within the program.  Model Tables enable an account of how program level learning outcomes are achieved and Degree Level Expectations are met.  A summative narrative is required of how courses as a collective achieve the intended learning outcomes.  Support will be provided in evaluating learning outcomes.

Volume II: Faculty CVs  The CVs of all faculty associated with the program need to be included.  Flexibility in format: No more OCGS template!  All CVs must be in same format. |Format to be approved by OQA.

Volume III: External Reviewers  10 external academic reviewers.  (4 external professional reviewers.)  An internal reviewer will be nominated by OQA, in consultation with the unit and Dean(s)

Timeline & Milestones September 2013  Review team is established October 2013  Attend OQA workshop  Review team begins developing student survey/focus group questions (template provided)  Review team confirms faculty/instructor information with OQA  Review team should begin work on Sections A-D of self-study, with particular focus on the development of program learning objectives/outcomes  OQA conducts student surveys/focus groups as applicable November/December 2013  Review team receives library reports, research funding tables, space management report, and survey/focus group reports

Timeline & Milestones January 2014  Review team receives data tables from OIRP February/March 2014  Review team should be finalizing the self-study  Faculty, students, associate dean(s), dean(s), should be consulted on drafts It is highly recommended that OQA review drafts of the brief and provide feedback before it is submitted. April 1, 2014  Volumes I, II, and III submitted to OQA

Timeline & Milestones  Once OQA confirms that the brief is ready, it is submitted to the Carleton University Committee on Quality Assurance (CUCQA): |Assigned a discussant, who prepares a report to CUCQA |Program Review Lead is invited to attend the CUCQA meeting at which the program is discussed |Supplementary questions (dean consulted) |Review Committee selected (dean consulted)

Timeline & Milestones Summer 2014  In consultation with the unit, OQA will arrange the site visit for fall Fall 2014  Two day site visit: external reviewers meet with dean(s), faculty, staff, and students  Within one month of the site visit, the external reviewers submit a report to OQA  The Review team prepares a response to the report, in consultation with the dean(s)  CUCQA receives the external reviewers report and the Review team ’s response.

Timeline & Milestones  CUCQA considers the brief, report and response: |Discussant recommendation report |CUCQA recommends a categorization: Good Quality with international or national presence Good Quality Good Quality with report Conditional approval Not approved to continue │Categorization sent to unit and dean(s)  Action plan requested

Timeline & Milestones Winter 2015  Action plan submitted to CUCQA  CUCQA: Final Assessment Report and Executive Summary │Submitted to the Provost  Provost considers Report and Summary  Documentation to SAPC/Senate for approval  academic unit;  dean(s);  Board of Governors;  Quality Council;  Carleton website

Participant Experiences Dr. Mitchell Frank Institute of Comparative Studies in Art and Culture

Help!?  Office of Quality Assurance |Office of Institutional Research and Planning |Carleton University Research Office |Office of Space Management and Capital Planning |University Library  Undergraduate Programs: Faculty undergraduate associate deans  Graduate Programs: Associate Dean (Programs and Awards), FGPA

Contact Office of Quality Assurance 421 Tory Building Jessica DeVries, Manager (Interim) & Program Review Coordinator x 3231 Ann Clarke-Okah (OQA Consultant)