“Pathways to Excellence” September 24 - 27, 2017 | Fort Worth, Texas

QM Peer Review Past, Present & Future Ron Legon, Executive Director Emeritus & Senior Advisor for Knowledge Initiatives Quality Matters Melissa Poole, Director of Quality Assurance

The INQAAHE Project INQAAHE = International Network of Quality Assurance Agencies in Higher Education Project: The Impact of Faculty Peer Review on External Quality Assurance:The Quality Matters Case Methodology: Interviews and surveys of QM founders; analysis of post-review surveys; review of literature on peer review in higher ed. Published Study: forthcoming in Quality Assurance in Higher Education

What We’ll Cover Brief description of QM peer review Origins and intended purpose of QM peer review The historic role of peer review in higher education Acceptance of QM peer review in online learning Challenges of sustaining peer review The future of QM peer review - Discussion

What is QM Peer Review? The Peer Review Team for Higher Ed Online Courses 3 members Led by Master Reviewer All must have recent online teaching experience All must be QM trained, certified, and periodically updated Team must include at least one external member Team must include one Subject Matter Expert Faculty Course Developer treated as member of the team

What is QM Peer Review? Reviews must be collegial – aiming to make courses better Focusing on design, not on reviewers’ content preferences Constructive – suggesting steps to eliminate any deficiencies Encouraging – identifying what is done well

When Did QM Adopt Peer Review? Faculty to faculty peer review - was assumed from the beginning of the QM development process in 2003 Carried over from precursor process at Frederick Community College And never seriously questioned thereafter

Why Did QM Adopt Peer Review? Broad agreement among founders on desired effects of requiring faculty peer review Assert Faculty Ownership of a faculty only review process Demonstrate Independence from administrative control and P&T considerations to encourage voluntary participation Build Trust among faculty course developers by relying on peers Establish Rigor and Credibility by requiring online teaching experience & subject matter expertise Insure Objectivity by including one or more external reviewers Spread Standards through peer reviewers and Create Champions

Summative vs. Formative Peer Review QM founders were heirs to a long-standing tradition Centuries’ old practice of senior faculty reviewing juniors More recent tradition of faculty to faculty mentoring Mid-20th Century distinction between Formative and Summative Peer Review QM Founders adopted the lessons of Formative Peer Review Principles developed for the classroom setting Relevant elements were easily adapted to the online environment

Principles of Formative Peer Review* “Formative evaluation including rigorous descriptive strategies, along with ample feedback and opportunities for practice and coaching, is necessary for improved teaching.” “Care must be taken to minimize potential problems that can arise from having the same faculty involved in formative and summative evaluation of a colleague.” See Keig, L., & Waggoner, M. D., 1994. ‘Collaborative Peer Review: The Role of Faculty in Improving College Teaching’, ASHE-ERIC Higher Education Report No. 2. ASHE-ERIC Higher Education Reports, George Washington University, pp. 134-138. Available at http://files.eric.ed.gov/fulltext/ED378925.pdf Gosling D, 2002. ‘Models of peer observation of teaching’, Generic Centre: Learning and Teaching Support Network; Retrieved, 8(10), 08. Google Scholar; Teoh, S. L., Ming, L. C., & Khan, T. M, 2016. ‘Faculty Perceived Barriers and Attitudes Toward Peer Review of Classroom Teaching in Higher Education Settings: A Meta-Synthesis’, SAGE Open, 6(3), 2158244016658085.

Principles of Formative Peer Review* “Formative evaluation should include nonjudgmental descriptions of faculty members’ teaching by colleagues, administrators, and, where available, teaching consultants as well as students; each constituency should be asked for data only in areas where it is qualified to provide it.” “Faculty should be provided opportunities for training in the skills needed to conduct formative peer evaluation.” See Keig, L., & Waggoner, M. D., 1994. ‘Collaborative Peer Review: The Role of Faculty in Improving College Teaching’, ASHE-ERIC Higher Education Report No. 2. ASHE-ERIC Higher Education Reports, George Washington University, pp. 134-138. Available at http://files.eric.ed.gov/fulltext/ED378925.pdf Gosling D, 2002. ‘Models of peer observation of teaching’, Generic Centre: Learning and Teaching Support Network; Retrieved, 8(10), 08. Google Scholar; Teoh, S. L., Ming, L. C., & Khan, T. M, 2016. ‘Faculty Perceived Barriers and Attitudes Toward Peer Review of Classroom Teaching in Higher Education Settings: A Meta-Synthesis’, SAGE Open, 6(3), 2158244016658085.

Evidence that QM Peer Review Has Achieved Its Aims Wide adoption of QM Standards and review practices From the original 20 MarylandOnline institutions in 2006 To 80 institutions that joined QM in 2006-7 To 1,000+ institutions by 2017 Grassroots support of faculty champions Steady conversion of skeptics to trust in QM’s review process

Surveys of Reviewers and Course Designers from 2008 – 2014 On a 5-point Likert Scale: 1,578 Course Developers gave the peer review process a 4.65 approval score, and 5,601 Peer Reviewers gave their reviewer colleagues a 4.78 approval score

Challenges to Sustaining QM Peer Review at Scale Constant efforts to replenish the reviewer pool Expanding training to certify QM reviewers and keep them current Ongoing communication and monitoring to assure inter-rater reliability and resolve disputes Time demands – As institutions increase their use of course reviews there are calls for expedited reviews Cost - As institutions increase their use of course reviews there are calls for less costly reviews Relevance for team-designed courses – client may have less concern that review is based on peer faculty principles

The Future of QM Peer Review Preparatory Reviews Expedited reviews from one Rubric to another New review types for those creating/revising all courses to meet QM Standards Coaching Resource Sites – MRRS/PRRS

Audience Discussion Thank you! Discussion Questions: Evidence of positive (or negative) impact of QM peer review Value of preserving QM peer review process Acceptable modifications and alternatives to meet challenges Your questions Thank you!