Quality Matters Overview Deborah Adair, Ph.D. Director, Quality Matters October 14, 2008

“Quality Matters: Inter-Institutional Quality Assurance in Online Learning”  Quality Matters is a not-for-profit subscription service providing tools and training for quality assurance of online courses  Colleges and universities, both public and private, throughout the country are joining our community of practitioners

QM Process Provides:  Institutional toolset and process to meet quality expectations:  Online course design  Student learning  Improved instruction  Assessment and feedback loops  Professional development

QM as a National Standard  current subscribers (  42 states represented  QM has trained faculty and instructional design staff  Recognized by - Sloan C Excellence in Online Teaching and Learning Award USDLA Outstanding Leadership in the field of Distance Learning

MONTANA WYOMING IDAHO WASHINGTON OREGON NEVADA UTAH CALIFORNIA ARIZONA NORTH DAKOTA SOUTH DAKOTA NEBRASKA COLORADO NEW MEXICO TEXAS OKLAHOMA KANSAS ARKANSAS LOUISIANA MISSOURI IOWA MINNESOTA WISCONSIN ILLINOIS INDIANA KENTUCKY TENNESSEE MISS ALABAMA GEORGIA FLORIDA SOUTH CAROLINA NORTH CAROLINA VIRGINIA WV OHIO MICHIGAN NEW YORK PENN MARYLAND DELAWARE NEW JERSEY CONN RI MASS MAINE VT NH ALASKA HAWAII PUERTO RICO VIRGIN ISLANDS Current Subscribers Statewide Subscribers QM Subscribers

Principles of QM The QM toolset and process  A faculty-driven, peer review process that is… Collaborative Collegial Continuous Centered - in academic foundation - around student learning  Courses do not have to be “perfect” but QM aims at better than just “good enough”

What Quality Matters is NOT  Not about an individual instructor (it’s about the course design)  Not about faculty evaluation (it’s about course quality)  Not a win/lose or pass/fail test (it’s a diagnostic tool to facilitate continuous improvement of online/hybrid courses)

Goal: Make online instruction as good as it can be  Better than average; more than “good enough”  An attempt to capture what’s expected in an effective online or hybrid course at about an 85% level 85 %

History of Quality Matters Winter 2008

Quality Matters Was Launched by MarylandOnline  Maryland consortium for distance learning received 3-year FIPSE grant in 2003  Motivation: Need to provide inter-institutional quality assurance for courses in consortium seat bank  Approach: Research-based, collaborative, peer- centered  Outcome: Sustainable quality assurance process embraced by institutions beyond MOL and Maryland

Success of the QM Grant  Early presentations generated widespread interest  MarylandOnline began to receive recognition for QM  WCET Outstanding Work (WOW) Award, 2005  USDLA 21 st Century Best Practice Award, 2005  Maryland Distance Learning Association (MDLA) Best Program Award,  The Sloan Consortium online workshops introduced hundreds of faculty members and staff to QM.  Peer reviewer training spread far beyond Maryland:  700+ faculty trained to review online courses using the rubric  individuals from 158 different institutions in 28 states

QM Today  Self-supporting non-profit program  subscriptions, trainings, and course reviews  Wide-spread adoption  Independent subscribers (75), consortium subscribers (67), and statewide system affiliates (136)  Program development through collaboration  User community directs program improvement  Continuous improvement of tools and services

QM Basics: More than the Sum of its (3) Parts Winter 2008

The Rubric

 Eight standards:  Course Overview and Introduction  Learning Objectives  Assessment and Measurement  Resources and Materials  Learner Interaction  Course Technology  Learner Support  Accessibility Key components must align.

Rubric Scoring  Points are awarded for 40 specific standards based on:  the team majority, AND  the pre-assigned weighting of each standard  Specific standards have a point value of 1, 2, or 3; the total points possible in a review is 85  If 2-3 Reviewers believe that a standard is:  met, then the full pre-assigned points are awarded  not met, then zero points are awarded

To Meet Expectations… A course must achieve:  “Yes” on all 17 of the 3-point “essential” standards  A minimum of 72 out of 85 points 72/85 = 85%

The Peer Review Process

Peer Course Review Feedback Course Instructional Designers Institutions Faculty Course Developers National Standards & Research Literature Rubric Course Meets Quality Expectations Course Revision Quality Matters: Peer Course Review Process Training Faculty ReviewersCourse continues to be offered

 Characteristics:  Experienced online/hybrid instructor  Trained in applying the Quality Matters Standards and certified by QM  Training updated with each new edition of the rubric  Receives a small stipend for each review The Peer Reviewer

 Consists of:  3 QM Certified peer reviewers  The chair is a Master Reviewer (having received additional training)  One reviewer must be a subject matter expert  At least one reviewer must external to the institution sponsoring the course The Peer Review Team

What to Expect from Course Reviews  Consistency and Rigor  Professionalism and Commitment  Useful and Constructive Feedback  Positive Outcomes for Faculty

Survey of faculty whose course was reviewed indicates that …  91% of respondents made changes in the course as a result of the review  89% of respondents felt that the quality of course design improved as a result of the review Sample comments: “I was too close to see what could be improved.” “Provides a great way to get an objective view of your course.” “It made all of my online courses better.” “It provides a view from a more student oriented perspective.” “Many elements that might contribute to a student withdrawing can be eliminated.”

Quality Matters Training

QM Training  QM trainings support the other program components and are integral to a quality assurance effort  QM principles and the rubric standards are at the heart of all the trainings  There are two general categories of training: Faculty Development and Implementing Quality Matters

Faculty DevelopmentImplementing Quality matters Types of Trainings  Applying the QM Rubric  Building Your Online Course Using the QM Rubric  Improving Your Online Course Using the QM Rubric  Design that Welcomes Your Students  Measurable Learning Objectives at the Course and Module Level  Choosing and Using Media Effectively  Peer Reviewer Certification  Master Reviewer  Train-the-Trainer  Institution Representative/ Course Review Manager

Why Quality Matters? Winter 2008

Multiple Uses of QM Reported Uses of QM Tools:  Guidelines for initial online course development  Quality assurance of existing courses  Ongoing faculty professional development  Institutional re-accreditation packages  Focus attention on distance learning policies & steering committees

Alignment with Accrediting Best Practices Best Practices Principles*Quality Matters Principles That education is best experienced within a community of learning where competent professionals are actively and cooperatively involved with creating, providing, and improving the instructional program; QM is a peer review process involving faculty, instructional designers and other support staff in a cooperative effort to continuously improve online instruction. That learning is dynamic and interactive, regardless of the setting in which it occurs; QM treats interactivity and active learning a critical component of every online course. That instructional programs leading to degrees having integrity are organized around substantive and coherent curricula which define expected learning outcomes; QM treats the alignment of expected learning outcomes with the contents, activities and assessments as a critical element in every online course. That institutions accept the obligation to address student needs related to, and to provide the resources necessary for, their academic success; QM expects every online course to address student access to the academic, technical, and student support services essential to student success.  hat institutions are responsible for the education provided in their name; Adoption of QM standards reflects institutional commitment to online instructional quality, wherever an institution has endorsed the rubric standards.  hat institutions undertake the assessment and improvement of their quality, giving particular emphasis to student learning; The QM standards are based on research and best practices to enhance student learning in online environments. Adoption of the QM review process is a clear demonstration of institutional or programmatic commitment to assessment and continuous improvement.  hat institutions voluntarily subject themselves to peer review. QM is essentially a peer review process involving both internal and external peers in the evaluation of courses. *“Best Practices for Electronically Offered Degree and Certificate Programs” adopted in 2001 by CHEA and 8 regional accreditation bodies.

Collaboration and Community in Distance Education  QM provides platforms to help users collaborate...  With QM in building tools that work for Distance Education  With their own institutional users to adapt QM tools to their specific institutional needs  Across the community of QM subscribers to share best practices, find new resources, and support collegial efforts to improve student learning outcomes

What’s In It For Faculty?  Improvement of online courses  External quality assurance  Review other courses and gain new ideas for own course; expand professional community  Participation useful for professional development plan and portfolio  Faculty development trainings

Winter 2008 Quality Matters Implementation

QM’s Role in Quality Assurance  QM looks at course design The harnessing of technology to deliver instruction and promote student learning  QM provides a process for peer-to-peer feedback for faculty in the continuous improvement of their course  Quality Matters is not the complete answer to quality assurance for online education, but it can be a critical component

Campus Decisions for Quality  Governance  Who will lead a QM project and where will it report?  Engagement  How will you gain faculty commitment?  Communication, training opportunities, incentives…?  Rubric Use  For course assessment, development, maintenance?  Course Reviews  Mandatory or optional, official or informal?  Selection Criteria  Which courses in what order?

Types of Course Reviews  QM-Managed Course Reviews  QM staff assigns review team and manages process  Available with a Supplemental Package, otherwise on an as-available basis  Official, Subscriber-Managed Course Reviews  Trained institution representative manages according to QM criteria  Independent Course Reviews  Schools set up own informal process

Scalability in Implementing QM Managing Your ReviewsDelivering Your Training (APPQMR) Customized (not recognized by QM) Internal QA No QM requirem ents No QM fees QM- Managed External QA Courses must meet submissio n guidelines Fee for Service Subscriber- Managed External QA Review follows official QM process Fee for certification Customized (not recognized by QM) Adapt for internal needs For internal use only No QM fees QM- Managed Required for peer reviewer & other QM roles Fee for Service Subscriber- Managed required for peer reviewer & other QM roles; Trainer must be certified Fee for certification

Peer Reviewers Applying the QM Rubric Training Peer Reviewer Certification (online) Current online instructor Independent Trainers Applying the QM Rubric Training Train-the Trainer Training Institution on Full Subscription Master Reviewers Certified QM Peer Reviewer Experience on 2+ course reviews Master Reviewer Training Course Review Manager Applying the QM Rubric Training Institution Representative Training Institution on Full Subscription Institution Representative Designated by institution Institution Representative Training Implementing QM – Roles and Criteria

Preparing for Course Reviews  Course is mature  Learning outcomes (course & unit) are specified  Review team has access to all elements students do  Use of QM rubric in course development  Faculty familiar with QM rubric

Helpful Links  Current Subscriber List  List for Publication.pdf List for Publication.pdf  Courses Recognized in 2008  htm htm  Training Calendar   Fee Schedule  e%20Schedule%20effective%208_1_2008.pdf e%20Schedule%20effective%208_1_2008.pdf  QM Staff 

Winter 2008 Thanks to YOU… Quality Matters! Thanks to YOU… Quality Matters!

2008 Post Review Interviews: The peer reviewers fulfilled their responsibilities on the review team.

2008 Post Review Interviews: The review followed the official QM course review process:

2008 Post Review Interviews: The feedback provided by the peer reviewers in the final report was constructive and useful: