1. Using Evidence to Inform Continuous Improvement in Online Course Design
……………… Kay Shattuck, Director of Research, Quality Matters

2. Using evidence to influence practice
Kay Shattuck, Director of Research, Quality Matters
April 13, 2011
Kay Shattuck, Quality Matters, ELI Focus Session, April 2011

3. Kay Shattuck, Quality Matters, ELI Focus Session, April 2011
A continuous improvement model for assuring the quality of online and online components of blended courses through a faculty review process.
Currently 497 subscribers (HE institutions)
Go to for more exciting details.
Underlying principles of QM:
a faculty-driven, peer review process
a collaborative process among faculty peers
committed to continuous quality improvement
based in national standards of best practice, the research literature and instructional design principles
designed to promote student learning
A collaborative process:
Faculty course developer works with peer review team
Rubric standards serve as the basis for dialog & sharing of experience, expertise and know-how
Common goal is to improve online learning
QM is NOT a process of:
Not about an individual instructor
(it’s about the course design)
Not about faculty evaluation
(it’s about course quality)
Not a win/lose or pass/fail test
Kay Shattuck, Quality Matters, ELI Focus Session, April 2011

4. Kay Shattuck, Quality Matters, ELI Focus Session, April 2011
RESEARCH & PRACTICE
Informed by existing research & practice
Extended by experts = QM’s Interaction Summit
Expanded by QM focused research
Interaction Summit: (Anderson, Berge, Gunawardena, Roblyer, Swan)
Kay Shattuck, Quality Matters, ELI Focus Session, April 2011

5. IMPACT OF US/DEPT OF ED FIPSE GRANT
Post grant not-for-profit QM program established
503 current subscribers in 45 states; growing internationally
3,157 certified QM Reviewers
1,540 courses QM reviewed
8,405 individuals participated in QM training
Response to growing educational shareholders
Grade 6-12 QM Rubric released 2010; developed and field tested in partnership with Florida Virtual Schools
Publisher QM Rubric now in field test; developed in partnership with Wiley Publishing
Online Learners Bill of Rights under review by QM Academic Advising Committee; developed as result of ongoing research study
Proposals for international versions under consideration
Kay Shattuck, Quality Matters, ELI Focus Session, April 2011

6. Kay Shattuck, Quality Matters, ELI Focus Session, April 2011
LEARNER SATISFACTION
Student satisfaction higher in QM reviewed courses & non-reviewed courses than in courses at non-QM institutions. (Aman dissertation, Oregon State, 2009)
FIPSE/QM research grant: Course evaluation data showed student satisfaction increased in redesigned courses. (Prince George’s Community College, MD, 2005)
QM research grant: Currently conducting a mixed methods study student & faculty perceptions of QM reviewed courses. (University of the Rockies)
Kay Shattuck, Quality Matters, ELI Focus Session, April 2011

7. Kay Shattuck, Quality Matters, ELI Focus Session, April 2011
STUDENT LEARNING
FIPSE/QM research grant: Grades improved with improvements in learner-content interaction (result of review). (Community College of Southern Maryland, 2005)
QM research grant: Differences approaching significance on outcome measures. (Swan, Matthews, Bogle, Boles, & Day, University of Illinois/Springfield, 2010+)
QM research grant: QM Rubric implementation positive effect on student higher-order cognitive presence & discussion forum grades via higher teaching presence. (Hall, Delgado Community College, LA, 2010)
Kay Shattuck, Quality Matters, ELI Focus Session, April 2011

8. Kay Shattuck, Quality Matters, ELI Focus Session, April 2011
TEACHER LEARNING
Use of QM design standards led to “development of a quality product, as defined by faculty, course designers, administrators, and students, primarily through faculty professional development and exposure to instructional design principles” (p. 214). (Greenberg dissertation, Ohio State, 2010)
QM Research Grant: Currently utilizing TPACK framework to explain process by which new online teachers use the QM rubric and process when designing an online course. (University of Akron)
(Technology, Pedagogy, Content Knowledge)
Kay Shattuck, Quality Matters, ELI Focus Session, April 2011

9. ORGANIZATIONAL LEARNING
Discussion: There may be a carryover effect to non-reviewed courses when institution commits to the QM standards. (Aman dissertation, Oregon State, 2009)
Discussion: Faculty/design team respond different when QM presented as a rule rather than a guideline. (Greenberg dissertation, Ohio State, 2010)
QM data analysis: Extended positive impact on faculty developers & on members of review teams. (Preliminary analysis 2009; comprehensive summer 2011)
QM collected data from reviewers (examples)
I was too close to see what could be improved.
It made all of my online courses better.
It provides a view from a more student oriented perspective.
Many elements that might contribute to a student withdrawing can be eliminated.
This is my main motivation for reviewing courses.
It helped me with directions [for using a Wiki] and gave me the confidence to try this tool with my course.
I better understand how [why] course outcomes and objectives are so crucial to the success of an online course.
I actually learned to appreciate the numerous trainings I was required to complete before teaching online.
Kay Shattuck, Quality Matters, ELI Focus Session, April 2011

10. RETENTION/PERSISTENCE
Noted multiple “outside” variables associated with retention: Unable to find relationship between faculty peer review and retention. (Aman dissertation)
QM Research Grant: Currently investigating impact of QM certification on course attrition rates. (Cleveland State University)
Kay Shattuck, Quality Matters, ELI Focus Session, April 2011

11. STATUS OF RESEARCH ON LEARNER-LEARNER INTERACTION:
So many variables influence outcomes of current studies; results are too limited to draw cross-study conclusions.
Lack of consistency:
For example: Of five studies focused on graded group interaction, one study used adults in Malaysia and another used fifth-grade geometry students.
Broad conclusions about the issue of group interaction would be unethical. 
Lack of replication:
Patterns of interaction cannot be determined unless studies are replicable.
Replicable patterns are not evident in the literature on the value of learner-learner interaction except for some work done using the Community of Inquiry framework. 
(Status of interaction research presented by Expert Panel during
Quality Matters’ Interaction Summit, November 9, 2010)
A review of the recent research literature on learner-learner interaction reveals lack of consistency and replication. So many variables influence outcomes of the studies that results are too limited to draw cross-study conclusions.
Lack of consistency: For example: Of five studies focused on graded group interaction, one study used adults in Malaysia and another used fifth-grade geometry students. Broad conclusions about the issue of group interaction would be unethical. 
Lack of replication: Patterns of interaction cannot be determined unless studies are replicable. Replicable patterns are not evident in the literature on the value of learner-learner interaction except for some work done using the Community of Inquiry framework. 
Kay Shattuck, Quality Matters, ELI Focus Session, April 2011

12. Kay Shattuck, Quality Matters, ELI Focus Session, April 2011
CHALLENGES (PRESENTED BY EXPERT PANEL DURING QUALITY MATTERS’ INTERACTION SUMMIT, NOVEMBER 9, 2010)
Researchers need to move beyond studying perceptions and satisfaction.
Interaction with others is such a human thing to do that it has become almost a "motherhood statement" when discussing learning.
But, as educators we need to stop and ask what’s the evidence?
Does it hold true for all learners, for all disciplines, for all topics?
Of research on interaction:
Researchers need to move beyond studying student/teacher perceptions and satisfaction. We need to determine if more and different types of interaction equate to increased learning outcomes. The use of academic analytics is promising approach that might help researchers identify interaction patterns in which successful students engage but unsuccessful students to not. 
Interaction with others is such a human thing to do that it has become almost a "motherhood statement" when discussing learning. But, as educators we need to stop and ask what’s the evidence? Does it hold true for all learners, for all disciplines, for all topics?
A review of the recent research literature on learner-learner interaction reveals lack of consistency and replication. So many variables influence outcomes of the studies that results are too limited to draw cross-study conclusions.
Lack of consistency: For example: Of five studies focused on graded group interaction, one study used adults in Malaysia and another used fifth-grade geometry students. Broad conclusions about the issue of group interaction would be unethical. 
Lack of replication: Patterns of interaction cannot be determined unless studies are replicable. Replicable patterns are not evident in the literature on the value of learner-learner interaction except for some work done using the Community of Inquiry framework. 
Kay Shattuck, Quality Matters, ELI Focus Session, April 2011

13. CHALLENGES/LESSONS LEARNED
Too much reliance on simple surveys without control or follow up.
Neglected relationship of course design to other components of quality assurance.
Kay Shattuck, Quality Matters, ELI Focus Session, April 2011

14. shattuck@qualitymatters.org http://www.qmprogram.org/research
Kay Shattuck, Quality Matters, ELI Focus Session, April 2011