1. Using Data to Turn the Tide on Course Reviews
September 30, 2014
Alise Hagan and Claire Arabie
Instructional Designers, Office of Distance Learning

2. Turning the Tide on Course Reviews
Office of Distance Learning
Turning the Tide on Course Reviews
"Just sit right back and you’ll hear our tale, a tale of course reviews That started with the rubric and overcoming the ‘not met’ blues. When the feedback started getting rough, instructors were growing weary So the design team found the standards causing the most misery."

3. Office of Distance Learning
Objectives
Learning Objective 1: Explain the internal peer- review course-certification process used.
Learning Objective 2: Identify QM standards continuously marked “not met” among internal review teams.
Learning Objective 3: Discuss methods for remedying the “not met blues” for both review teams and faculty members.

4. Peer-Review Course Certification Process at UL Lafayette

5. Peer-Review Process: By the Numbers
Office of Distance Learning
Peer-Review Process: By the Numbers
Over 70 courses have been through the subscriber-managed peer-review process
average 12 reviews / cycle, or 24 reviews / year
Over 85 faculty have served as either a department or university reviewer on at least one review
39 faculty in 17 departments have at least one certified course

6. Peer-Review Process: How are courses selected?
Office of Distance Learning
Peer-Review Process: How are courses selected? 1
Course is part of a fully online or fully hybrid program (Program coordinators identify the rotation ) 2
Course supports a fully online or fully hybrid program (such as Gen Ed courses) 3
Course designed through Course Design Practicum 4
At faculty member or department head request

7. Peer-Review Process: Who’s on the review team?
Office of Distance Learning
Peer-Review Process: Who’s on the review team?
The review team includes:
Instructor (course designer)
University Reviewers (two)
Department Reviewer (one minimum)
Department Head (or designee)
Review Chair (ODL staff)

8. Peer-Review Process: Reviewer Roles
Office of Distance Learning
Peer-Review Process: Reviewer Roles
QM Standards
Department Reviewer
University Reviewer
21 Essential (3-point) standards
21 (all)
12 Very Important (2-point) standards 6
8 Important (1-point) standards 4
TOTAL 31

9. Peer-Review Process: What’s the path to certification?
Office of Distance Learning
Peer-Review Process: What’s the path to certification?
Identify Courses (ODL)
Identify Review Team (ODL)
Complete Course Worksheet (Instructor)
Approve CW and Clone Course (ODL)
Conduct Reviews (Reviewers)
Discuss Review Summary Report (ODL, Instructor)
Make & Record Changes (Instructor)
Review Changes (Department Head)
Final Changes and Decisions (Instructor, ODL)
Course Earns ULearn Certification
{1 week}
{1 week}
{4 weeks}
{4 weeks}

10. The “Not Met” Blues: Standards Most Frequently Marked as “Not Met”

11. Not Met Blues: Curiosity led us to the Data
Office of Distance Learning
Not Met Blues: Curiosity led us to the Data
Which standard(s) were marked “not met” by the review team?
Which standards were most frequently marked “not met”?
What could we learn from the most frequently “not met” standards?
From our faculty meetings, we knew we were explaining some of the same “not met” standards over and over. But, was it a valid hunch?

12. “Not Met” Blues: A Closer Look (FA11-FA13)
Office of Distance Learning
“Not Met” Blues: A Closer Look (FA11-FA13)
Frequency of Not Met
by review team
# of standards (out of 41)
The standards
(Essential)
50%+ (at least 33/66) 3
1.7, 7.4, 8.2
40-49% (at least 27/66) 4
3.3, 7.1, 7.3, 8.4
30-39% (at least 20/66) 7
1.3, 1.5, 1.6, 1.8, 5.3, 5.4, 8.1
20-29% (at least 14/66)
1.1, 2.2, 3.5, 4.6, 6.2, 6.3, 6.5
TOTAL 21
Research / reviews used the QM Rubric
For at least 20% of the reviews:
Half (21) of all standards are not met
Eight (8) essential standards are not met
General Standards causing the most difficulty:
GS1: Course Overview and Introduction (6 of 8, or 75%)
GS7: Learner Support (3 of 4, or 75%)
GS8: Accessibility (3 of 4, or 75%)
Followed by GS6, GS5, GS3, GS2, GS4

13. Remedies for the “Not Met” Blues

14. Remedies for the Blues: Actions Taken for Reviewers
Office of Distance Learning
Remedies for the Blues: Actions Taken for Reviewers
“Reviewer Refresher” workshop (SP14)
Most missed (50%+)
Essentials most missed (20%+)
General standards

15. Closer Look at Standard 8.2 (Not Met 43/66; 65%)
Office of Distance Learning
Closer Look at Standard 8.2 (Not Met 43/66; 65%)
Standard: The course contains equivalent alternatives to auditory and visual content.
For this standard to be met at UL Lafayette:
Link to transcripts provided by textbook publisher (for publisher content)
Transcripts provided by instructor for instructor-made audio
Descriptions / captioning for video or images
A statement explaining how to seek accommodations or content in alternative formats.
Tip: Get in the habit of having a written script prior to recording a lecture. Then, once the lecture is posted, you can post the script as well.

16. Closer Look at Standard 7.4 (Not Met 41/66; 62%)
Office of Distance Learning
Closer Look at Standard 7.4 (Not Met 41/66; 62%)
Standard: Course instructions articulate or link to an explanation of how the institution’s student support services can help students succeed and how students can access the services.
For this standard to be met at UL Lafayette:
A description of the Office of Student Affairs and how to access it (provide link to the website:
A list of the types of services available and websites:
Advising
Registration
Financial Aid
Student / Campus Life
Tip: Create a Learner Support block in the LMS that includes these links for your institution.
Advising:
Registration:
Financial Aid:
Student / Campus Life:
Counseling / Testing:
Career Services:
Student Organizations:
Counseling / Testing
Career Services
Student Organizations

17. Closer Look at Standard 1.7 (Not Met 34/66; 51.5%)
Office of Distance Learning
Closer Look at Standard 1.7 (Not Met 34/66; 51.5%)
Standard: The self-introduction by the instructor is appropriate and is available online.
For this standard to be met at UL Lafayette, look for both:
The essentials: the instructor’s name, title, field of expertise, address, phone number and times when the instructor is typically online or may be reached by phone.
Something extra:
A welcome note (with picture) or recorded message
A bio or history of teaching
A link to a personal website / blog for additional personal information
Tip: Instructors should consider having the self-introduction reside in a more permanent place in the course.
Many UL faculty include their own introduction in the student introduction forum. In that case, it would technically be “met” but the review team doesn’t have access to the responses of the forum and thus would not see the instructor’s post.

18. Remedies for the Blues: Actions Taken for Reviewers
Office of Distance Learning
Remedies for the Blues: Actions Taken for Reviewers
“Reviewer Refresher” workshop (SP14)
Most missed (50%+)
Essentials most missed (20%+)
General standards

19. Office of Distance Learning
Essential Standards Not Met
Consider this…
1.1 (26%)
Are there course instructions that direct students what to do first? Are the course overview and schedule of activities easy to locate?
2.2 (24%)
Does the instructor include module / unit objectives in the syllabus, checklist, or topic / module block?
3.3 (42%)
Has the instructor stated the criteria for evaluation of papers and assignments, such as rubrics or a list of criteria with associated point values?
5.3 (33%)
Is there a “clear statement of instructor responsibilities” which states the timeframe for responding to student s / discussion posts, and when grades will be posted?

20. Office of Distance Learning
Essential Standards Not Met
Consider this…
6.2 (24%)
Does the instructor use technology to increase students’ comfort with course material and facilitate learner engagement?
6.3 (24%)
Does the learner move from one place to another in the course in a logical manner, and with ease?
7.1 (42%)
Does the instructor explain the links and resources provided in the Learner Support block?
8.1 (33%)
Does the instructor explain services provided by the Office of Disability Services and how to access these services?

21. Remedies for the Blues: Actions Taken for Reviewers
Office of Distance Learning
Remedies for the Blues: Actions Taken for Reviewers
“Reviewer Refresher” workshop (SP14)
Most missed (50%+)
Essentials most missed (20%+)
General standards

22. Office of Distance Learning
Closer Look at General Standard 1 (6 of 8 were “not met” at least 20% of the time)
GS1: The course introduction sets the tone for the course, lets students know what to expect, and provides guidance to ensure they get off to a good start.
As reviewers, look for:
How are students welcomed to the course? Is there a welcome video or note?
Do students know where to begin, and specifically what to do to begin the course?
Is the purpose and structure of the course explained? Look for: course schedule, delivery modality, modes of communication, learning activities, and how learning will be assessed.
Is “Day 1” information available easily?
Are the course calendar and syllabus easy to access and understand?
Are the course policies clearly stated?
Does the instructor provide a self-introduction?
Are students expected to introduce themselves? (note: check instructor worksheet)
Are the pre-requisite skills / knowledge clearly stated? (note: this is different from course pre-reqs)

23. Office of Distance Learning
Closer Look at General Standard 7 (3 of 4 were “not met” at least 20% of the time)
GS7: In the learner support standard, four different kinds of support services are addressed: technical, accessibility, academic services, and student services.
As reviewers, look for:
A statement in the course (typically the syllabus) that directs students to each of the links within the Learner Support block. A brief explanation for each link should be provided.
Note: While the Learner Support block does provide some links, not every service is accounted for. Further, for the standard to be met, the instructor must also provide information on how to access the service (it is not enough just to have the link in Moodle but, from the annotation, the instructor must also direct the students to the link).

24. Office of Distance Learning
Closer Look at General Standard 8 (3 of 4 were “not met” at least 20% of the time)
GS8: The accessibility standard incorporates the principles of Universal Design for Learning and is consistent with Web Content Accessibility Guidelines.
As reviewers, look for:
Statement about the Office of Disability Services and how to access these services.
A link (through Learner Support) for Accessibility policies.
Equivalents to auditory and visual content.
Fonts, colors, graphics that promote navigation and readability. (For example, blinking icons may induce seizures, or some colors are not visible to students with certain color-blindness.)
Use the “description” fields to include additional information about an assignment, page, link, image, etc. (Many faculty put “x” or “.” so that is what is read by the screen reader.)

25. Remedies for the Blues: Actions Taken for DL Faculty
Office of Distance Learning
Remedies for the Blues: Actions Taken for DL Faculty
Mini-Reviews
Accessibility Workshop
Learner Support “block” in Moodle

26. Remedies for the Blues: Early Results (SU14)
Office of Distance Learning
Remedies for the Blues: Early Results (SU14)
29% reduction in standards not met 40%+
86% reduction overall in “not met” standards (of standards addressed)
Frequency of Not Met
by review team
# of standards not met FA11-FA13 (out of 41)
# of standards not met FA11-SU14 (out of 41)
50%+ 3 2
40-49% 4
30-39% 7
20-29% 9
TOTAL 21

27. Your Remedies
we will invite audience members to share their feedback and recommendations for reviewer training and tips.

28. Ahoy! In this workshop, we:
Office of Distance Learning
Ahoy! In this workshop, we:
Explained the internal peer-review course- certification process used.
Identified QM standards continuously marked “not met” among internal review teams.
Discussed methods for remedying the “not met blues” for both review teams and faculty members.

29. Thank You! Alise Hagan: alise@louisiana.edu
Claire Arabie: