Using Data to Turn the Tide on Course Reviews September 30, 2014 Alise Hagan and Claire Arabie Instructional Designers, Office of Distance Learning

Turning the Tide on Course Reviews Office of Distance Learning Turning the Tide on Course Reviews "Just sit right back and you’ll hear our tale, a tale of course reviews That started with the rubric and overcoming the ‘not met’ blues. When the feedback started getting rough, instructors were growing weary So the design team found the standards causing the most misery."

Office of Distance Learning Objectives Learning Objective 1: Explain the internal peer- review course-certification process used. Learning Objective 2: Identify QM standards continuously marked “not met” among internal review teams. Learning Objective 3: Discuss methods for remedying the “not met blues” for both review teams and faculty members.

Peer-Review Course Certification Process at UL Lafayette

Peer-Review Process: By the Numbers Office of Distance Learning Peer-Review Process: By the Numbers Over 70 courses have been through the subscriber-managed peer-review process average 12 reviews / cycle, or 24 reviews / year Over 85 faculty have served as either a department or university reviewer on at least one review 39 faculty in 17 departments have at least one certified course

Peer-Review Process: How are courses selected? Office of Distance Learning Peer-Review Process: How are courses selected? 1 Course is part of a fully online or fully hybrid program (Program coordinators identify the rotation ) 2 Course supports a fully online or fully hybrid program (such as Gen Ed courses) 3 Course designed through Course Design Practicum 4 At faculty member or department head request

Peer-Review Process: Who’s on the review team? Office of Distance Learning Peer-Review Process: Who’s on the review team? The review team includes: Instructor (course designer) University Reviewers (two) Department Reviewer (one minimum) Department Head (or designee) Review Chair (ODL staff)

Peer-Review Process: Reviewer Roles Office of Distance Learning Peer-Review Process: Reviewer Roles QM Standards Department Reviewer University Reviewer 21 Essential (3-point) standards 21 (all) 12 Very Important (2-point) standards 6 8 Important (1-point) standards 4 TOTAL 31

Peer-Review Process: What’s the path to certification? Office of Distance Learning Peer-Review Process: What’s the path to certification? Identify Courses (ODL) Identify Review Team (ODL) Complete Course Worksheet (Instructor) Approve CW and Clone Course (ODL) Conduct Reviews (Reviewers) Discuss Review Summary Report (ODL, Instructor) Make & Record Changes (Instructor) Review Changes (Department Head) Final Changes and Decisions (Instructor, ODL) Course Earns ULearn Certification {1 week} {1 week} {4 weeks} {4 weeks}

The “Not Met” Blues: Standards Most Frequently Marked as “Not Met”

Not Met Blues: Curiosity led us to the Data Office of Distance Learning Not Met Blues: Curiosity led us to the Data Which standard(s) were marked “not met” by the review team? Which standards were most frequently marked “not met”? What could we learn from the most frequently “not met” standards? From our faculty meetings, we knew we were explaining some of the same “not met” standards over and over. But, was it a valid hunch?

“Not Met” Blues: A Closer Look (FA11-FA13) Office of Distance Learning “Not Met” Blues: A Closer Look (FA11-FA13) Frequency of Not Met by review team # of standards (out of 41) The standards (Essential) 50%+ (at least 33/66) 3 1.7, 7.4, 8.2 40-49% (at least 27/66) 4 3.3, 7.1, 7.3, 8.4 30-39% (at least 20/66) 7 1.3, 1.5, 1.6, 1.8, 5.3, 5.4, 8.1 20-29% (at least 14/66) 1.1, 2.2, 3.5, 4.6, 6.2, 6.3, 6.5 TOTAL 21 Research / reviews used the 2011-2013 QM Rubric For at least 20% of the reviews: Half (21) of all standards are not met Eight (8) essential standards are not met General Standards causing the most difficulty: GS1: Course Overview and Introduction (6 of 8, or 75%) GS7: Learner Support (3 of 4, or 75%) GS8: Accessibility (3 of 4, or 75%) Followed by GS6, GS5, GS3, GS2, GS4

Remedies for the “Not Met” Blues

Remedies for the Blues: Actions Taken for Reviewers Office of Distance Learning Remedies for the Blues: Actions Taken for Reviewers “Reviewer Refresher” workshop (SP14) Most missed (50%+) Essentials most missed (20%+) General standards

Closer Look at Standard 8.2 (Not Met 43/66; 65%) Office of Distance Learning Closer Look at Standard 8.2 (Not Met 43/66; 65%) Standard: The course contains equivalent alternatives to auditory and visual content. For this standard to be met at UL Lafayette: Link to transcripts provided by textbook publisher (for publisher content) Transcripts provided by instructor for instructor-made audio Descriptions / captioning for video or images A statement explaining how to seek accommodations or content in alternative formats. Tip: Get in the habit of having a written script prior to recording a lecture. Then, once the lecture is posted, you can post the script as well.

Closer Look at Standard 7.4 (Not Met 41/66; 62%) Office of Distance Learning Closer Look at Standard 7.4 (Not Met 41/66; 62%) Standard: Course instructions articulate or link to an explanation of how the institution’s student support services can help students succeed and how students can access the services. For this standard to be met at UL Lafayette: A description of the Office of Student Affairs and how to access it (provide link to the website: http://studentaffairs.louisiana.edu/) A list of the types of services available and websites: Advising Registration Financial Aid Student / Campus Life Tip: Create a Learner Support block in the LMS that includes these links for your institution. Advising: http://studentsuccess.louisiana.edu/content/advising Registration: http://registrar.louisiana.edu/content/registration Financial Aid: http://financialaid.louisiana.edu/ Student / Campus Life: http://www.louisiana.edu/campus-life Counseling / Testing: http://counselingandtesting.louisiana.edu/ Career Services: http://career.louisiana.edu/ Student Organizations: http://getinvolved.louisiana.edu/student-organizations Counseling / Testing Career Services Student Organizations

Closer Look at Standard 1.7 (Not Met 34/66; 51.5%) Office of Distance Learning Closer Look at Standard 1.7 (Not Met 34/66; 51.5%) Standard: The self-introduction by the instructor is appropriate and is available online. For this standard to be met at UL Lafayette, look for both: The essentials: the instructor’s name, title, field of expertise, email address, phone number and times when the instructor is typically online or may be reached by phone. Something extra: A welcome note (with picture) or recorded message A bio or history of teaching A link to a personal website / blog for additional personal information Tip: Instructors should consider having the self-introduction reside in a more permanent place in the course. Many UL faculty include their own introduction in the student introduction forum. In that case, it would technically be “met” but the review team doesn’t have access to the responses of the forum and thus would not see the instructor’s post.

Remedies for the Blues: Actions Taken for Reviewers Office of Distance Learning Remedies for the Blues: Actions Taken for Reviewers “Reviewer Refresher” workshop (SP14) Most missed (50%+) Essentials most missed (20%+) General standards

Office of Distance Learning Essential Standards Not Met Consider this… 1.1 (26%) Are there course instructions that direct students what to do first? Are the course overview and schedule of activities easy to locate? 2.2 (24%) Does the instructor include module / unit objectives in the syllabus, checklist, or topic / module block? 3.3 (42%) Has the instructor stated the criteria for evaluation of papers and assignments, such as rubrics or a list of criteria with associated point values? 5.3 (33%) Is there a “clear statement of instructor responsibilities” which states the timeframe for responding to student emails / discussion posts, and when grades will be posted?

Office of Distance Learning Essential Standards Not Met Consider this… 6.2 (24%) Does the instructor use technology to increase students’ comfort with course material and facilitate learner engagement? 6.3 (24%) Does the learner move from one place to another in the course in a logical manner, and with ease? 7.1 (42%) Does the instructor explain the links and resources provided in the Learner Support block? 8.1 (33%) Does the instructor explain services provided by the Office of Disability Services and how to access these services?

Remedies for the Blues: Actions Taken for Reviewers Office of Distance Learning Remedies for the Blues: Actions Taken for Reviewers “Reviewer Refresher” workshop (SP14) Most missed (50%+) Essentials most missed (20%+) General standards

Office of Distance Learning Closer Look at General Standard 1 (6 of 8 were “not met” at least 20% of the time) GS1: The course introduction sets the tone for the course, lets students know what to expect, and provides guidance to ensure they get off to a good start. As reviewers, look for: How are students welcomed to the course? Is there a welcome video or note? Do students know where to begin, and specifically what to do to begin the course? Is the purpose and structure of the course explained? Look for: course schedule, delivery modality, modes of communication, learning activities, and how learning will be assessed. Is “Day 1” information available easily? Are the course calendar and syllabus easy to access and understand? Are the course policies clearly stated? Does the instructor provide a self-introduction? Are students expected to introduce themselves? (note: check instructor worksheet) Are the pre-requisite skills / knowledge clearly stated? (note: this is different from course pre-reqs)

Office of Distance Learning Closer Look at General Standard 7 (3 of 4 were “not met” at least 20% of the time) GS7: In the learner support standard, four different kinds of support services are addressed: technical, accessibility, academic services, and student services. As reviewers, look for: A statement in the course (typically the syllabus) that directs students to each of the links within the Learner Support block. A brief explanation for each link should be provided. Note: While the Learner Support block does provide some links, not every service is accounted for. Further, for the standard to be met, the instructor must also provide information on how to access the service (it is not enough just to have the link in Moodle but, from the annotation, the instructor must also direct the students to the link).

Office of Distance Learning Closer Look at General Standard 8 (3 of 4 were “not met” at least 20% of the time) GS8: The accessibility standard incorporates the principles of Universal Design for Learning and is consistent with Web Content Accessibility Guidelines. As reviewers, look for: Statement about the Office of Disability Services and how to access these services. A link (through Learner Support) for Accessibility policies. Equivalents to auditory and visual content. Fonts, colors, graphics that promote navigation and readability. (For example, blinking icons may induce seizures, or some colors are not visible to students with certain color-blindness.) Use the “description” fields to include additional information about an assignment, page, link, image, etc. (Many faculty put “x” or “.” so that is what is read by the screen reader.)

Remedies for the Blues: Actions Taken for DL Faculty Office of Distance Learning Remedies for the Blues: Actions Taken for DL Faculty Mini-Reviews Accessibility Workshop Learner Support “block” in Moodle

Remedies for the Blues: Early Results (SU14) Office of Distance Learning Remedies for the Blues: Early Results (SU14) 29% reduction in standards not met 40%+ 86% reduction overall in “not met” standards (of standards addressed) Frequency of Not Met by review team # of standards not met FA11-FA13 (out of 41) # of standards not met FA11-SU14 (out of 41) 50%+ 3 2 40-49% 4 30-39% 7 20-29% 9 TOTAL 21

Your Remedies we will invite audience members to share their feedback and recommendations for reviewer training and tips.

Ahoy! In this workshop, we: Office of Distance Learning Ahoy! In this workshop, we: Explained the internal peer-review course- certification process used. Identified QM standards continuously marked “not met” among internal review teams. Discussed methods for remedying the “not met blues” for both review teams and faculty members.

Thank You! Alise Hagan: alise@louisiana.edu Claire Arabie: claire@louisiana.edu http://distancelearning.louisiana.edu/