1. The Impact of QM: Before and After
New Mexico State University Alamogordo
Introduce ourselves.

2. Outcomes for the Session
Discuss NMSU-A’s distance education before and after Quality Matters™
Identify the impact Quality Matters™ had on the NMSU-A campus
Hear what impacts Quality Matters has had on your campus
Sherrell will lead.
Kim and I will discuss where we were before QM (spring 2013) and where we are now.
Claire will give you some of our data.
Karen will take a look at our reflections after the three years
Then we will ask for you to share where QM has taken you.

3. Leadership After QM (Now) Before QM (2012)
No oversight of distance education
No staff dedicated to distance education
No dedicated budget
Director of Online Quality Assurance
Online Quality Assurance Team
Dedicated Staff
Director
Instructional Consultant
Administrative Assistant
Transcriptionist
Dedicated Budget
Sherrell

4. Guidelines After QM (Now) Before QM (2012)
No real guidelines to indicate expectations
Distance Education Guidelines Manual
Forms
Procedures
Examples
QM Approved Courses
Sherrell

5. Trained Online Instructors
After QM (Now)
Before QM (2012)
No specific training required
No formal mentorship program for online faculty
No professional development specific to distance education
Required
Applying the Quality Matters Rubric (APPQMR)
Online Teaching Course
LMS (Canvas) Training
Quality Assurance Team
Monday Matters
Sherrell

6. Online Delivery Expectations
After QM (Now)
Before QM (2012)
No delivery expectations specific to online
Delivery Standards
Incorporated into Online Teaching Course
Incorporated into Observation Document
Kim
While QM addressed design, our campus needed to address online delivery. There we found that we had several challenges to resolve in order to create the best online learning experience for our students.

7. No Planned Services for Students
After QM (Now)
Before QM (2012)
Student Confusion on Contacts
No virtual meetings through Student Services
Limited tutoring services
No provided option for proctored testing
No “community”
Homepage directs to one person
Virtual meetings available
Online tutoring through academic support center
ProctorU
Canvas Community
Kim
First, we addressed the lack of connection of online students to the services that are available to face to face students. Go through slide.

8. No Check of Course Design
After QM (Now)
Before QM (2012)
Quantity priority over quality
Correspondence-type courses
Put on schedule at last minute
Quality over quantity
Courses with a high level of interaction
No courses offered that have not gone through a QM review or pre-review
Kim
Next, we had to face the practices that had become common in the scheduling of online courses. Go through slide.

9. “eLearning doesn’t just ‘happen’
“eLearning doesn’t just ‘happen’! It requires careful planning and implementation.”
-Anonymous
Kim
In short, we discovered that this quotation is true! Then we took steps to follow it by planning and implementing a carefully designed plan on our campus.

10. Early Surveys of Students
Was this course easier to navigate than a typical online course you have taken in the past?
Spring % - Yes
Fall 14 – 59% - Yes
How well did this course engage you?
Spring - 77% - More engaged than other online courses
How clear were the expectations?
Spring - 69% - Better than other online courses
Claire
This survey was given the first semester after a few courses had completed the QM review (Fall 2014). It was only given to those taking a course that had gone through the review. It was sent to 95 students. In the fall there were more courses that had gone through the review. It was sent to 220 students.
Response rate in the spring 2014 was 27% (during finals week) and fall 2014 it was 53% - both high for a student online survey.
How well did this course engage you? In the spring 43.7% more engaged and 43.7% about the same. Some of those may have only been in QM approved courses from the semester before % had never taken another online course.
How clear were the expectations? Spring 39.3% better, 45.9 about the same. (Again, may have been they had a QM approved the semester before and most faculty had gone through the training so there was a QM influence in a lot of courses).
After Fall 2014, all of our faculty had gone through training and there was an influence in all courses so there was not a reason to utilize this particular survey again.

11. Early Student Comments
“New and Improved!!! Love the easy navigation!!”
“I liked taking this course a lot!! It was very easy to navigate and the requirements were very clear.”
“I look forward to taking more courses designed by Quality Matters Standards.”
“Excellent course, easy to understand the require-ments of the students, communication with the instructor was better than past online experiences.”
Claire
Comments made on the same surveys.

12. Noel-Levitz PSOL Survey Satisfaction
Student assignments are clearly defined in the syllabus.
Navigation in the online courses is logical, consistent, and efficient.
The technologies required in my courses are readily available, provided or easily downloaded.
Course learning objectives, and instructions on how to meet them, are made clear in my courses.
Claire –
Priorities Survey for Online Learners – administered Spring about 10% response rate
Most satisfied with. Many faculty kept saying having the objectives throughout the course was not important to students. So…we were surprised that they even cared about last one. They do want those objectives to be clearly defined!
Can you see that all of these are directly related to the QM review process?

13. Noel-Levitz PSOL Survey Importance
Faculty provide timely feedback about student progress.
The quality of online instruction is excellent.
Courses have clear instructions about getting started and where to find things.
Student assignments are clearly defined in the syllabus.
The learning experience is engaging and supportive.
Claire
Now looking at the importance rather than the satisfaction, we can see that our students now have higher expectations for us!!! Many of these were also on the challenges list. Actually the importance on some of these was much higher than the national average. It shows higher expectations.
From 2014 to 2016 we saw a decrease in satisfaction in some of these same items, but since the importance went up, they are just expecting more.

14. Have you noticed an impact from our new distance education initiative?
Staff Survey – Spring 2016
Have you noticed an impact from our new distance education initiative?
75% - Yes
“The expectations that our students have of the online courses have increased. The students have now been trained to expect a certain quality and anything less than that is now a problem.”
Claire
80% response rate. Included admissions, academic advisors, financial aide staff, tutors, etc.

15. Staff Survey – Spring 2016
Based on working with students taking online courses, have you noticed a change in attitude about online courses at NMSU-A?
71.43% - Yes
“As the process was ongoing, the students would complain about the non-QM courses, because there was no order to them. The students like the structure of the courses.”
Claire
Another comment too large for the slide.
Students used to complain about not being able to find assignments in Canvas because every instructor did things differently. As the process was ongoing the students would complain about the non-QM courses, because there was no order to them. The students like the structure of the courses since they now understand where to find information and the layout of the courses are similar.

16. Faculty Survey – Spring 2016
Did the changes made in your course to meet the Quality Matters™ Standards make your courses more engaging?
75% - Yes
“Students are more actively engaged through the QM improvements.”
Claire
46% response rate.
Many of the 25% of no responses made comments that their courses were already engaging.

17. Faculty Survey – Spring 2016
Have you applied concepts learned through Quality Matters™ and other distance education training to your face-to-face courses?
18.75% - Yes, the training had a significant impact on my face-to-face classes.
25% - Yes, there are some concepts I have transferred.
37.5% - I do not teach face-to-face courses.
Claire
We found this interesting. We have actually seen an increase overall of improved pass rates in both online and face-to-face since implementation of QM. We hope our strong focus on QM based on the results of this questions has something to do with that increase. Notice that only 18.75% said no.

18. Pass Rates and Completion Rates
Slight increase overall in pass rates.
Only small differences in completion rates between online and face-to-face.
Larger difference in pass rates between online and face-to-face indicating there are more Fs in online courses.
Claire
Pass (A,B,C, D)
Completion rates (A, B, C, D and F)

19. Reflection after three years
Karen
After looking at this date and much more, we reflected on the last three years. Here are some questions we asked ourselves and our reflective responses.

20. Did it make a difference for students to have all courses go through a Quality Matters™ review?
Karen
Overall faculty members feel their courses are stronger and better for students than before Quality Matters. A larger percentage of faculty feel their courses are less confusing, more engaging, and provide greater opportunities for success for students. On course student evaluations, 34% of faculty said positive student comments related back to changes made in their courses based on Quality Matters. Overall, pass rates are up in both online and face-to-face courses. Based on student surveys, they feel the courses have easier navigation, include clearer expectations, are more engaging, and have an overall better designed than courses they have taken that had not gone through a Quality Matters™ review. In early reviews of the data, there was an increase in the number of “A’s” earned in courses that had gone through a Quality Matters™ review when compared to the number of “A’s” in previous semesters of that same course. Student services staff are hearing positive comments about online courses and fewer complaints. They feel more confident in encouraging students to take online courses. Of the seven strengths of NMSU-A on the Noel-Levitz survey five were specific to course design. Students are seeing the difference and are now expecting more based on the results of the Noel-Levitz survey. And one of the greatest statements in the data is that 91% of students would take another online course at NMSU-A.

21. Did it warrant the financial and human resources that were committed to it?
Karen
For the reasons provided in the previous question, the answer is yes. Students are seeing the difference in online courses and the majority of them are happy with the changes. Students who are not happy were probably satisfied with the “read and take the test” version of some of the classes before review. Faculty members are seeing the difference and the majority of them are happy with the changes. Those who have attended training sessions and have taken the best practices and applied them to their courses are seeing student success for students who take advantage of the opportunities provided. The reality is less than 40 percent of our sections are offered in online/hybrid format and over 50 percent of our enrollment is in these courses in the regular fall and Spring semesters. About 60% of the sections offered during the summer are in online format and about 75% of our enrollment is in those courses. Without even offering or advertising online programs, 20% of our students are working to earn a degree 100% online. How large will that number become now that we honestly have 100% online degrees?

22. Have the gaps in course retention rates declined?
Karen
There was a decrease for a short period of time, but unfortunately that gap has grown again. However, even though the gap is still there, overall the course completion rates have gone up over the past two years. This is definitely an area that needs to be considered and addressed. Working on the issue of retention is one of the goals of the Online Quality Assurance Team for the next three years. What is causing this trend? What can be done as a campus to provide the support and encouragement to online students to help them be successful? There is not a significant difference between completion rates for online and face-to-face courses, but there is a large gap between course pass rates and completion rates thus indicating there are a large number of “F’s” in online courses. Word of mouth from faculty indicates these students simply stop completing work and, therefore, earn an F. How can we keep those students engaged? These are concerns nation-wide, but will be a focus for the next three years with a goal to narrow that gap.

23. Are the services supporting online instruction appropriate?
Karen
Unfortunately, this question was not included on either the student or faculty surveys. However, on the Noel-Levitz survey, all questions involving student services have increased in satisfaction over the previous year. On the 2016 Noel Levitz survey two of the seven strengths pertain to student service areas: “Billing and payment procedures are convenient for me” and “Registration for online courses is convenient”.

24. Are the changes made in online design and delivery being reflected in face-to-face courses?
Karen
Although not an original goal of the distance education plan, it appears this is occurring. There are probably many reasons for it, but overall pass rates have increased since the implementation of Quality Matters. Since many faculty members do not have a background in education, is the pedagogy and design being required to teach online impacting the overall teaching procedures of some faculty members and thus having an impact on their face-to-face courses? According to the faculty survey conducted in the Spring, only 19% felt it had no impact on their face-to-face courses. A large percentage did not teach face-to-face, but over 50% stated they were utilizing concepts learned through the online instruction training that was impacting their face-to-face courses.

25. Is the program sustainable?
Karen
In viewing the data, faculty and students see a difference in course design and rigor. Since over 50% of the seat count is in online courses during the fall and spring and over 75% during the summer, it is obvious that distance education is an important part of the institution. It is also important to remember that there are more course offerings in face-to-face courses, but more than half take online classes. It was discovered that about 20% of the NMSU-A students noted they are working to obtain an online degree when at the time of the serve, there were no advertised online degree programs. As more degrees go online and the 2+2 Program through the main campus becomes more viable, this could be a way to improve enrollment. NMSU-A does have a credit hour online fee that is also used to offset the expenses of the distance education program.

26. Who would like to share differences you have seen before and after Quality Matters™?
Sherrell

27. Questions?
Sherrell Wheeler, Director of Online Quality Assurance,
Kim Lopez Gallagher, Professor,
Karen May, Part-time Instructor,
Claire Forsmann, Part-time Instructor,
Sherrell