Instruction, Assessment, and Student Outcomes in Online Learning Environments Eric Riedel Rebecca Jobe Jim Lenio Kimberlee Bethany Bonura 2016 AALHE Annual Conference June 6-8 Milwaukee, WI

Walden’s Student Body 2 SIZE: Over 52,000 students; 85% graduate. AGE: Over 78% between ages 24 and 50. EMPLOYMENT: More than 89% working full time, part time, or self-employed. PRIOR EXPERIENCE: Over 66% had prior online experience. GENDER: Women comprise more than 75% of Walden’s student body. RACE/ETHNICITY: Ethnic minorities make up more than 45% of the student body. DOCTORATE AWARDS: Largest awarder of doctorates to African- American students from INTERNATIONAL: 7.8% international enrollment.

Walden Faculty Recruited from national and international pool of applicants. – 4% of applicants are accepted as faculty. All graduate programs taught by doctorally-prepared faculty. All faculty must complete new faculty orientation course (simulates actual classroom) before teaching. Ongoing evaluation through classroom metrics, peer review, student ratings, & review by program director. Faculty participate in ongoing teaching development through Center for Faculty Excellence which includes individual coaching, faculty study groups, webinars on specific topics taught by peers, and a face-to-face conference 2x/year. Over 3,300 Faculty 3,000 Contributing 300 Core 3

Purpose of Study Identify whether we are putting best faculty in first term courses. Identify if there are key metrics that program directors can focus on when assigning faculty to first term courses. 4

Sample and Methods Sample – Fall 2012 (n=3,868) and Spring 2013 (n=3,977) terms combined (Total n=7,845) – All Grad and UG courses included except RSCH, Doc Study, Dissertation, Field Experience, and Internship courses – Used Banner, Course Evals, and KPI data Methods 1.Identified faculty related variables. Demographics, workload, classroom behavior, perceived performance, perceived feedback 2.Examined relationships between faculty variables and within course retention. Grad and Undergrad separately 3.Identified variables found to be related to retention. 4.Measured whether best faculty are teaching Foundation courses. 5

Variables Examined Demographics Faculty age Years at Institution FT or PT Status (1=FT, 0=PT) Instructor Highest Degree (1=Bach, 3=PhD) Earned Degree from Online Institution (1=Yes, 0=No) Workload Total credits taught/term Total students taught/term Total sections taught/term Section Size Classroom Behavior Doc Sharing Login Avg Days/Week Gradebook Login Avg Days/Week Discussion Login Avg Days/Week Other Login Avg Days/Week Total Login Avg Days/Week Dropbox Posts/Student/Week Avg Discussion Posts/Student/Week Avg Other Posts/Student/Week Avg Total Posts/Student/Week Avg Perceived Performance Course Eval: Overall Instructor Satisfaction Course Eval: Returned Assignments w/in 10 Days Course Eval: Returned w/in 2 Days Perceived Feedback Course Eval: Mutual Respect Course Eval: Addressed Learner Needs Course Eval: Encouraged Independent Thinking Course Eval: Showed how to Improve Course Eval: Followed Grading Policies Course Eval: Challenged to Improve Within Course Retention Proportion of students who passed the course 6

Graduate Results – What variables are related to retention? Graduate Level 1st Term and Other Courses: Statistically Significant Correlation Coefficients with Course Retention Faculty CharacteristicsVariable 1st Term Courses (n=1,503) Other Courses (n=4,154) Demographics Faculty age Years at Institution-- FT or PT Status (1=FT, 0=PT) Instructor Highest Degree (1=Bach, 3=PhD) Earned Degree from Online Institution (1=Yes, 0=No) Workload Total credits taught/term Total students taught/term Total sections taught/term Section Size Classroom Behavior Doc Sharing Login Avg Days/Week-- Gradebook Login Avg Days/Week Discussion Login Avg Days/Week-- Other Login Avg Days/Week-- Total Login Avg Days/Week-- Dropbox Posts/Student/Week Avg-- Discussion Posts/Student/Week Avg-- Other Posts/Student/Week Avg Total Posts/Student/Week Avg-- Perceived Performance Course Eval: Overall Instructor Satisfaction Course Eval: Returned Assignments w/in 10 Days Course Eval: Returned w/in 2 Days Perceived Feedback Course Eval: Mutual Respect Course Eval: Addressed Learner Needs Course Eval: Encouraged Independent Thinking Course Eval: Showed how to Improve Course Eval: Followed Grading Policies Course Eval: Challenged to Improve There may be something to look at regarding faculty workload. Focused on Performance and Feedback 1 st term courses always have the strongest relationship. 7

Graduate Results - Continued Reexamining Perceived Performance and Perceived Feedback, we broke them down into more practical terms: – Timeliness of Feedback – Quality of Feedback 8

Graduate Results – Are best faculty teaching 1 st Term Courses? At a very general level, by comparing mean scores on the course eval items, we can say the best faculty are in first term courses. All differences are statistically significant; however effect sizes are small. 9

Undergraduate Results – What variables are related to retention? Undergraduate Level 1st Term and Other Courses: Statistically Significant Correlation Coefficients with Course Retention Faculty CharacteristicsVariable 1st Term Courses (n=281) Other Courses (n=1,602) Demographics Faculty age Years at Institution-- FT or PT Status (1=FT, 0=PT) Instructor Highest Degree (1=Bach, 3=PhD) Earned Degree from Online Institution (1=Yes, 0=No)-- Workload Total credits taught/term Total students taught/term Total sections taught/term Section Size Classroom Behavior Doc Sharing Login Avg Days/Week Gradebook Login Avg Days/Week Discussion Login Avg Days/Week Other Login Avg Days/Week-- Total Login Avg Days/Week-- Dropbox Posts/Student/Week Avg Discussion Posts/Student/Week Avg Other Posts/Student/Week Avg Total Posts/Student/Week Avg Perceived Performance Course Eval: Overall Instructor Satisfaction Course Eval: Returned Assignments w/in 10 Days Course Eval: Returned w/in 2 Days Perceived Feedback Course Eval: Mutual Respect Course Eval: Addressed Learner Needs Course Eval: Encouraged Independent Thinking Course Eval: Showed how to Improve Course Eval: Followed Grading Policies Course Eval: Challenged to Improve st term courses always have the strongest relationship. Possibly something to look into. Little bit of information here, but nothing very clear. Focused on Performance and Feedback 10

Undergraduate Results - Continued Focus on Timeliness and Quality of Feedback: – Much stronger association for 1 st term courses 11

Undergraduate Results – Are best faculty teaching 1 st Term Courses? At a very general level, by comparing mean scores on the course eval items, we can say the best faculty are in first term courses. All differences are statistically significant; however effect sizes are small. 112

Summary of Findings Statistically significant positive correlation between retention and faculty responsiveness item on course evaluations. (Primary for UG). Statistically significant correlation between retention and quality of feedback item on course evaluations. (Primary for Grad). Statistically significant mean differences between 1 st term courses and other courses on responsiveness and quality of feedback items. Graduate level correlation strength is lower than undergraduate and differences between 1 st term courses and other courses are smaller. 13

How has this information been used? Adds to research on importance of first term and efforts to provide students support. Joint research with UMUC. Pilot program training first-term faculty. CFE support for program directors in using faculty performance in course assignment. Refining measurement of quality feedback. Limiting exclusive focus on responsiveness. Next Steps: Faculty and Student focus groups 14

Discussion Do these results apply equally to first terms in online and face-to-face settings? What are the implications for faculty development in providing student feedback? How do we measure responsiveness and quality of feedback? Do criteria differ by students early versus later in their program? While retention was the focus here, what are other student outcomes linked to faculty behavior in the classroom? 15

Contacts Feel free to reach out with any questions to: Eric Riedel, Rebecca Jobe, Jim Lenio, Kimberlee Bethany Bonura, Thank You! 16