How to Evaluate Online Teaching & Improve Faculty Development Thomas J. Tobin, B. Jean Mandernach, & Ann H. Taylor OLC International Conference, Orlando - October 16, 2015

Our Live Webcast Tweet your questions and comments using hashtags #olc15 #OnlineQuality Our session evaluation: goo.gl/v6kiaP Image © 2008 Anthony Quintano. Used under Creative Commons BY license from Flickr.com

The goal Support Faculty Improve Teaching Enhance Student Learning

The challenge Evaluation of Teaching Faculty Development

1.Identify Desired Results How do successful online/blended educators look/act? What are the skills you want them to have? What competencies should they master? 2.Determine Acceptable Evidence of Learning How will you and your faculty know if they’ve mastered those skills and competencies? 3.Plan Learning Experiences How will you help prepare them for success? Start at the End: Backward Design

Integration of services and support 6 TrainingMentoring EVALUATION DEVELOPMENTCoaching

selfstudents peers Admini- strators Holistic decision-making trainingmentoringevaluation development coaching 7 asklook

Six Strategies for Evaluating Online Teaching 1.Know your context. 2.Study the foundations of evaluation & analytics. 3.Implement formative data & evaluation first. 4.Automate summative methods next. 5.Create a holistic campus plan for implementation. 6.Implement and cycle the analytics plan.

How do faculty know if they are effective in the online classroom?

Are students learning? Student Learning Do the students enjoy the experience? Student Satisfaction Is online learning equivalent to face-to- face learning? Accreditation Does the instructor foster the online learning experience? Quality Indicators of Effectiveness

Online Learning Experience Impact Factors Institutional Resources Program & Curriculum Course Content & Design Teaching Behaviors Institutional Context Student Characteristics Technology

Faculty Impact Teaching Behaviors Course Content & Design

Six Strategies for Evaluating Online Teaching 1.Know your context. 2.Study the foundations of evaluation & analytics. 3.Implement formative data & evaluation first. 4.Automate summative methods next. 5.Create a holistic campus plan for implementation. 6.Implement and cycle the analytics plan.

1.Encourage student-faculty contact 2.Develop reciprocity and cooperation among students 3.Use active learning techniques 4.Give prompt feedback 5.Emphasize time-on-task 6.Communicate high expectations 7.Respect diverse talents and ways of learning Principles of Good Practice: Technology Chickering & Gamson (1987) Chickering & Ehrman (1996)

1.Attend to unique challenges of distance learning 2.Be familiar with unique learning needs 3.Master course content, structure, organization 4.Respond to student inquiries 5.Provide detailed feedback 6.Communicate effectively 7.Promote a safe learning environment 8.Monitor student progress 9.Communicate course goals 10.Provide evidence of teaching presence Faculty Competencies for Online Teaching Pedagogical Technical Administrative Penn State Competencies for Online Teaching Success

Measurable Online Teaching Behaviors Do you log in at least every other day? Have you posted a biography of at least a paragraph, in addition to contact info? Do you post announcements at least weekly? Do you answer students in two days or less? Do you participate in discussion forums? Do you give detailed assignment feedback? (Piña and Bohn, 2014) Image ©2010 Steve Harris. Used under Creative Commons BY license from Flickr.com

Who has access to this information? How is the information used to improve faculty support and programming? Who is responsible for evaluation of online teaching? Who is responsible for faculty development? What standards, competencies, and expectations exist for “good” online teaching at your institution?

Six Strategies for Evaluating Online Teaching 1.Know your context. 2.Study the foundations of evaluation & analytics. 3.Implement formative data & evaluation first. 4.Automate summative methods next. 5.Create a holistic campus plan for implementation. 6.Implement and cycle the analytics plan.

Formative Evaluation: The SCARF Loop Image ©2014 Thomas J. Tobin. Used under Creative Commons BY-SA license.

improve the educational experiences you provide for your students, identify the professional education you need to further develop your capacity to teach well, prepare for your performance review with your supervisor, and assess your readiness to apply for promotion and tenure. Image ©2011 F Delventhal. Used under Creative Commons BY license from Flickr.com Self-evaluation

Formative: Finding the Right Tool Techniques for Students Surveys One-minute papers Muddiest point “Suggestion box” Anonymous tip line Techniques for Instructors Self reviews Peer reviews Tools Qualtrics SurveyMonkey Survey Gold Google Template Your LMS Administrative assistant or TA

Six Strategies for Evaluating Online Teaching 1.Know your context. 2.Study the foundations of evaluation & analytics. 3.Implement formative data & evaluation first. 4.Automate summative methods next. 5.Create a holistic campus plan for implementation. 6.Implement and cycle the analytics plan.

Student Ratings of Teaching Effectiveness SEEQ SALG SRTE Data Analytics LMSs and other technology- based teaching tools Administrative Evaluations Checklist for Online Interactive Learning (COIL) Checklist for Online Interactive Learning Quality Online Course Initiative (QOCI) Quality Online Course Initiative Quality Matters (QM) Rubric Quality Matters Online Instructor Evaluation System (OIES): Park University Online Instructor Evaluation System Peer Review of Online Teaching: Penn State Peer Review of Online Teaching Summative: Finding the Right Tool

What can you do to promote formative evaluation? How can you utilize summative evaluation to offer more effective and/or targeted faculty development initiatives? What other campus groups, organizations or departments can you work with to integrate evaluation and faculty support services? How can you use evaluation information (formative or summative) to enhance faculty support?

Six Strategies for Evaluating Online Teaching 1.Know your context. 2.Study the foundations of evaluation & analytics. 3.Implement formative data & evaluation first. 4.Automate summative methods next. 5.Create a holistic campus plan for implementation. 6.Implement and cycle the analytics plan.

Campus Policy Create campus policies and procedures for using online data for formative and summative teaching evaluation. Annual Review, Merit Pay Student ratings Self-ratings Teaching scholarship Administrator ratings Promotion and Tenure Administrator ratings Teaching portfolio Program Decisions Student ratings Exit and alumni ratings Employer ratings

Six Strategies for Evaluating Online Teaching 1.Know your context. 2.Study the foundations of evaluation & analytics. 3.Implement formative data & evaluation first. 4.Automate summative methods next. 5.Create a holistic campus plan for implementation. 6.Implement and cycle the analytics plan.

The Teaching-Learning Lifecycle Integrate teaching evaluations and analytics to continuously gauge and foster effective online instruction (minus the wires). Image ©2006 Glogger~commonswiki. Used under Creative Commons BY-SA license from Wikimedia.com

short sessions Faculty members want informal learning opportunities, flexible scheduling, short sessions, and one-on-one support. anytime, anywhere There isn’t a “best time.” Faculty members want anytime, anywhere professional development. assessment of your own faculty Rely on your assessment of your own faculty needs when designing the “right” program and resources for your institution.

What evaluations are currently done at your institution that can inform faculty development initiatives? What evaluations can you integrate to more effectively support faculty? How can you integrate evaluation and faculty development to create more holistic support for faculty at your institution?

Questions and Comments

Thank You!

Let’s Continue the Discussion Thomas J. Tobin B. Jean Mandernach Ann H. #olc15 #OnlineQuality

Our Live Webcast Tweet your questions and comments using hashtags #olc15 #OnlineQuality Our session evaluation: goo.gl/v6kiaP Image © 2008 Anthony Quintano. Used under Creative Commons BY license from Flickr.com