th Annual PABUG Conference Bursting the Bubble [Sheets]: Moving Course Evaluations from Paper to Online Shippensburg University John VandeBrake Andrea Rosenberry

th Annual PABUG Conference General Announcements: Please turn off all cell phones/pagers If you must leave the session early, please do so as discreetly as possible Please avoid side conversations during the session Questions will be answered ….. Thank you for your cooperation

th Annual PABUG Conference To receive CPE credits for this session (if eligible), complete the CPE Attendance Form on the PABUG Annual Conference website group-annual-conference/cpe-credits/ or via the PABUG conference app. For additional questions please contact Kim Fremont CPE - Coordinator CPE Credits - We’ve gone electronic!

th Annual PABUG Conference About Founded 1871 Public Comprehensive Master’s 7,500 Students 325 Faculty & 450 Staff 13 Banner IT Staff

th Annual PABUG Conference Agenda Overview Opportunities Challenges Keys to Success Outcomes Explorance Blue Peak Future Enhancements Q & A

th Annual PABUG Conference Legacy system using Opscan paper bubble sheets Collective Bargaining Agreement (CBA) specifications Collaboration between provost and faculty key SIS integration – Explorance Blue Evaluation chosen due to Banner integration Data security: Explorance versus Survey Monkey or other non-SSO options Three year pilot launched starting with Athletics Overview

th Annual PABUG Conference Paper versus Online It goes without saying…..lots of manual work! Overview

th Annual PABUG Conference Paper process Pre Printing of the Opscan forms with course information Distribution of the paper forms to department offices then class Collection of paper forms and security concerns Functional issues with the Opscan machine Issues with bubble forms filled incorrectly Issues with pre printing on forms Computation of results in legacy software Printing and distribution of paper results Storage and retrieval of paper results Overview

th Annual PABUG Conference Eliminate paper processing saving time for both provost office, secretaries, and IT Security of documents improved Student convenience improved Student responses more accurate and thoughtful Computation, distribution and retrieval of results easier Opportunities

th Annual PABUG Conference SIS Integration: Banner to Blue Process changes: Explorance Blue vs paper Online response and evaluation rate variance Data integrity and automation for provost office CBA requirements & faculty adoption Challenges

th Annual PABUG Conference SIS Integration Oracle Views connect to Explorance Blue Authentication - SSO using CAS Challenges

th Annual PABUG Conference Banner Data Integration Data is term driven Non-distinct CRN – Numbers reused across terms – Team Taught Courses Not all courses should be evaluated Challenges

th Annual PABUG Conference Challenges

th Annual PABUG Conference Process Changes: Explorance Blue Pre-survey sent to enlist “volunteers” from Blue – Volunteer versus required to evaluate – Excel flag file data extension block used in Blue Volunteer list from survey combined with required Evaluation survey scheduled for launch One click thru for students to evaluation – Easy to use – Mobile enabled – Only one time completion allowed Challenges

th Annual PABUG Conference Process limitations Delivery mechanism changes with online Instrument moved to online – Challenge with collection of written responses and privacy Process changes – 24/7 access for two week window – Lack of “Captive audience” Reporting changes Mirrored reporting as much as possible Privacy of written responses Challenges

th Annual PABUG Conference Evaluation Rate Variances Evaluation rate slightly lower,.2 to.5 on Likert scale. – Announcement made to faculty “…a negative variance in this range may be observed when comparing online to paper. And that such variation should be considered insignificant and may not be used as the basis for negative evaluation.” Challenges

th Annual PABUG Conference Response Rate Variance Response rates have somewhat dropped – “Captive audience” effect neutralized – Reminder s sent to those not responding – Faculty encouraged to announce in class Helps increase participation More could be done to encourage online participation – Advertising? – Contests? – LMS Integration? Challenges

th Annual PABUG Conference Data Integrity and Automation Paper process had potential inaccuracies – Paper responses were sometimes incomplete and hard to validate – Response rates were not always accurate – Paper slugging of forms was not always accurate Maintaining data integrity online in Banner Moving offline data into Blue thru Banner integration Challenges

th Annual PABUG Conference CBA Requirements – Faculty Adoption Shippensburg works with unionized faculty under a collective bargaining agreement (CBA) – Mirrored process as much as possible – Mirrored paper documents as much as possible – Mirrored time lines as much as possible Faculty adoption encouraged thru collaboration with representatives and committee as well as through the pilots Challenges

th Annual PABUG Conference Communication between provost, faculty, athletics, and IT – Complete support of the Provost – Representation included leadership, athletics, faculty, union, and IT in forming a process plan – Regular committee meetings to receive input – Implementation of changes as technology permits – Communication to the larger campus community vital Keys to Success

th Annual PABUG Conference Three year pilot with three phases – Phase 1: Focus on athletic coach evaluations and early adoption as well as training Athletic evaluations established success Training meetings held for pilot faculty – Phase 2: Process focused: adjustments and improvements Meetings held to review process and gather input Online reporting improved to match paper reporting Reminders tweaked Keys to Success

th Annual PABUG Conference Three year pilot comprised of three phases – Phase 3: Adoption: Improve participation and tweaks Encourage participation Added faculty monitoring of response rates during evaluation cycle Supervisory report viewing enhanced Keys to Success

th Annual PABUG Conference Budget considerations help drive adoption – Good pilots and minor technical glitches made adoption more likely – Show how budget savings could benefit faculty Software ease of use: Explorance Blue has served us well Responsive leadership will help pave the way Keys to Success

th Annual PABUG Conference Slight decrease in quantitative responses Increase in qualitative data – Written responses more detailed and legible – Quality of responses improved. Longer and more thoughtful Use of own time to complete allows for more thought Reduces the use of paper and processing Saves class time for faculty Insures data integrity Quick reporting turn around Distance Ed and offsite course evaluations easier Outcomes

th Annual PABUG Conference Evaluation Instrument: sent with specific class details to each student Explorance Blue Sample

th Annual PABUG Conference Three reminder s sent to those not completed Explorance Blue Sample Dear Susan Hickenbaugh, REMINDER: We have not yet received your feedback for these course(s): -Charles Stanley (CSC Database Mgt Systems), -Joe Leevy (CSC Operating Systems), - Joe Leevy (CSC Automation Theory) We would really appreciate it if you can take a few minutes now to complete the student course evaluations. We are using online survey software to make it easier for students to give us their feedback. Click here now to provide us with your feedback. Remember: You will need to make sure you click the "Submit" button to complete your evaluation. If you experience any technical difficulties or have any questions please contact T Pearson in the Provost office at Click here now to provide us with your feedback.

th Annual PABUG Conference Explorance Blue Sample

th Annual PABUG Conference Explorance Blue Sample

th Annual PABUG Conference Subject View: Response rate review during evaluation cycle Provides the ability for faculty members to view the cumulative response rate as the evaluations is being administered. Explorance Blue Sample

th Annual PABUG Conference Mobile enabled for more input options – Potential for classroom administration – Easy access Explorance Blue Sample

th Annual PABUG Conference Further automation in the provost office – Volunteer and required selection process in SIS LMS Integration Participation promotion CBA requirements reassessment Parts of term and interactive start/stop times Long term storage of reporting Survey add-on tool for Title IX, Snyder File V data collection, and other surveys Future Enhancements

th Annual PABUG Conference Open to the Floor Questions Comments

th Annual PABUG Conference Please take time to complete our Conference and Session evaluations. YOUR INPUT MATTERS!!!