1. STUDENT EVALUATION OF FACULTY: Policies and Processes Kay Turpin Alison Joseph Western Carolina University SAIR 2010 New Orleans, LA

2. Please… turn off cells phones and other devices if you must leave the session early – leave quietly avoid side conversations SAIR 2010 New Orleans, LA This presentation is available at: http://www.wcu.edu/27729.asp

3. Location: Cullowhee, NC Member of UNC System Fall 2010 enrollment - 9407 1440 FTF 7503 UGs 1904 GRs Western Carolina University SAIR 2010 New Orleans, LA

4. History of Evaluations at WCU No university-wide policy on administration, instruments, courses covered, and use of results Scanned, paper forms administered during class time Long wait for results No flexibility in types of reports Security dismal – esp. comments IR office: Purdue University’s Cafeteria Evaluation System 7-12 departments select undergraduate courses inefficient, time-consuming process SAIR 2010 New Orleans, LA

5. Planning for University-Wide Evaluations 2002 through 2006 Faculty Senate estab. ‘teaching evaluation committee’ Provost mandated online evaluations (PACE initiative) Developed SAI instruments based on course types Developed guidelines for results usage Initial discussion on admin. evals and accessing results Researched available options for administering evals and selected Academic Management System’s CoursEval Faculty “own” the evaluation process! SAIR 2010 New Orleans, LA

6. Course Evaluation Pilot – Spring 2007 5 departments (Mkt/BLaw, Chem/Phys, CJ, PoliSci, Psych) 328 courses (5689 surveys) 45% response rate Problems: login & email issues inexperienced staff oversaw administration End Result: CoursEval online system worked fine BUT…. SAIR 2010 New Orleans, LA

7. Full Implementation – Fall 2007 late start no scheduling guidelines key people not involved minimal error checking lack of advertising and incentives inappropriate default forms login issues from Spring pilot not addressed questions added, doubling length 44% response rate Began Fall without established policies and processes in place for administering the survey and interpreting results These issues would directly affect acceptance of online evaluations by both faculty and students! SAIR 2010 New Orleans, LA

8. Establishing Policies & Processes Faculty Senate scheduling guidelines low-enrolled courses student incentives refined SAI instruments planned validity study Implementation Committee advertising/incentives faculty training summer parts-of-term instrument defaults student email policy login issues Dec. 2007 through Summer 2008 SAIR 2010 New Orleans, LA

9. Spring 2007: 45% response rate(Pilot) Fall 2007: 44% response rate Were these major or minor issues? Spring 2008: 33% response rate (Drop of 11%!) Was our effort worth it? Fall 2008:46% response rate Spring 2007: 45% response rate(Pilot) Fall 2007: 44% response rate Were these major or minor issues? Spring 2008: 33% response rate (Drop of 11%!) Was our effort worth it? Fall 2008:46% response rate Implementation Review SAIR 2010 New Orleans, LA

10. Lesson 1: Involve everyone with a stake in the evaluation procedures and results; everyone with experience in processing & administering evaluations Lesson 2: Consider all issues before implementing Lesson 3: Build a strong communication system between faculty and survey administrators Lesson 1: Involve everyone with a stake in the evaluation procedures and results; everyone with experience in processing & administering evaluations Lesson 2: Consider all issues before implementing Lesson 3: Build a strong communication system between faculty and survey administrators Lessons Learned SAIR 2010 New Orleans, LA

11. Overview of Current Policies all course sections, all terms open period set by course length university-wide incentives allowed low-enrolled courses identified crosslisted courses periodic response rates & student reminders faculty evaluation (not course content) results availability use in faculty AFE/TPR process IR office responsible for administration SAIR 2010 New Orleans, LA

12. IR Responsibilities Set-up & administer each semester’s evaluations “Help desk” for faculty and students Pursue new technologies & procedures to improve process, reporting, and response rates Communicate to Faculty Senate any issues arising affecting senate policies and survey administration EVERYTHING! SAIR 2010 New Orleans, LA

13. Evaluation Set-up (each term) – Back End Set evaluation schedule for all courses Request SAI instrument modifications SAIR 2010 New Orleans, LA

14. Default instrument examples: 1) instructional method of ‘OA=online activity’ the instrument “Online Form” is default 2) instructional method of ‘F=face-to-face’ & course type ‘L=lab’ the instrument “Lab Form” is default TERMYRCOURSESECTCRSTYPEINSTMETHXLISTCODEFACULTYPreferred Survey FormDEFAULTINSTR FALL2010ATTR22101LECF Scifers, James 1. Standard Lecture Form FALL2010ATTR22175LABF Scifers, James 2. Lab Form FALL2010ATTR32301LECF Scifers, James 1. Standard Lecture Form FALL2010ATTR32330LABF Scifers, James 2. Lab Form FALL2010EDM32259LECOA0ZBerry, Robert 3. Online Form FALL2010EMC44550LECOA Hubble, Michael 3. Online Form SAIR 2010 New Orleans, LA

15. Evaluation Set-up (each term) Set evaluation schedule for all courses Request SAI instrument modifications Data extracts from Banner (withdrawals & grades) Error checks in Access database Upload files to CoursEval website: faculty records students records course information (includes primary instructor only) secondary faculty (not included in course info.) enrollment SAIR 2010 New Orleans, LA

16. Evaluation Set-up (each term) – Front End Assign instrument to courses Establish open/close dates and times – automated Set-up group emails for students/faculty – automated Set release date for survey results - automated For the main evaluation period each term – total time for front end set-up = approx. 4-5 hours SAIR 2010 New Orleans, LA

17. Features of CoursEval Course Designator – helps us match instruments to courses Students and Faculty – Find all associated courses and surveys Consolidation of people Active Directory login Automated emails, open/close, results release Response rates – real time, easy to locate and email SAIR 2010 New Orleans, LA

18. Features We Would Like Better reporting – more flexibility Advance notice of outages and problems Ability to access individual responses Faculty member attached to course cannot be changed Single sign-on with other systems (Blackboard) Separate surveys for each instrument and time period add to workload (84 surveys for Summer 2010) SAIR 2010 New Orleans, LA

19. End Results Consistent open periods No lost class time Single administrator; increased efficiency All faculty/courses under same policies 100% class coverage Results available quickly, better reports, survey history Costs: $40,000 vs $84,000+ annually Lower response rate; more thoughtful comments High level of security Environmentally friendly SAIR 2010 New Orleans, LA

20. Future Plans and Ongoing Issues Improve response rates Implement Blackboard plug-in Improve Reports Available Improve process to change SAI instruments: Set-up Sharepoint site for department heads Use Banner and Blackboard to automate changes SAIR 2010 New Orleans, LA

21. WCU’s Course Evaluation website: http://www.wcu.edu/8356.asp “ Guidelines & Procedures for Administration & Oversight of Student Assessment of Instruction” Institutional Planning & Effectiveness: http://www.wcu.edu/12829.asp 828-227-7239 Kay Turpin turpin@email.wcu.edu 828-227-3041 Alison Joseph ajoseph@email.wcu.edu 828-227-3042 AMS website: http://www.academicmanagement.com/