1. C.E.L.T Developing a Comprehensive Faculty Evaluation System Jeff Schlicht, Ph.D. Associate Professor Department of Health Promotion and Exercise Sciences schlichtj@wcsu.edu

2. Measurement vs. Evaluation Measurement… is the process of systematically assigning numbers to the individual members of a set of objects or persons for the purpose of indicating differences among them in the degree to which they possess the characteristics being measured. Robert L. Ebel, “Measuring Educational Achievement”, 1965

3. Measurement vs. Evaluation Evaluation is the process of interpreting a measurement by comparing it to a value (or set of values) that represent an agreed upon standard. This process involves judgment.

4. A Comprehensive System = Measurement and Evaluation In order to develop a comprehensive faculty evaluation system, we must: 1) Develop technically good (valid & reliable) measurement instruments 2) Agree upon a university standard for desirable faculty performance

5. How Do We Currently Measure Faculty Teaching Performance? In the HPX department we use: 1) Student rating forms 2) Peer observation (in classroom) Are either of these forms of measurement valid and/or reliable? The answer is 1) we don’t know and 2) no

6. Student Ratings of Faculty Performance The development of a valid and reliable questionnaire, i.e. a student rating form, is a technical process that has a body of science behind it. Our department does not know who created the current rating form, and whether it has ever been tested for validity and reliability.

7. Student Ratings of Faculty Performance Dos and Don’ts DO include a balanced response scale equal number of positive and negative responses mirror-image opposites –Strongly agree agree disagree strongly disagree DO develop items (questions) important to faculty AND students DO conduct field tests of the items ( = collect data) to do reliability and validity analyses DO develop norms (a year’s worth of data) to allow fair and appropriate interpretation (i.e. comparison) DO consider using a professionally designed student rating form

8. Student Ratings of Faculty Performance Dos and Don’ts DON’T Have two items in one question (double- barreled response item) Simply hand back a summary sheet of student ratings to instructor – include personal consultation to foster instructional improvement Consider individual student responses in faculty evaluation

9. Peer Observation The short answer: DON’T do it for personnel decisions (can be useful for faculty enrichment) The addition of another faculty member or administrator into the classroom ALWAYS alters the classroom dynamic in such a way as to provide an abnormal view of what usually happens there THOSE DATA ARE NEVER VALID

10. Peer Observation If you must do it for faculty evaluation consider: multiple visits by multiple (4) peers –8 visits = adequate sampling –develop valid, reliable checklist train observer team in use –prepare students, instructor –schedule post-observation conference videography

11. 8 Steps to a Comprehensive Faculty Evaluation System Step 1: Define the faculty role model What should be evaluated? –Start at the department level Have faculty write down everything they do at WCSU to create a master list of responsibilities Determine faculty roles –What are the things we are accountable for? Assign the items from the master list of faculty responsibilities to the various faculty roles

12. 8 Steps to a Comprehensive Faculty Evaluation System Step 2: Determine faculty role model parameters on a University (system?) level How important should each role be? –Static vs. dynamic role model Static = fixed weights Dynamic = range of weights This is missing from our current system, and therefore we do not control subjectivity as well as we could

13. 8 Steps to a Comprehensive Faculty Evaluation System Step 2: Determine faculty role model parameters on a University (system?) level Static Role Model –Load Credit = 50% –Creative Activity = 25% –Productive Service = 15% –Professional Service = 10%

14. 8 Steps to a Comprehensive Faculty Evaluation System Step 2: Determine faculty role model parameters on a school, then University (system?), level Dynamic Role Model MinimumMaximum –Load Credit35%75% –Creative Activity20%50% –Productive Service5%45% –Professional Service0%40%

15. 8 Steps to a Comprehensive Faculty Evaluation System Step 2: Determine faculty role model parameters on University (system?) level Dynamic Role Model –The advantage of a dynamic role model is that it allows various levels of the University to play to their strengths Allow each department to set dynamic range values, then select extreme max. and min. as University standards

16. 8 Steps to a Comprehensive Faculty Evaluation System Step 3: Use observable or otherwise documentable achievements/products/performances to define all the roles in the faculty role model Teaching (student interactions that promote learning) –Content appropriateness/currency –Instructional design skills –Instructional delivery skills –Instructional assessment skills –Course management skills

17. 8 Steps to a Comprehensive Faculty Evaluation System Step 4: Define weights for each component of a single role – departments should be allowed to have their own weights, as long as they fit into the University structure. This embeds department values into the evaluation design: Teaching minmax –Content appropriateness/currency10%40% –Instructional design skills10%30% –Instructional delivery skills20%60% –Instructional assessment skills20%50% –Course management skills0%30%

18. 8 Steps to a Comprehensive Faculty Evaluation System Step 5: Identify where you will collect the data to evaluate the roles Use sources that have first hand experience with the performance in question – preferably multiple sources of input Teaching –Content appropriateness/currency syllabus review by peer Self-review –Instructional delivery skills Student evaluation Peer evaluation??? Self-review –Course management skills Chair or administrative assistant evaluation Self-review

19. 8 Steps to a Comprehensive Faculty Evaluation System Step 6: Define weighting for information provided by different sources Teachingminmax –Students15%50% –Self15%40% –Peer15%30% –Chair15%30%

20. 8 Steps to a Comprehensive Faculty Evaluation System Step 7: Determine how to gather information Questionnaire Interview Checklist

21. 8 Steps to a Comprehensive Faculty Evaluation System Step 8: Design data collection instruments

22. Evaluation for Promotion and Tenure Evaluation is the process of interpreting a measurement by comparing it to a value (or set of values) that represent an agreed upon standard. This process involves judgment. In a system like the one described today, evaluation is an on-going process involving multiple sources and evaluators. We are constantly comparing ourselves against the WCSU Faculty Role Model, so we have immediate feedback about our performance.

23. Evaluation for Promotion and Tenure Promotion and tenure decisions can be based on comparing our performance against a pre-determined, minimum required composite Faculty Role Model score. In other words, prior to becoming eligible for advancement, we would already know the outcome.

24. Comprehensive Faculty Evaluation Practical steps we can take to improve the system: Initiate a discussion about what it means to be a faculty member at WCSU in the 21 st century Discover new ways to measure faculty performance, tests those ways to make sure they are valid and reliable, and create a feedback loop that uses that information to enhance faculty performance Lobby the union and the University to more actively support faculty enrichment

25. Final Thoughts About Faculty Evaluation Why should we do it? Promotion and tenure is important, but… the true value of assessment lies in its ability to direct faculty enrichment. If evaluation is not tied to enhancing faculty performance, it will be perceived as punitive, a hurdle, rather than as a path toward self-improvement.

26. Acknowledgements These materials are summarized from the ideas of Raoul A. Arreola, University of Tennessee Health Science Center, presented at the Center for Educational Development and Assessment professional enrichment seminar in Orlando, FL. March 10-11, 2008, and available in the text: “Developing a Comprehensive Faculty Evaluation System,” 3 rd edition, Arreola RA, Anker Publishing Co., Bolton, MA 2007

27. C.E.L.T Developing a Comprehensive Faculty Evaluation System Jeff Schlicht, Ph.D. Associate Professor Department of Health Promotion and Exercise Sciences schlichtj@wcsu.edu