Faculty Evaluation Project Montgomery Community College Fall 2010

Why? Alignment Alignment Feedback for self improvement Feedback for self improvement Data for personnel decisions Data for personnel decisions

8 Steps Determine faculty role model Determine faculty role model Determine parameter values Determine parameter values Define roles Define roles Define roles component weights Define roles component weights

8 Steps Determine appropriate sources of information Determine appropriate sources of information Determine source and source impact weights Determine source and source impact weights Determine HOW information from each source is gathered Determine HOW information from each source is gathered Design or select appropriate forms Design or select appropriate forms

Good Measurement Comprehensive System Shared Values

Terminology Measurement Measurement Evaluation Evaluation “Positive” (Good) Evaluation “Positive” (Good) Evaluation “Negative” (Poor) Evaluation “Negative” (Poor) Evaluation

Terminology Measurement Measurement Evaluation Evaluation “Positive” (Good) Evaluation “Positive” (Good) Evaluation “Negative” (Poor) Evaluation “Negative” (Poor) Evaluation

Measurement Result is a number. Result is a number. Tools may achieve high levels of objectivity and reliability Tools may achieve high levels of objectivity and reliability Rating forms Rating forms Observational checklists Observational checklists Etc. Etc.

Terminology Measurement Measurement Evaluation Evaluation “Positive” (Good) Evaluation “Positive” (Good) Evaluation “Negative” (Poor) Evaluation “Negative” (Poor) Evaluation

Evaluation The result of an evaluation is a judgment as to whether the measurement or aggregate of measurements represents a desirable condition. The result of an evaluation is a judgment as to whether the measurement or aggregate of measurements represents a desirable condition.

Terminology Measurement Measurement Evaluation Evaluation “Positive” (Good) Evaluation “Positive” (Good) Evaluation “Negative” (Poor) Evaluation “Negative” (Poor) Evaluation

“Positive” “Negative” Process implies existence and use of a contextual system or structure of values associated with the characteristic(s) being measured Process implies existence and use of a contextual system or structure of values associated with the characteristic(s) being measured

Comprehensive System Systematic observation (measurement) of relevant faculty performance to determine the degree to which that performance is consonant with the values and needs of the educational institution. Systematic observation (measurement) of relevant faculty performance to determine the degree to which that performance is consonant with the values and needs of the educational institution.

Purposes 1. Provide feedback – for self-improvement 2. Provide data – for personnel decisions

Effective systems... Provide resources and professional enrichment opportunities designed to assist faculty to perform at a level consonant with the values and needs of the institution. Provide resources and professional enrichment opportunities designed to assist faculty to perform at a level consonant with the values and needs of the institution.

Effective systems... Are seen by faculty as a valuable resource in assisting them to solve problems or achieve goals both they and the administration consider important. Are seen by faculty as a valuable resource in assisting them to solve problems or achieve goals both they and the administration consider important.

Issues Culture – support versus militate against behavior change Culture – support versus militate against behavior change Continual support to sustain behavioral change in daily practice Continual support to sustain behavioral change in daily practice Target intact groups Target intact groups Focus on instructional delivery and instructional design skills Focus on instructional delivery and instructional design skills

Issues Enhancement of performance NOT remediation Enhancement of performance NOT remediation Reward both high quality performance and documented efforts to improve Reward both high quality performance and documented efforts to improve Should recognize colleague mentoring activities Should recognize colleague mentoring activities Combine test scoring, media center, instructional support into a faculty development center Combine test scoring, media center, instructional support into a faculty development center Faculty Development Director Faculty Development Director

College Teaching as a meta- profession Professional Skills & Knowledge Instructional design & delivery

Professional Skills & Knowledge Content expertise Content expertise Practice/Clinical Skills Practice/Clinical Skills Research techniques Research techniques Strategies for keeping current Strategies for keeping current

Traditional Assumption Base Professional Knowledge = Good Teacher Base Professional Knowledge and expertise is a necessary but insufficient condition for Good Teaching

Dilemma Faculty development programs should focus on the development of additional skills and knowledge required of the meta- profession of college teaching for which most faculty have had no prior formal education or training.

Possible? 1. Provide feedback – for self-improvement 2. Provide data – for personnel decisions

Yes, if... A single, comprehensive faculty evaluation system can do both if Detailed, diagnostic information is provided in confidence for self- improvement purposes Detailed, diagnostic information is provided in confidence for self- improvement purposes Only summary data is forwarded for decision-making purposes Only summary data is forwarded for decision-making purposes

Summary data Accurate picture of faculty member’s pattern of performance over time. Accurate picture of faculty member’s pattern of performance over time.

Objectivity? Objective components but process by nature is subjective Objective components but process by nature is subjective Appeal: Consistency of conclusions based on the same data. Appeal: Consistency of conclusions based on the same data.

Problem How do we achieve consistency of conclusions in a subjective evaluation system? How do we achieve consistency of conclusions in a subjective evaluation system?

Controlled Subjectivity Consistent application of a consensus- based set of values in the interpretation of data Consistent application of a consensus- based set of values in the interpretation of data

8 Steps Determine faculty role model Determine faculty role model Determine parameter values Determine parameter values Define roles Define roles Define roles component weights Define roles component weights

Faculty Role Model Conventional Conventional Teaching Teaching Research Research Service Service

MCC Faculty Role Model

8 Steps Determine faculty role model Determine faculty role model Determine parameter values Determine parameter values Define roles Define roles Define roles component weights Define roles component weights

Parameter Values Establish relative importance of each role to the institution. I.E. Determine how much value or weight may be placed on each role in the Faculty Role Model Establish relative importance of each role to the institution. I.E. Determine how much value or weight may be placed on each role in the Faculty Role Model

Static vs. Dynamic Teaching40% Research40% Service20% Static

Static vs. Dynamic Minimum Weight Maximum Weight 50%Teaching85% 0%Research35% 10% Faculty Service 25% 5% Community Service 15% Dynamic

Static vs. Dynamic Minimum Weight Maximum Weight D1D2D3D4 50%55%Teaching85%70% 0%20%Research25%35% 10%5% Faculty Service 25%15% 5%5% Community Service 25%15% Sample College Parameters

8 Steps Determine faculty role model Determine faculty role model Determine parameter values Determine parameter values Define roles Define roles Define roles component weights Define roles component weights

Define FRM Roles Observable or Observable or Documentable achievements Documentable achievements Products Products Performances Performances

Teaching Components Teaching Components Content Expertise Content Expertise Instructional Delivery Skills Instructional Delivery Skills Instructional Design Skills Instructional Design Skills Course Management Course Management

Content Expertise Formally recognized knowledge, skills, abilities in a chosen field by virtue of Formally recognized knowledge, skills, abilities in a chosen field by virtue of Advanced Training Advanced Training Education Education Experience Experience

Instructional Delivery Skills Human interactive skills which promote or facilitate learning including Human interactive skills which promote or facilitate learning including Organized presentations Organized presentations Motivates students Motivates students Generates enthusiasm Generates enthusiasm Communicates effectively Communicates effectively

Instructional Design Skills Technical skills in Technical skills in Designing Designing Sequencing Sequencing Presenting... Presenting... Experiences which induce learning; and.. Those skills in developing and using tools and procedures for assessing student learning (i.e. testing and grading)

Course Management Organizational and bureaucratic tasks involved in maintaining and operating a course, including Organizational and bureaucratic tasks involved in maintaining and operating a course, including Keeping grade records Keeping grade records Processing drop add forms, incomplete grade notifications, final grades, etc. Processing drop add forms, incomplete grade notifications, final grades, etc. Arranging for supplementary resources Arranging for supplementary resources Scheduling guest lecturers Scheduling guest lecturers

GOOD Instructional Delivery Skills POOR Instructional Delivery Skills GOOD Content Expertise POOR Content Expertise Type A Type C Type B Type D

8 Steps Determine faculty role model Determine faculty role model Determine parameter values Determine parameter values Define roles Define roles Define roles component weights Define roles component weights

Define FRM Component Weights How much value or weight should be placed on the several components of each role? How much value or weight should be placed on the several components of each role?

8 Steps Determine appropriate sources of information Determine appropriate sources of information Determine source and source impact weights Determine source and source impact weights Determine HOW information from each source is gathered Determine HOW information from each source is gathered Design or select appropriate forms Design or select appropriate forms

Sources of information Which source/sources should provide the information on which the evaluation of each role will be based? Which source/sources should provide the information on which the evaluation of each role will be based?

**Best Source Principle** Get information from those who have first- hand experience with the performance in question Get information from those who have first- hand experience with the performance in question

Sources Role Components StudentsPeers Dept. Head Instructional Delivery Skills YESNONO Instructional Design Skills YESYESNO Content Expertise NOYESYES Course Management NONOYES Source Identification Matrix for Teaching

Sources Expanded Role Components StudentsPeers Dept. Head SELF Instructional Delivery Skills YESNONOYES Instructional Design Skills YESYESNOYES Content Expertise NOYESYESYES Course Management NONOYESYES Student Achievement YESNONOYES Source Identification Matrix for Teaching (expanded)

8 Steps Determine appropriate sources of information Determine appropriate sources of information Determine source and source impact weights Determine source and source impact weights Determine HOW information from each source is gathered Determine HOW information from each source is gathered Design or select appropriate forms Design or select appropriate forms

Source & Impact Weights Determine the impact the information from the various sources will have on the overall evaluation of each role. Determine the impact the information from the various sources will have on the overall evaluation of each role.

8 Steps Determine appropriate sources of information Determine appropriate sources of information Determine source and source impact weights Determine source and source impact weights Determine HOW information from each source is gathered Determine HOW information from each source is gathered Design or select appropriate forms Design or select appropriate forms

How? Determine type of: Determine type of: Form Form Questionnaire Questionnaire Checklist Checklist Other method/procedure Other method/procedure

8 Steps Determine appropriate sources of information Determine appropriate sources of information Determine source and source impact weights Determine source and source impact weights Determine HOW information from each source is gathered Determine HOW information from each source is gathered Design or select appropriate forms Design or select appropriate forms

Appropriate Forms Design/Develop/Select and/or adapt from professionally developed forms, procedures & protocols Design/Develop/Select and/or adapt from professionally developed forms, procedures & protocols Strive for: Strive for: Objectivity Objectivity Reliability Reliability Validity Validity

Composite Role Rating Assume information from all sources and forms is reported on a common scale Assume information from all sources and forms is reported on a common scale Student Ratings Student Ratings Peer Ratings Peer Ratings Dept. Head Ratings Dept. Head Ratings Alumni Ratings Alumni Ratings Etc. Etc.

Composite Role Rating All use the same number point scale, although the response definitions may be different All use the same number point scale, although the response definitions may be different

Composite Role Rating AS= Agree Strongly =4 A=Agree=3 D=Disagree=2 DS= Strongly Disagree =1 VG= Very Good =4G=Good=3 P=Poor=2 VP= Very Poor =1

A common numerical scale provides the necessary tool for building a fairer and most consistent system by enabling us to express both individual composite role ratings and overall evaluation using the principles of controlled subjectivity...

Sources Role Components StudentsPeers Dept. Head Instructional Delivery Skills 4 Instructional Design Skills 33 Content Expertise 43 Course Management 2 4 (30%) = (10%) + 3(30%)= (20%) + 3(5%)= (5%)= 0.10 Composite Role Rating (Teaching)= 3.45

Individualizing Example: Professor Drake has selected the following roles with the relative “weights” shown: Teaching = 50% Teaching = 50% Scholarly Activity = 35% Scholarly Activity = 35% Faculty Service = 10% Faculty Service = 10% Community Service = 5% Community Service = 5% Total: 100% Total: 100%

Individualizing Example: Professor Drake has selected the following roles with the relative “weights” shown: RoleComposite Teaching3.54 Scholarly Activities 3.20 Faculty Service 3.60 Community Service 2.60

Individualizing Role Assigned Weight X Composite Role Rating = Weighted Composite Rating Teaching50%X3.45=1.73 Research35%X3.20=1.12 Faculty Service 10%X3.60=.36 Community Service 5%X2.90=.15 Overall Composite Rating = 3.34 Professor Drake

Individualizing Role Assigned Weight X Composite Role Rating = Weighted Composite Rating Teaching85%X3.53=3.00 Faculty Service 10%X2.00=.20 Community Service 5%X2.90=.15 Overall Composite Rating = 3.35 Professor Lamb

Drake & Lamb Role Drake’s Assigned Weight Lamb’s Assigned Weight Teaching50%85% Research35%----- Faculty Service 10%10% Community Service 5%5% OCR

Overall Composite Rating (OCR) Using the OCR in... Using the OCR in... Promotion decisions Promotion decisions Merit Pay Raise decisions Merit Pay Raise decisions

OCR Policy: Promotions based on the achievement of a specified Minimum Overall Composite Rating (OCR) for a specific number of consecutive years

8 Steps Determine faculty role model Determine faculty role model Determine parameter values Determine parameter values Define roles Define roles Define roles component weights Define roles component weights

8 Steps Determine appropriate sources of information Determine appropriate sources of information Determine source and source impact weights Determine source and source impact weights Determine HOW information from each source is gathered Determine HOW information from each source is gathered Design or select appropriate forms Design or select appropriate forms