Measuring Human Performance

Introduction n Kirkpatrick (1994) provides a very usable model for measurement across the four levels; Reaction, Learning, Behavior, and Results. These categories are discrete and can be measured. The goal of this presentation is to bring to light many of the topics, concerns, and issues that must be understood before carrying out the business of testing, measuring, or evaluating the success of training in the work force today.

What is a test? What is testing? n The instrument used to collect data n A process of collecting quantifiable information about the degree to which a competence or ability is present in the test taker. (Anderson, BC)

Reasons for Testing n Prerequisite tests n Entry test n Diagnostic test n Post test n Equivalency test

Norm Reference Vs Criterion Reference

Norm Referenced Testing n Test items separate test-takers one from another n Normal distribution curve

Criterion Referenced Testing n Test items based on specific objectives n Mastery Curve / Skewed from Normal Distribution

SKA n Skill n Knowledge n Attitude

Domains of Learning n Cognitive n Affective n Psychomotor

Bloom’s Taxonomy for Cognitive Levels n Knowledge n Comprehension n Application n Analysis n Synthesis n Evaluation

Krathwohl’s Taxonomy for Affective Levels n Receiving n Responding n Valuing n Organization n Characterization by a value or value complex

Simpson’s Taxonomy for Psychomotor Levels n Perception n Set n Guided Response n Mechanism n Complex Overt Response n Adaptation n Origination

Test Items Related to Bloom’s Taxonomy n Multiple Choice – Most flexible across the Taxonomy spectrum, especially first three levels n Advantages: – Guessing probability low – Diagnostic capabilities – East to grade – Statistical Analysis

Multiple Choice cont….. n Disadvantages – Difficult to write – Provides keys for recall – doesn’t do well for high level cognition evaluation

True and False n Could be used at all levels but…. n Advantages – easy to write – easy to score – can to item analysis

T/F cont…. n Disadvantages – 50/50 guess factor – often used when M/C seems too hard to write n Reliability is so poor…..Very little evaluation value. n So why do teachers often include T/F?

Matching n Best suited for Application level….not recommended for any by me. n Advantages – Easy to write – East to Grade – Statistical Analysis

Matching cont… n Disadvantage: – Requires the two lower learning level – Process of elimination diminishes probability – low reliability n Why would a teacher use Matching?

Fill in the Blank n Best suited for the lower levels n Advantage – Recall is essential, few clues n Disadvantage – Single word or phrase – grading beyond single word or phase is in trouble – enters the realm of subjective grading..poor reliability

Short Answer n Can get to the high order thinking n Advantages – Easy to write – produces original responses n Disadvantages – Basically same as fill in….reliability

Essay n The best for higher order n Advantage – high order – creative ability – writing ability

Essay cont… n Disadvantage – Tough to grade – forget stats n You’ll see this often in Master’s and Ph.d. classes

Validity n Does the test measure what it is suppose to measure. n How close to the bull’s eye did it hit.

Reliability n How consistent is the test n Is there a tight pattern of hits

Types of Validity n Concurrent Validity n Content Validity n Criterion Related Validity n Predictive Validity n Construct Validity

Types of Reliability n Test-Retest Reliability n Inner-Rater Reliability

What is the real score of a test? n An error factor must be considered n test score + error factor

Ten Evaluation Instruments for Technical Training n Interviews n Questionnaires n Group Discussion n Critical Incident n Work Diaries

Instruments cont... n Performance Records n Simulation Role-Play n Observation n Written Test n Performance Test

Designing Tests n Questions you must ask yourself – Who is the test designed for? – What do you want to know? – How many Questions will be required? – How will it be administered? – How will it be scored?

3 Methods of Test Construction n Topic Based n Statistical Based n Objective Based

Topical Based Test n Selection done by chapter n Selection done by topic n Selection done by the importance of the topic

Limitations of Topic System n Procedure lacks precision n Doesn’t identify test takers n Not designed on learners level n Doesn’t specify competencies

Statistical Selection n Items statistically selected n Standardized n Norm Referenced

Limitations of Statistical n What is measured not specific n Lacks precision of CRT n Difficult to select items

Objectives Based Test n Based on defined competencies n Applies to criterion referenced tests and scores

Testing and Kirpatrick’s Four Levels n The more downward, from the performance of the company to the performance of the individuals, the more difficult to obtain. n The more downward...the more usable the information

Four Levels n REACTION n LEARNING n BEHAVIOR n RESULTS

Reaction n Checking individuals reaction often means, measuring “Customer Satisfaction” n Happy rating sheets n observations n other n How can you quantify the responses?

Learning n Measurable behavior changes in the three “SKA” Dimensions

Behavior n Behavior change due to training program. n Surveys n Interviews n Other

Results n Measurable by looking at changes in: n production n quality n Safety n Sales n other