1. Measuring Human Performance

2. Introduction n Kirkpatrick (1994) provides a very usable model for measurement across the four levels; Reaction, Learning, Behavior, and Results. These categories are discrete and can be measured. The goal of this presentation is to bring to light many of the topics, concerns, and issues that must be understood before carrying out the business of testing, measuring, or evaluating the success of training in the work force today.

3. What is a test? What is testing? n The instrument used to collect data n A process of collecting quantifiable information about the degree to which a competence or ability is present in the test taker. (Anderson, BC)

4. Reasons for Testing n Prerequisite tests n Entry test n Diagnostic test n Post test n Equivalency test

5. Norm Reference Vs Criterion Reference

6. Norm Referenced Testing n Test items separate test-takers one from another n Normal distribution curve

7. Criterion Referenced Testing n Test items based on specific objectives n Mastery Curve / Skewed from Normal Distribution

8. SKA n Skill n Knowledge n Attitude

9. Domains of Learning n Cognitive n Affective n Psychomotor

10. Bloom’s Taxonomy for Cognitive Levels n Knowledge n Comprehension n Application n Analysis n Synthesis n Evaluation

11. Krathwohl’s Taxonomy for Affective Levels n Receiving n Responding n Valuing n Organization n Characterization by a value or value complex

12. Simpson’s Taxonomy for Psychomotor Levels n Perception n Set n Guided Response n Mechanism n Complex Overt Response n Adaptation n Origination

13. Test Items Related to Bloom’s Taxonomy n Multiple Choice – Most flexible across the Taxonomy spectrum, especially first three levels n Advantages: – Guessing probability low – Diagnostic capabilities – East to grade – Statistical Analysis

14. Multiple Choice cont….. n Disadvantages – Difficult to write – Provides keys for recall – doesn’t do well for high level cognition evaluation

15. True and False n Could be used at all levels but…. n Advantages – easy to write – easy to score – can to item analysis

16. T/F cont…. n Disadvantages – 50/50 guess factor – often used when M/C seems too hard to write n Reliability is so poor…..Very little evaluation value. n So why do teachers often include T/F?

17. Matching n Best suited for Application level….not recommended for any by me. n Advantages – Easy to write – East to Grade – Statistical Analysis

18. Matching cont… n Disadvantage: – Requires the two lower learning level – Process of elimination diminishes probability – low reliability n Why would a teacher use Matching?

19. Fill in the Blank n Best suited for the lower levels n Advantage – Recall is essential, few clues n Disadvantage – Single word or phrase – grading beyond single word or phase is in trouble – enters the realm of subjective grading..poor reliability

20. Short Answer n Can get to the high order thinking n Advantages – Easy to write – produces original responses n Disadvantages – Basically same as fill in….reliability

21. Essay n The best for higher order n Advantage – high order – creative ability – writing ability

22. Essay cont… n Disadvantage – Tough to grade – forget stats n You’ll see this often in Master’s and Ph.d. classes

23. Validity n Does the test measure what it is suppose to measure. n How close to the bull’s eye did it hit.

24. Reliability n How consistent is the test n Is there a tight pattern of hits

26. Types of Validity n Concurrent Validity n Content Validity n Criterion Related Validity n Predictive Validity n Construct Validity

27. Types of Reliability n Test-Retest Reliability n Inner-Rater Reliability

28. What is the real score of a test? n An error factor must be considered n test score + error factor

29. Ten Evaluation Instruments for Technical Training n Interviews n Questionnaires n Group Discussion n Critical Incident n Work Diaries

30. Instruments cont... n Performance Records n Simulation Role-Play n Observation n Written Test n Performance Test

31. Designing Tests n Questions you must ask yourself – Who is the test designed for? – What do you want to know? – How many Questions will be required? – How will it be administered? – How will it be scored?

32. 3 Methods of Test Construction n Topic Based n Statistical Based n Objective Based

33. Topical Based Test n Selection done by chapter n Selection done by topic n Selection done by the importance of the topic

34. Limitations of Topic System n Procedure lacks precision n Doesn’t identify test takers n Not designed on learners level n Doesn’t specify competencies

35. Statistical Selection n Items statistically selected n Standardized n Norm Referenced

36. Limitations of Statistical n What is measured not specific n Lacks precision of CRT n Difficult to select items

37. Objectives Based Test n Based on defined competencies n Applies to criterion referenced tests and scores

38. Testing and Kirpatrick’s Four Levels n The more downward, from the performance of the company to the performance of the individuals, the more difficult to obtain. n The more downward...the more usable the information

39. Four Levels n REACTION n LEARNING n BEHAVIOR n RESULTS

40. Reaction n Checking individuals reaction often means, measuring “Customer Satisfaction” n Happy rating sheets n observations n other n How can you quantify the responses?

41. Learning n Measurable behavior changes in the three “SKA” Dimensions

42. Behavior n Behavior change due to training program. n Surveys n Interviews n Other

43. Results n Measurable by looking at changes in: n production n quality n Safety n Sales n other