STRATEGIES TO EVALUATE COMPETENCE Christina B. DeBiase, EdD WVU School of Dentistry

STEPS IN EVALUATION Define the objectives of the learning activity – a test should evaluate the degree of student learning Develop a test blueprint

TEST BLUEPRINTS How much emphasis should be given to each objective? What type of question would best assess the students’ attainment of objectives? How difficult should the questions be to adequately discriminate the stronger students from the weaker students? What considerations should be given to test format and administration?

Basic Forms of Evaluation 1.Objective – data gathered represents some predetermined standard 2.Subjective – no prior standard of performance has been specified

TESTS CBL / PBL SCENARIOS WRITTEN & ORAL PRESENTATIONS SIMULATIONS PRACTICUMS PORTFOLIOS OSCE ETC. METHODS FOR EVALUATING STUDENTS

OPTIONS FOR TEST ITEMS Stand-alone multiple-choice items Case-based multiple-choice items True/False Fill-in the blank/ Short-answer/Completion Essay Listing Oral Rating sheets

COMMON TEST-WRITING ERRORS

MULTIPLE-CHOICE COMPOSITION STEM: SET UP THE PROBLEM OR SET OF CIRCUMSTANCES; SHOULD HAVE AT LEAST ONE VERB; CAN BE WRITTEN IN QUESTION FORM OR AS A PARTIAL STATEMENT DISTRACTORS: ONE CORRECT OR BEST RESPONSE (tests higher level learning) AND 3-4 ALTERNATIVES; ALL OF THE ABOVE OR NONE... ARE NOT SPECIFIC OPTIONS OR SOLUTIONS AND SHOULD BE AVOIDED

KEEP IT SIMPLE! DO NOT PUT EXTRA, UNNECESSARY INFORMATION IN THE STEM: Mary, a 16-year old female with cerebral palsy, presents with a bruxing habit. Bruxism is the rhythmic grinding of teeth that can cause overdevelopment of a muscle of mastication. Which of the following muscles may be overdeveloped in a patient with bruxism?

POSITIVE/NEGATIVE STEMS POSITIVE - Usually more effective because the item tests what examinees know to be correct. Which of the following antibiotics is recommended for SBE coverage? NEGATIVE - Sometimes appropriate to distinguish the incorrect response among several correct alternatives. All of the following congenital cardiac defects require antibiotic premedication prior to dental procedures except one. The EXCEPTION is ___?

RULES TO FOLLOW WHEN CREATING ALTERNATIVES INCLUDE REPITITIOUS WORDS IN THE STEM TO AVOID REPEATING WORDS IN EVERY ALTERNATIVE. ALTERNATIVES SHOULD BE SIMILAR IN CONSTRUCTION AND OF APPROXIMATELY EQUAL LENGTH DISTRACTORS SHOULD BE PLAUSIBLE & MAY REPRESENT COMMONLY HELD MISCONCEPTIONS ABOUT A SUBJECT AVOID RESPONSE PATTERNS

The correct answer is longer or more specific than the distractors: All of the following determine the latent period’s degree of biologic effect except one. The EXCEPTION is ___? a. The exposure time b. The type of radiation c. The type of tissue irradiated d. The ability to cause substances to fluoresce * HINT- Put “The” in the stem!

ADDITIONAL PRACTICES TO TO AVOID WHEN DESIGNING ALTERNATIVES FOR MULTIPLE-CHOICE QUESTIONS

The alternatives are not mutually exclusive (overlapping alternatives); if numbers, always list in ascending or descending order & if ranges, intervals should be equivalent: W hich of the following I.Q. ranges is descriptive of a person with moderate mental retardation? a b c d. <20

Distractors are not plausible: W hich of the following primary molars is a smaller replica of the permanent first molar: a. First b. Second c. Third

Grammatical clues in the stem suggest the correct answer: Which of the following drugs is an antihypertensive agent? a. Procardia b. Prednisone c. Cyclosporine and Imuran d. Amoxicillin and Erythromycin

Information in the stem is also found in the correct answer: Which of the following forms of diabetes requires insulin for effective management? a. Insulin - dependent b. Non-insulin dependent c. Hyperglycemia d. Hypoglycemia

CAUSE AND EFFECT ITEMS (Statement)Prior to cardiac surgery, a patient should receive a dental cleaning, (Reason) BECAUSE in a plaque-infected oral cavity the likelihood of bacteremia is greatly increased. a. Both the statement and reason are correct and related. b. Both the statement and the reason are correct, but NOT related. c.The statement is correct, but the reason is NOT. d.The statement is NOT correct, but the reason is correct. e. NEITHER the statement NOR the reason is correct.

PAIRED TRUE & FALSE ITEMS I. Medical record entries should be completed in non-erasable ink. II. If a mistake is made when making a notation, the incorrect information should be covered with correction fluid. A. Both statements are true. B. Both statements are false. C. The first statement is true, the second is false. D. The first statement is false, the second is true.

ADVANTAGES OF MULTIPLE- CHOICE ITEMS Flexible Measure student’s knowledge in a subject area without the influence of writing skills, spelling, or grammar Scored quickly Content of course can be adequately sampled Comparable results can provide diagnostic information about class The best answer does not have to be the only correct response, but should be defensible Item- analysis is very feasible

DISADVANTAGES Questions are difficult to develop with sufficient plausible alternatives Cannot measure higher level learning unless case-based; stand-alone items usually measure factual recall only Guessing the correct answer can make the test ineffective

GUIDELINES FOR TRUE & FALSE STATEMENTS AVOID: Negatively worded statements Testing trivial detail Specific determiners that act as clues – usually, never, always, all

GUIDELINES FOR MATCHING ITEMS Specify whether alternatives can be used once or more than once It is recommended to provide more alternatives than premises, but the number of alternatives should not greatly exceed the premises List premises or phrases on the left and alternatives on the right

GUIDELINES FOR COMPLETION QUESTIONS Include sufficient information to tell student what type of answer is expected Omit as few words as possible Use a uniform blank space Avoid grammatical clues Place the blank toward the end of the sentence

GUIDELINES FOR ESSAY QUESTIONS Cover the objectives Prepare a structured response with elements students should include for credit Give clear directions so that the student knows what is expected Ensure the test is short enough to allow time to complete the essay(s) Avoid testing trivial details Avoid giving students the option to answer 5 of 7 essays, for example Grade by reading all students’ responses to one essay before going on to the next question