Good MC Questions for Training and Evaluation Irene Becker, LCSW West Coast Child Welfare Trainer Conference 2017 Boise, Idaho msibecker@gmail.com

Learning Objectives Identify at least three purposes for creating multiple choice (MC) questions Recognize at least 6 components of developing MC questions Analyze at least 6 MC and discuss the strengths and weaknesses of each Develop one MC question and receive feedback on its strengths and weaknesses

Welcome & Introductions – Match.com Training for Trainers Skill Development #4: Evaluation and Transfer of Learning September 19, 2014 Welcome & Introductions – Match.com Find a person whom you have a common interest in one of the following categories Find your match and take a few minutes to discuss your interest You and your match will be working on an activity together later in the training Sports Food & Beverage Travel Arts & Music Literature/Reading Games IRENE 2 Irene Becker & James Coloma

Purpose in Creating the MC Questions Low Stakes – for course improvement without any “consequences” to trainer and participant High Stakes – involve data collection and some course of action for improvement. This can include: Improvement in the curriculum May be used as a feedback tool for trainers using a standardized curriculum May find unexpected trends that lead to organizational improvement May lead to consequences of employment or certification The higher the stakes – the more rigorous the process needs to be in creating, analyzing, vetting and revising the questions

Questions for Team Based Learning Focus on accuracy but also on critical thinking – the purpose is to discuss and come to a consensus answer via a group process Sometimes consensus can’t be achieved, sometimes the group arrives at an incorrect answer – this allows for deepening discussion about the issue the question addresses This process acknowledges where the participants are in their experience and learning processes with a goal of increasing accuracy through a group process For more indepth information on creating questions via Bloom’s Taxonomy go to our wiki website

Example of Higher Stakes (but not the highest!) Use of pre/post tests for measuring knowledge Purpose is to gather data to look at trends, consistency across the state What data are we looking for? (This is where demographic data is important – data gathering as an aggregate vs individuals). Each has implications on what you analyze Questions that fall outside the range of 20-80% correct are looked at first Deeper analysis of the questions by demographic groups ( IV-E vs non-IV-E, ESL, race, gender etc) provide more information about the multiple choice question. (Is the question a “fair” question?) Determine revision process and test again

What is the MOST LIKELY reason for this? Adrianna, age 3, and her 6 month old twin brothers, Aaron and Eddie, are in an emergency foster home because of the murder of their mother by a drug dealer. Their father is incarcerated on charges of drug dealing, a second offense. The CWS Permanency Resource Division is doing an evaluation of the children’s aunt as a possible placement. The worker assigned to the case has been told that he should not do a concurrent plan for the children. What is the MOST LIKELY reason for this? It will likely not be difficult to place the three siblings together because of their young ages. A concurrent plan would be done only if more than one relative were to be assessed. The court has determined that the children will never be returned to father. Moving the placement along as quickly as possible by initiating emergency placement

Example of Analysis Answer Option Percent ESL Trainees who chose it Percent non-ESL Trainees who chose it a. 4.7 2.6 b. 8.2 3.6 c. 47.1 62.6 d. 40.0 31.3 Answer Option Percent who chose it a. 3.2 b. 5.5 c. 56.5 √ d. 34.9

A multiple choice question1 “A multiple choice item consists of a problem, known as the stem, and a list of suggested solutions, known as alternatives. The alternatives consist of one correct or best alternative, which is the answer, and incorrect or inferior alternatives, known as distractors. for slides 9-16 1 Brame, C., (2013) Writing good multiple choice test questions. Retrieved 8/3/2017 from https://cft.vanderbilt.edu/guides-sub-pages/writing-good-multiple-choice-test-questions/.

Example of meaningful stem 1. The stem should be meaningful by itself and should present a definite problem. A stem that presents a definite problem allows a focus on the learning outcome. A stem that does not present a clear problem, however, may test students’ ability to draw inferences from vague descriptions rather serving as a more direct test of students’ achievement of the learning outcome.

Example of irrelevant material 2. The stem should not contain irrelevant material, which can decrease the reliability and the validity of the test scores (Haldyna and Downing 1989).

3. The stem should be negatively stated only when significant learning outcomes require it. Students often have difficulty understanding items with negative phrasing (Rodriguez 1997). If a significant learning outcome requires negative phrasing, such as identification of dangerous laboratory or clinical practices, the negative element should be emphasized with italics or capitalization.

Use of interior blank in stem vs a question 4. The stem should be a question or a partial sentence. A question stem is preferable because it allows the student to focus on answering the question rather than holding the partial sentence in working memory and sequentially completing it with each alternative (Statman 1988). The cognitive load is increased when the stem is constructed with an initial or interior blank, so this construction should be avoided.

5. All alternatives should be plausible 5. All alternatives should be plausible. The function of the incorrect alternatives is to serve as distractors, which should be selected by students who did not achieve the learning outcome but ignored by students who did achieve the learning outcome. Alternatives that are implausible don’t serve as functional distractors and thus should not be used. Common student errors provide the best source of distractors. 6. Alternatives should be stated clearly and concisely. Items that are excessively wordy assess students’ reading ability rather than their attainment of the learning objective 7. Alternatives should be mutually exclusive. Alternatives with overlapping content may be considered “trick” items by test-takers, excessive use of which can erode trust and respect for the testing process.

Heterogeneous Content Which of the following is an engagement strategy in working with families? Asking questions about what the family does well Filling out paperwork about the interview Talking to neighbors about the family Discussing the family with your supervisor 8. Alternatives should be homogenous in content. Alternatives that are heterogeneous in content can provide cues to student about the correct answer.

Case Planning and Service Delivery Monitoring and Adapting Transition In the case planning process, in which phase is evaluating an initial report of child maltreatment most likely to occur? Engagement Assessment Case Planning and Service Delivery Monitoring and Adapting Transition In County X, on average, how many social workers are involved with a family throughout the life of the case when in the child welfare system? 1-3 3-5 6-8 9-11 9. Alternatives should be free from clues about which response is correct. Sophisticated test-takers are alert to inadvertent clues to the correct answer, such differences in grammar, length, formatting, and language choice in the alternatives. It’s therefore important that alternatives have grammar consistent with the stem. are parallel in form. are similar in length. use similar language (e.g., all unlike textbook language or all like textbook language). 10. The alternatives “all of the above” and “none of the above” should not be used. When “all of the above” is used as an answer, test-takers who can identify more than one alternative as correct can select the correct answer even if unsure about other alternative(s). When “none of the above” is used as an alternative, test-takers who can eliminate a single option can thereby eliminate a second option. In either case, students can use partial knowledge to arrive at a correct answer. 11. The alternatives should be presented in a logical order (e.g., alphabetical or numerical) to avoid a bias toward certain positions.  12. The number of alternatives can vary among items as long as all alternatives are plausible. Plausible alternatives serve as functional distractors, which are those chosen by students that have not achieved the objective but ignored by students that have achieved the objective. There is little difference in difficulty, discrimination, and test score reliability among items containing two, three, and four distractors. Brame, C., (2013) Writing good multiple choice test questions. Retrieved [todaysdate] from https://cft.vanderbilt.edu/guides-sub-pages/writing-good-multiple-choice-test-questions/.

Review and Discuss Review and discuss questions in your handouts What do you think is the right answer? How did you come to this conclusion? Any ideas on the strengths or weaknesses of this question?

Multiple Choice Question Activity Match.com Part 2 Training for Trainers Skill Development #4: Evaluation and Transfer of Learning September 19, 2014 Multiple Choice Question Activity Match.com Part 2 Pairs identified on match.com exercise will be asked to create a question based on their interest Once question is completed, post question on the wall near you (do not provide correct answer) Once all of the questions are correct, you’ll be “voting” on the correct answer and we’ll discuss IRENE 20 Irene Becker & James Coloma