1. Good MC Questions for Training and Evaluation
Irene Becker, LCSW
West Coast Child Welfare Trainer Conference 2017
Boise, Idaho

2. Learning Objectives
Identify at least three purposes for creating multiple choice (MC) questions
Recognize at least 6 components of developing MC questions
Analyze at least 6 MC and discuss the strengths and weaknesses of each
Develop one MC question and receive feedback on its strengths and weaknesses

3. Welcome & Introductions – Match.com
Training for Trainers Skill Development #4: Evaluation and Transfer of Learning
September 19, 2014
Welcome & Introductions – Match.com
Find a person whom you have a common interest in one of the following categories
Find your match and take a few minutes to discuss your interest
You and your match will be working on an activity together later in the training
Sports
Food & Beverage
Travel
Arts & Music
Literature/Reading
Games
IRENE 2
Irene Becker & James Coloma

4. Purpose in Creating the MC Questions
Low Stakes – for course improvement without any “consequences” to trainer and participant
High Stakes – involve data collection and some course of action for improvement. This can include:
Improvement in the curriculum
May be used as a feedback tool for trainers using a standardized curriculum
May find unexpected trends that lead to organizational improvement
May lead to consequences of employment or certification
The higher the stakes – the more rigorous the process needs to be in creating, analyzing, vetting and revising the questions

5. Questions for Team Based Learning
Focus on accuracy but also on critical thinking – the purpose is to discuss and come to a consensus answer via a group process
Sometimes consensus can’t be achieved, sometimes the group arrives at an incorrect answer – this allows for deepening discussion about the issue the question addresses
This process acknowledges where the participants are in their experience and learning processes with a goal of increasing accuracy through a group process
For more indepth information on creating questions via Bloom’s Taxonomy go to our wiki website

6. Example of Higher Stakes (but not the highest!)
Use of pre/post tests for measuring knowledge
Purpose is to gather data to look at trends, consistency across the state
What data are we looking for? (This is where demographic data is important – data gathering as an aggregate vs individuals). Each has implications on what you analyze
Questions that fall outside the range of 20-80% correct are looked at first
Deeper analysis of the questions by demographic groups ( IV-E vs non-IV-E, ESL, race, gender etc) provide more information about the multiple choice question. (Is the question a “fair” question?)
Determine revision process and test again

7. What is the MOST LIKELY reason for this?
Adrianna, age 3, and her 6 month old twin brothers, Aaron and Eddie, are in an emergency foster home because of the murder of their mother by a drug dealer.
Their father is incarcerated on charges of drug dealing, a second offense. The CWS Permanency Resource Division is doing an evaluation of the children’s aunt
as a possible placement. The worker assigned to the case has been told that he should not do a concurrent plan for the children.
What is the MOST LIKELY reason for this?
It will likely not be difficult to place the three siblings together because of their young ages.
A concurrent plan would be done only if more than one relative were to be assessed.
The court has determined that the children will never be returned to father.
Moving the placement along as quickly as possible by initiating emergency placement

8. Example of Analysis Answer Option Percent ESL Trainees who chose it
Percent non-ESL Trainees who chose it a.
4.7
2.6 b.
8.2
3.6 c.
47.1
62.6 d.
40.0
31.3
Answer Option
Percent who chose it a.
3.2 b.
5.5 c.
56.5 √ d.
34.9

9. A multiple choice question1
“A multiple choice item consists of a problem, known as the stem, and a list of suggested solutions, known as alternatives. The alternatives consist of one correct or best alternative, which is the answer, and incorrect or inferior alternatives, known as distractors.
for slides Brame, C., (2013) Writing good multiple choice test questions. Retrieved 8/3/2017 from

10. Example of meaningful stem
1. The stem should be meaningful by itself and should present a definite problem. A stem that presents a definite problem allows a focus on the learning outcome. A stem that does not present a clear problem, however, may test students’ ability to draw inferences from vague descriptions rather serving as a more direct test of students’ achievement of the learning outcome.

11. Example of irrelevant material
2. The stem should not contain irrelevant material, which can decrease the reliability and the validity of the test scores (Haldyna and Downing 1989).

12. 3. The stem should be negatively stated only when significant learning outcomes require it. Students often have difficulty understanding items with negative phrasing (Rodriguez 1997). If a significant learning outcome requires negative phrasing, such as identification of dangerous laboratory or clinical practices, the negative element should be emphasized with italics or capitalization.

13. Use of interior blank in stem vs a question
4. The stem should be a question or a partial sentence. A question stem is preferable because it allows the student to focus on answering the question rather than holding the partial sentence in working memory and sequentially completing it with each alternative (Statman 1988). The cognitive load is increased when the stem is constructed with an initial or interior blank, so this construction should be avoided.

14. 5. All alternatives should be plausible
5. All alternatives should be plausible. The function of the incorrect alternatives is to serve as distractors, which should be selected by students who did not achieve the learning outcome but ignored by students who did achieve the learning outcome. Alternatives that are implausible don’t serve as functional distractors and thus should not be used. Common student errors provide the best source of distractors.
6. Alternatives should be stated clearly and concisely. Items that are excessively wordy assess students’ reading ability rather than their attainment of the learning objective
7. Alternatives should be mutually exclusive. Alternatives with overlapping content may be considered “trick” items by test-takers, excessive use of which can erode trust and respect for the testing process.

15. Heterogeneous Content
Which of the following is an engagement strategy in working with families?
Asking questions about what the family does well
Filling out paperwork about the interview
Talking to neighbors about the family
Discussing the family with your supervisor
8. Alternatives should be homogenous in content. Alternatives that are heterogeneous in content can provide cues to student about the correct answer.

16. Case Planning and Service Delivery Monitoring and Adapting Transition
In the case planning process, in which phase is evaluating an initial report of child maltreatment most likely to occur?
Engagement
Assessment
Case Planning and Service Delivery
Monitoring and Adapting
Transition
In County X, on average, how many social workers are involved with a family throughout the life of the case when in the child welfare system?
1-3
3-5
6-8
9-11
9. Alternatives should be free from clues about which response is correct. Sophisticated test-takers are alert to inadvertent clues to the correct answer, such differences in grammar, length, formatting, and language choice in the alternatives. It’s therefore important that alternatives
have grammar consistent with the stem.
are parallel in form.
are similar in length.
use similar language (e.g., all unlike textbook language or all like textbook language).
10. The alternatives “all of the above” and “none of the above” should not be used. When “all of the above” is used as an answer, test-takers who can identify more than one alternative as correct can select the correct answer even if unsure about other alternative(s). When “none of the above” is used as an alternative, test-takers who can eliminate a single option can thereby eliminate a second option. In either case, students can use partial knowledge to arrive at a correct answer.
11. The alternatives should be presented in a logical order (e.g., alphabetical or numerical) to avoid a bias toward certain positions. 
12. The number of alternatives can vary among items as long as all alternatives are plausible. Plausible alternatives serve as functional distractors, which are those chosen by students that have not achieved the objective but ignored by students that have achieved the objective. There is little difference in difficulty, discrimination, and test score reliability among items containing two, three, and four distractors.
Brame, C., (2013) Writing good multiple choice test questions. Retrieved [todaysdate] from

17. Review and Discuss Review and discuss questions in your handouts
What do you think is the right answer?
How did you come to this conclusion?
Any ideas on the strengths or weaknesses of this question?

18. Multiple Choice Question Activity Match.com Part 2
Training for Trainers Skill Development #4: Evaluation and Transfer of Learning
September 19, 2014
Multiple Choice Question Activity Match.com Part 2
Pairs identified on match.com exercise will be asked to create a question based on their interest
Once question is completed, post question on the wall near you (do not provide correct answer)
Once all of the questions are correct, you’ll be “voting” on the correct answer and we’ll discuss
IRENE 20
Irene Becker & James Coloma