1. How to Write Exams An Office of Learning Excellence Workshop Art Hill, OMS Department Steve Huchendorf, OMS Department February 8, 2006

2. Agree or disagree?
Student learning depends heavily on the assessment method.
Ungraded assessments can be even more important than graded assessments in the student learning experience.
Assessments should always be aligned with course goals.
It is almost impossible to measure higher order thinking.
An in-class test is usually the best way to assess learning.
It is usually best to have only one type of assessment method.
Assessments should be primarily at the end of the course.

3. McKeachie’s Nine Assertions
Student learning depends on assessment. What is measured is valued.
Tests are for facilitating learning, not merely assigning grades
Include informal (non-graded) Classroom Assessment Techniques (CATs) for feedback about learning.
Match assessment methods with course goals. If higher order thinking is a goal, is it being assessed?
Some goals (motivation, attitudes, values, ethics) may not be measurable by conventional means. Develop other methods for assessment
Testing is not the only assessment technique. Look for other classroom and out-of-class activities: Fits, Portfolios, Peer Assessment
Develop student self-assessment skills through training and practice.
Provide a variety of assessments, don’t rely solely on mid-term and final exams.
Assessment is not only an end-of-class activity. Assessment can be an active learning technique throughout the course.
Source: McKeachie's’ Teaching Tips, 12th edition, Chapter 7, pgs

4. Outline Warm-up exercise How to write multiple-choice questions (Art)
On-line exams at the U of M (Art)
Classroom assessment techniques – CATs (Steve)
Conclusions (Art)

5. How to write multiple-choice questions
Multiple-choice items have three components:
Item stem
Question to be answered
Problem to be solved
Incomplete statement to be completed
The item key – the response with the correct answer.
The item distracters – the incorrect responses.

6. How to write multiple-choice questions
Types of MC questions:
One correct answer – One unambiguous answer.
One best answer – Best answer based on judgment.
Multiple answers – Multiple correct answers.
Negative stem – Which one of is NOT correct?
Combined response – Pick combinations of alternatives that are correct (I only, II only, III only, I and II, etc.).
True-false (yes/no or right/wrong).

7. How to write multiple-choice questions
Require clarity – Define the context, make all necessary qualifications, use precise meanings of words in standard English, and avoid superfluous information.
Focus on the content that matters – not peripheral or trivial.
Randomize the order of the responses.
Use parallel grammatical construction for responses.
Do not repeat words from the stem in the responses.
Avoid the words all, none, certainly, never, and always.
Avoid double barreled questions – especially for true/false questions.
Avoid negative questions.
Make the distracters plausible and avoid silly or absurd distracters.
Use “none of the above” and “all of the above” rarely and with caution.

8. How to write multiple-choice questions
Assemble in groups of 2 or 3 people. (This might be a good chance to make some new friends.)
Evaluate this question.
A good job description usually contains all of the following except:
a. The performance level of the current employee
b. A general job summary
c. An identification of the job
d. The duties and responsibilities of the job
e. All of the above are included
Write one or two multiple choice questions each for your discipline – and then have your other group members evaluate your questions.

9. Outline Warm-up exercise How to write multiple-choice questions (Art)
On-line exams at the U of M (Art)
Classroom assessment techniques – CATs (Steve)
Conclusions (Steve)

10. On-line exams at the U of M
My experience:
Take as many times as they want.
Any time until the day before grades are due.
Open book and open notes, but must work individually.
One-hour time limit.
Exactly 50 test items randomly selected from database.
Database is organized into 14 categories, with a set number for each.
Students see their score immediately after completing the exam.
Students get extensive feedback immediately after the exam.
Only the highest exam score is recorded in the grade book.
All scores, previous exams, and previous exam feedback is available.

11. On-line exams at the U of M
Benefits:
Frees up significant class time for class discussion and interaction.
Provides real accountability for learning the basic material.
Gets the “basic” material out of the way so that the classroom can focus on the higher-order critical thinking skills.
Takes a lot of the stress out the testing process – which is appropriate for adult learners.
Maximizes the learning experience – students can keep trying and keep learning until they get the grade they want (or run out of time).

12. On-line exams at the U of M
Some challenges:
Only for the U of M community
Slightly difficult for data entry and editing
Somewhat difficult data management (e.g., import/export to Excel was nearly impossible).
Very limited test structure – only one level.
Extremely limited test statistics – wanted response level and item statistics.

13. Outline Warm-up exercise How to write multiple-choice questions (Art)
On-line exams at the U of M (Art)
Classroom assessment techniques – CATs (Steve)
Conclusions (Art)

14. Classroom Assessment Techniques (ALT-CATs)
Angelo & Cross, Classroom Assessment Techniques
McKeachie, Teaching Tips
Active Learning Techniques (ALTs)

15. Classroom Assessment Techniques (ALT-CATs)
Angelo & Cross, Classroom Assessment Techniques
McKeachie, Teaching Tips
Active Learning Techniques (ALTs) +
Classroom Assessment Techniques (CATs)

16. Classroom Assessment Techniques (ALT-CATs)
Angelo & Cross, Classroom Assessment Techniques
McKeachie, Teaching Tips
Active Learning Techniques (ALTs) + =
ALT-CATs
Classroom Assessment Techniques (CATs)

17. Classroom Assessment Techniques (ALT-CATs)
Informal assessments of classroom learning
Did the students get it? Find out!
Bush Grant “Improving Student Learning in Large Lectures”
ALT-CATs in MBA 6120 Data Analysis and Statistics for Managers, Online Section – Spring 2006

18. Classroom Assessment Techniques (ALT-CATs)
Instructor
Final Exam
Mid-Term Exam
Quiz
Assignment
Active Learning
Assessment & Feedback
Lecture
Pre-read
Student Learning

19. Classroom Assessment Techniques (ALT-CATs)
Instructor
Final Exam
Assessment & Feedback
Mid-Term Exam
Quiz
Assignment
Active Learning
Assessment & Feedback
Lecture
Pre-read
Student Learning

20. Classroom Assessment Techniques (ALT-CATs)
Incorporating ALT-CATs into course design:
ALT-CAT opportunities provided throughout the semester
Each ALT-CAT completed in a team environment
Teams of 2, 3 or 4 Team members sign their names
ALT-CATs are turned in and scanned for informal assessment
Recorded as 5 points  Participation points
Begin the next class period with an overview
‘Teachable Moment’ - Most common mistakes
Bridge from “Where we’ve been” to “Where we’re going”

21. Classroom Assessment Techniques (ALT-CATs)
Informal (non-graded) Classroom Assessment Techniques (ALT-CATs):
Angelo & Cross Classroom Assessment Techniques
Think-Pair-Share, Write-Pair-Share
One Minute Write / Muddiest Point
Buzz Groups
Focused Listings
Shared brainstorming
Applications cards
Background knowledge probe
Concept maps
Invented dialogues
Analytic Memos
Empty Outlines

22. Classroom Assessment Techniques (ALT-CATs)
My Experience with ALT-CATS:
Additional incentive for attendance
Additional incentive for participating in Active Learning
Additional incentive for cooperative learning
Assists with the two endpoints of student participation
Isolates the ‘dominator’
More opportunities for the non-participant
Rapid Response - More immediate feedback to the student
Comfort level of class  Working in same small group
Connects the ‘islands’

23. An ALT-CAT for the ALT-CATs
Assemble in groups of 2 or 3 people.
Create an Applications Card ALT-CAT.
Select an ALT-CAT and a concept that you teach in your course
Identify how you would adapt the ALT-CAT in creating an active learning experience for your students
Buzz Groups – discuss with your group.

24. Outline Warm-up exercise How to write multiple-choice questions (Art)
On-line exams at the U of M (Art)
Classroom assessment techniques – CATs (Steve)
Conclusions (Art)

25. Conclusions Formal (graded) assessments
Exams, homework assignments, quizzes, case studies, projects, portfolios, etc.
Informal (non-graded) assessments
Think-pair-share, buzz groups, problem-based learning, concept maps, analytic memos, etc.
Online assessment tools
Writing multiple-choice exams is really hard!
The vision …