Constructed Response Tests Maureen Spelman, Ed. D. Saint Xavier University EDU 343 – Online Meeting # 1

Constructed-Response Tests Short Answer Items Essay Items: Development Essay Items: Scoring

Short Answer Items Short answer items call for students to supply a word, a phrase, or a sentence in response to either a direct question or an incomplete statement. If an item requires a fairly lengthy response – it is an essay item, not a short answer item. Short answer items are suitable for assessing simple, knowledge level learning outcomes.

One disadvantage is that short-answer items are more difficult to score. The longer the response you seek – the tougher it will be to score the item accurately. Thus, when you choose constructed- response items you must be willing to trade-off some scoring accuracy.

Remember your “5 Commandments” for creating effective assessments... they hold for constructed response assessments as well.

Item Writing Guidelines for Short-Answer Items In the case of short-answer items it is generally better to use direct questions vs. incomplete statements – particularly for younger and/or less able students. * Direct questions are less confusing * Direct question format forces a tightly phrased response. * It is best to decide upon the correct answer, then build the question to elicit that response.

It is important to structure short-answer items to nurture a concise response from students. * You may want to place word limits on each response... * or, at the very least, indicate in the directions that only a short one-sentence response will be allowed.

Carefully construct items to elicit a concise and accurate response... A.An animal that walks on two feet is _______. * There are all sorts of answers that a student might legitimately make to such a test item

B.An animal that walks on two feet is technically classified as _____________. * This type of incomplete statement is more likely to elicit the desired response – “biped”.

Try to put yourself, mentally, inside the heads of your students and strive to anticipate how they are apt to interpret an item.

Placing blanks in the margin (for direct questions) or at the end of incomplete statements helps by lining up student responses nicely for scoring purposes.

Examine this ‘fill-in-the-blank’ item and rephrase it easier scoring... * The ____________ is the governmental body which, based on the U. S. Constitution, must ratify all U.S. treaties with foreign nations.

Does your new question look something like this? The governmental body which, based on the U. S. Constitution, must ratify all U. S. treaties with foreign nations is the _________________.

In the case of incomplete statements, it is best to use only one or two blanks. * Too many blanks and you will end up with a “Swiss cheese” assessment item. After a series of major conflicts with natural disasters, in the year ____, the explorers ____ & ____, accompanied by their ____, discovered ____.

Please be sure that all blanks are treated equally!!! * If you vary the length of the blanks according to the length of the correct response – you are supplying unintended clues to your students. * However, it is also important to provide blank spaces that are sufficient for students’ responses.

PAUSE! Please go to our discussion site – You will find 4 discussion prompts related to Short Answer test items. In your discussion group find the flaws in each item. Be sure to use evidence from the ‘5 Commandments and/or the Item Writing Guidelines to support your comments.

Essay Items The essay is the most commonly used form of constructed-response assessment items. Essay items are particularly useful in gauging a student’s ability to synthesize, evaluate and compose (higher order skills in Bloom’s Cognitive Taxonomy).

One concern with essay items is that it is much more difficult to design good ones than you might imagine. Generating really good essay items is a tough task for most teachers. AND, the biggest challenge with essay items is the difficulty in reliably scoring items.

Item Writing Guidelines for Essay Items It is imperative that teachers clearly convey just how extensive of a response is desired. * You can request just a bit more than a short-answer item (restricted response items), or * You can ask for extremely lengthy responses (extended response items).

The following slide contains an example of a well written restricted response item and a well written extended response item.

Describe the 3 most common ways HIV, the AIDS virus, is transmitted. Take no more than 25 words to describe each method of transmission. Identify the chief factors contributing to the U.S. government's financial deficit during the past 2 decades. Having identified those factors, decide which factors, if any, have been explicitly addressed by the U.S. branches of government in the last 5 years. Critically evaluate whether or not proposed remedies will bring about significant reductions in the national debt.

* Think carefully about whether you want more essay items requiring shorter responses... OR fewer essay items requiring extensive responses.

It is good practice to provide students with time to be expended AND each item’s value – up front. * If students are unaware of which items count most they may put more time and energy into low-value items... leaving insufficient time to give more than a “nod” to higher-value items later in the test.

This problem has an easy remedy -simply provide this information in the directions – or parenthetically at the end of each item.

Much as we may think we are being “kind” and/or doing students a “favor” – it is best not to offer optional items on essay exams – i.e. choose any 3 of the following 5 essay items... * When students select from different items, they are essentially taking different exams. This makes it impossible for the teacher to judge/compare exam performance on a common scale!

Remember that you are aiming to make better teaching decisions using test-based inferences. It is tough enough to make those inferences when you have only one test to consider. Offering what essentially amounts to a medley of exams will make your task infinitely more difficult.

Do it yourself! * One of the best ways to judge an essay item’s quality is by composing (mentally or in writing” a sample response. * This practice can aid in detecting flaws while there is still time to “fix” the item.

PAUSE! Please go to our discussion site – You will find 4 discussion prompts related to Essay test items. In your discussion group find the flaws in each item. Be sure to use evidence from the ‘5 Commandments and/or the Item Writing Guidelines to support your comments.

Guidelines for Scoring Responses to Essay Items Almost any constructed-response item can be scored either holistically or analytically. * Holistic scoring – focuses on the essay response or written composition as a whole. * The teacher focuses on a general impression – an overall grade to the student’s response.

* Analytical Scoring – requires that the teacher isolate essential evaluative criteria that will add up to an overall score (generally 4 – 6 points per essay). For example: * organization * clarity * adaptation to audience * word choice * mechanics

The advantage of analytical scoring is that it can help you to identify the specific strengths and weaknesses of your students’ performances and then equip you to communicate specific points for improvement.

The disadvantage is that sometimes teacher become so focused on the evaluative criteria they will miss the student’s overall response (failure to see the forest because of the trees).

A middle of the road compromise might be to initially grade all responses holistically, then revisit those that were judged unsatisfactory for an analytical evaluation. As always – your decisions should be based upon the intended use of the results – form does follow function!

Once a gain, it may be useful to develop a scoring key in advance of test administration. * If you wait until you begin scoring individual test papers – there is a strong possibility that you will be influenced by the responses of the first few papers you grade!

Make your decisions regarding the importance of MUGS (mechanics, usage, grammar, syntax/spelling) prior to grading. * Few things influence scores of students’ essays as much as MUGS.

* Simply make up your mind about this issue early in the process so that if you find MUGS are not the issue – then you won’t let MUGS subconsciously influence your scoring of responses.

Troublesome as it may seem – it is more effective to score all responses to a single item before scoring the responses to the next item. * In other words, don’t score all the responses of student “A” – then proceed to student “B”.

* There is too much danger that a student’s response to item #1 will influence your scoring of subsequent items (2, 3, 4,etc. for that same student). * Scoring all responses to item #1 and then all responses to item #2 will lead to more consistent, unbiased scoring – and therefore more accurate data regarding your students.

If at all possible, evaluate responses anonymously. * One trick is to ask students write their names on the reverse side of the last exam sheet. Force yourself not to peek until you have scored all of the exams.

* You may sincerely be surprised that “Silent Sarahs” and “Quiet Quentins” may produce solid exam performances.

References Marzano, R.J. (2006). Classroom assessment & grading that work. Alexandria, VA: Association for Supervision and Curriculum Development. Popham, W.J. (2008). Classroom assessment: What teachers need to know. New York: Allyn & Bacon, Pearson. Popham, W. J. (2003). Test better, teach better: The role of instructional assessment. Alexandria, VA: Association for Supervision and Curriculum Development.