English Language Arts Selected Response, Constructed Response, Technology-Enhanced Item Design Welcome to the Smarter Balanced Assessment Consortium’s English Language Arts Selected Response, Constructed Response, and Technology-Enhanced Item Design Training Module. The Smarter Balanced assessment system uses a variety of items and tasks, ranging from brief selected response and constructed response items that target specific concepts or skills to more extended performance tasks that require the application of English Language Arts skills to complex real-world scenarios.

Learning Outcomes Understand the purpose of selected response, constructed response, and technology-enhanced items Describe the components of selected response, constructed response, and technology-enhanced items Understand the essential requirements of quality items Apply general guidelines for writing selected response, constructed response, and technology-enhanced items Identify exemplary and flawed items This module explores commonly accepted best practices for writing selected response, constructed response, and technology-enhanced items. {+} The module will help participants deepen their understanding of the purpose of selected response, constructed response, and technology-enhanced items, describe the components of selected response, constructed response, and technology-enhanced items, understand the essential requirements of quality items, apply general guidelines for writing selected response, constructed response, and technology-enhanced items, and identify exemplary and flawed items.

Benefits and Limitations of Selected Response Items Answered quickly Assess a large range of content on one test Inexpensive to score Results collected quickly Limitations Limited ability to reveal a student’s reasoning process Difficult to assess higher- order thinking skills Selected response items have many benefits. {+} Selected response items are designed to be answered within 1 or 2 minutes and allow the opportunity to assess a broad range of content in one test. Selected response items are objectively scored and student results are collected quickly. Despite these benefits, there are two limitations. With selected responses, it is difficult to understand a student’s reasoning process and to assess higher-order thinking skills.

Formats and Components of Selected Response Items Lizards are fascinating creatures. There are over 3,000 known species, including monitors, skinks, geckos, chameleons, and iguanas, and they vary greatly in appearance. The largest lizard, the Komodo dragon, can grow over ten feet long, and the smallest, the Jaragua lizard, can fit on a dime. Skinks usually have smooth scales like snakes, iguanas have mohawk-like crests running down their backs, and the moloch is covered with spikes from head to tail. Lizards vary in color from shades of gray and brown to bright red or green, spotted or striped. Most have four legs but some are legless and easily confused with snakes (Hint: if it has external eardrums and eyelids it’s a lizard). Geckos can walk up walls. Chameleons not only change color but also have prehensile tails, similar to those of monkeys, that wrap around branches and their eyes can move in different directions. What is the best way to revise the highlighted sentence to match the language and style of the paragraph? Geckos are able to adhere to flush surfaces because setae on their footpads facilitate van der Waals forces between the setae structures and the surface. Geckos are awesome because they have sticky toes that allow them to climb windows like Spiderman. Geckos have the remarkable ability to walk up walls thanks to tiny hair-like structures on their toes that cling to smooth surfaces. Geckos scurry up walls like tiny dancers gliding effortlessly across a stage, their movements as natural as a well-rehearsed ballet. STIMULUS Lizards are fascinating creatures. There are over 3,000 known species, including monitors, skinks, geckos, chameleons, and iguanas, and they vary greatly in appearance. The largest lizard, the Komodo dragon, can grow over ten feet long, and the smallest, the Jaragua lizard, can fit on a dime. Skinks usually have smooth scales like snakes, iguanas have mohawk-like crests running down their backs, and the moloch is covered with spikes from head to tail. Lizards vary in color from shades of gray and brown to bright red or green, spotted or striped. Most have four legs but some are legless and easily confused with snakes (Hint: if it has external eardrums and eyelids it’s a lizard). Geckos can walk up walls. Chameleons not only change color but also have prehensile tails, similar to those of monkeys, that wrap around branches and their eyes can move in different directions. STEM What is the best way to revise the highlighted sentence to match the language and style of the paragraph? OPTIONS Geckos are able to adhere to flush surfaces because setae on their footpads facilitate van der Waals forces between the setae structures and the surface. Geckos are awesome because they have sticky toes that allow them to climb windows like Spiderman. Geckos have the remarkable ability to walk up walls thanks to tiny hair-like structures on their toes that cling to smooth surfaces. Geckos scurry up walls like tiny dancers gliding effortlessly across a stage, their movements as natural as a well-rehearsed ballet. The Smarter Balanced Assessment System uses both traditional and non-traditional selected response items. Let’s take a look at the format and components of a traditional selected response item. {+} This item is a traditional Grade 9 selected response that is sometimes referred to as a multiple choice item. This item contains a stimulus, a stem and four options. The stimulus is the passage that sets the context for the item. The stem is the statement of the question to which the student responds. The stem is sometimes also referred to as a prompt. The options are possible answers the students must select from. Options should be arranged according to a logical order such as numerically, alphabetically, or the order in which information is presented in the stimulus. There are four different ways to respond to this item but only one correct answer. Distractors are incorrect answer options and the key is the correct answer. The distractor analysis explains why the correct answer is correct. Distractor Analysis A. This option uses too much scientific language to fit with the rest of the paragraph. B. This option is more informal than the rest of the paragraph. C. KEY. This option uses formal, non-technical language that fits well with the rest of the paragraph. D. This option uses too much figurative language to fit with the rest of the paragraph.

Formats and Components of Selected Response Items Read the passage and then answer the question. Users may experience missions in real-time, and "Eyes on the Solar System" also allows them to travel through time. The tool is populated with NASA data dating back to 1950 and projected to 2050. The National Aeronautics and Space Administration (NASA) leads research in space exploration and aeronautics. NASA Gives Public New Internet Tool To Explore The Solar System The playback rate can be sped up or slowed down. When NASA's Juno spacecraft launched on Aug. 5, 2011, users could look ahead to see the mission's five-year journey to Jupiter in a matter of seconds. PASADENA, Calif. – NASA is giving the public the power to journey through the solar system using a new interactive Web-based tool. Point of view can be switched from faraway to close-up to right "on board" spacecraft. Dozens of controls on a series of pop-up menus allow users to fully customize what they see, and video and audio tutorials explain how to use the tool's many options. Users may choose from 2-D or 3-D modes, with the latter simply requiring a pair of red-cyan glasses to see. The "Eyes on the Solar System" interface combines video game technology and NASA data to create an environment for users to ride along with agency spacecraft and explore the cosmos. Screen graphics and information such as planet locations and spacecraft maneuvers use actual space mission data. "This is the first time the public has been able to see the entire solar system and our missions moving together in real-time," said Jim Green, director of NASA's Planetary Science Division at the agency's Headquarters in Washington. "It demonstrates NASA's continued commitment to share our science with everyone." "By basing our visualization primarily on mission data, this tool will help both NASA and the public better understand complex space science missions," said Kevin Hussey, manager of Visualization Technology Applications and Development at JPL, whose team developed "Eyes on the Solar System." "Eyes on the Solar System" is in beta release. It has been demonstrated at science conferences, in classrooms and at the 2011 South by Southwest Interactive Conference in Austin, Texas. The virtual environment uses the Unity game engine to display models of planets, moons, asteroids, comets and spacecraft as they move through our solar system. With keyboard and mouse controls, users cruise through space to explore anything that catches their interest. A free browser plug-in, available at the site, is required to run the Web application. Designers are updating "Eyes on the Solar System" to include NASA science missions launching during the coming months, including GRAIL to the moon and the Mars Science Laboratory Curiosity rover. "You are now free to move about the solar system," said Blaine Baggett, executive manager in the Office of Communication and Education at NASA's Jet Propulsion Laboratory (JPL) in Pasadena, Calif. "See what NASA's spacecraft see – and where they are right now – all without leaving your computer." Select the two highlighted sentences that show that NASA makes predictions about the data space missions will generate before the missions occur. This is an example of a non-traditional selected-response item written for grade 9. Like the previous item, there is a stimulus and a prompt. {+} For this item, however, the student is asked to select multiple pieces of text. To respond correctly, students must select two correct sections of text. {+} The key indicates which two highlighted portions of the text are correct. An item writer should consider creating alternate selected-response items when, by doing so, the alternate items provide more information about a student’s knowledge, skills, and abilities than can be gained through a traditional selected-response item. KEY: “The tool is populated with NASA data dating back to 1950 and projected to 2050.” AND “When NASA's Juno spacecraft launched on Aug. 5, 2011, users could look ahead to see the mission's five-year journey to Jupiter in a matter of seconds.” In order for NASA to include projected data to 2050 and the five-year mission of a spacecraft that just launched the agency must predict data from missions before the missions occur. Distractors: The other options provide important information about the new internet tool but none of them support the inference that NASA makes predictions about the data space missions will generate before the missions occur.

Purpose of Constructed Response Items Address assessment targets and claims that are of greater complexity Require more analytical thinking and reasoning Constructed response items are brief open-response items that focus on a particular skill or concept and require students to produce a short written response. Constructed response items address assessment targets and claims that are of greater complexity, requiring more analytical thinking and reasoning than a selected response can elicit.

Administration of Constructed Response Items Administered during the computer-adaptive component Scored using artificial intelligence Most constructed response items take between 1 and 5 minutes to complete Some more complex items may take up to 10 minutes to complete Unlike Performance Tasks, which are the topic of another module, {+} constructed response items are designed to be administered during the computer-adaptive component of the assessment. {+} In order to score constructed response items quickly, automated scoring using artificial intelligence will be employed. {+} Most constructed response items should take between 1 and 5 minutes to complete. {+} Some more complex items may take up to 10 minutes to complete.

Components of a Constructed Response Item The Shepherd’s Boy and the Wolf A Shepherd's Boy was tending his flock near a village, and thought it would be great fun to trick the villagers by pretending that a Wolf was attacking the sheep: so he shouted out, "Wolf! Wolf!" and when the people came running up he laughed at them because they believed him. He did this more than once, and every time the villagers found they had been tricked, for there was no Wolf at all. At last a Wolf really did come, and the Boy cried, "Wolf! Wolf!" as loud as he could: but the people were so used to hearing him call that they took no notice of his cries for help. And so no one came to help the boy, and the Wolf attacked the sheep. In a few sentences, explain what lesson the reader can learn from the shepherd’s boy. Use details from the story to support your response. STIMULUS The Shepherd’s Boy and the Wolf A Shepherd's Boy was tending his flock near a village, and thought it would be great fun to trick the villagers by pretending that a Wolf was attacking the sheep: so he shouted out, "Wolf! Wolf!" and when the people came running up he laughed at them because they believed him. He did this more than once, and every time the villagers found they had been tricked, for there was no Wolf at all. At last a Wolf really did come, and the Boy cried, "Wolf! Wolf!" as loud as he could: but the people were so used to hearing him call that they took no notice of his cries for help. And so no one came to help the boy, and the Wolf attacked the sheep. STEM In a few sentences, explain what lesson the reader can learn from the shepherd’s boy. Use details from the story to support your response. All constructed response items are worth 2 to 4 points. {+} Let’s take a look at an example of a 2 point constructed response item. Like selected response items, Constructed response items have a stimulus and a stem.

Components of a Constructed Response Item 2-point Scoring Rubric 2 The response: gives evidence of the ability to explain inferences about theme includes specific inferences that make reference to the text supports the inferences with relevant details from the text 1 gives limited evidence of the ability to explain inferences about theme includes inferences but they are not explicit or make only vague references to the text supports the inference with at least one detail but the relevance of that detail to the text must be inferred A response gets no credit if it provides no evidence of the ability to explain inferences about theme and includes no relevant information from the text. Scoring Notes Response may include but is not limited to: The shepherd’s boy learned that he shouldn’t call wolf unless there is really a wolf. The shepherd’s boy learned that he should only ask for help if he needs it or else he wouldn’t get help when he really needs it. “The people were so used to hearing him call that they took no notice of his cries.” The shepherd’s boy learned not to have fun by tricking people because the people learn not to trust you. Score Point 2 Sample: The lesson learned from this story is do not cry for help when nothing is wrong. The shepherd’s boy pretends that a big wolf is attacking his sheep and yells, “Wolf! Wolf!” The people in the village run out to help him because they believe he needs help. After he tricks the villagers more than once, they realize he is just pretending. Score Point 1 Sample: The lesson learned from this story is do not cry for help when nothing is wrong. The shepherd’s boy cries wolf when there is no wolf and the people come to help him. Score Point 0 Sample: Readers learn a good lesson about how to cry wolf. All constructed response items also must include {+} a scoring rubric, scoring notes, and sample responses. The scoring rubric for each item distinguishes between characteristics of responses that provide evidence that the student has partially or fully developed the skill or knowledge defined by the assessment target. The scoring notes detail the information that should be included in a correct response. The sample responses provide concrete examples of what a response for each point value might look like. The top score sample should showcase a complete and thorough response. The language contained in samples should model what is expected from a student at the grade level being assessed.

Essential Requirements of Selected Response and Constructed Response Items Alignment to claims and assessment targets Appropriate content and contexts Proper presentation: clarity, readability, and accessibility Stimulus text is at or below the assessed grade level for assessed claim Free of bias issues Claim 1 and 4 constructed responses require support Plausible distractors Distractors represent common mistakes When developing selected and constructed response items, there are several requirements that are important to keep in mind. These requirements include {+} proper alignment to claims and assessment targets, appropriate English language arts content and contexts, and proper presentation. In addition, the stimulus text should be at or below the assessed grade level as required by the claim. Claim 1 stimuli should represent a range of difficulty. Stimulus texts for Claims 2 and 4 should be below the assessed grade level. Items should be free of bias issues in regards to age, gender, race, ethnicity, language, religion, socioeconomic status, disability or geographic region. Constructed-response items for Claim 1 reading and Claim 4 research must ask students to support their response with information from the stimulus text or texts. Selected-response items must contain plausible but incorrect distractors. Finally, selected-response distractors should not be any incorrect answer but carefully constructed incorrect answers that represent common mistakes in order to elicit information about student misconceptions.

General Guidelines for Writing Selected Response and Constructed Response Items Aligned to the English Language Arts Standards Aligned to claims and targets in the Smarter Balanced Content Specifications When writing an item, it is essential that {+} each item is aligned to the Common Core State Standards for English Language Arts. Each item should also be written to assess a primary claim and target from the Smarter Balanced Content Specifications. Each selected response and constructed response item should be written to focus primarily on one assessment target. Secondary targets are acceptable for some Claim 1 targets.

General Guidelines for Writing Selected Response and Constructed Response Items Concepts detailed in the standards for lower grades Clearly stated so students understand the task Written to clearly elicit the desired evidence of a student’s knowledge, skills, and abilities Appropriate reading level as required by the claim Follow Universal Design principles Depth of Knowledge considered Items may include {+} English language arts concepts detailed in the Common Core State Standards of lower grades. Items should be clearly stated to ensure that students understand the task. Items should be written to clearly elicit the desired evidence of a student’s knowledge, skills, and abilities. Item stimuli should be at an appropriate reading level as required by the claim. Items should follow Universal Design principles, And depth of knowledge level should be considered.

Flawed Elementary Selected Response Example Item Grade: 3 Claim 1: Students can read closely and analytically to comprehend a range of increasingly complex literary and informational texts Target 3. Word Meanings: Determine intended meanings of words, including words with multiple meanings (academic/tier 2 words), based on context, word relationships, word structure (e.g., common roots, affixes), or use of resources (e.g., beginning dictionary) DOK: 2 DOES NOT MATCH TARGET TARGET Amelia Earhart Amelia Earhart Learns to Fly Amelia Earhart was born in Atchison, Kansas, on July 24, 1897. In those days, airplanes were not nearly as common as they are today. Earhart was 12 years old before she ever saw an airplane, and she did not take her first flight until 1920. Amelia Earhart was so thrilled by her first airplane ride that she quickly began to take flying lessons. She wrote, "As soon as I left the ground, I knew I myself had to fly."   Earhart excelled as a pilot. Her first instructor was Neta Snook, one of the first women to graduate from the Curtiss School of Aviation. Earhart borrowed money from her mother to buy a two-seat plane. She got her U.S. flying license in December 1921, and by October 1922, she set an altitude record for women of 14,000 feet. In 1923, Earhart received her international pilot's license - only the 16th woman to do so. At the same time, she was becoming famous for her aviation achievements. Amelia Earhart Flies Across the Atlantic In 1928, Amelia Earhart received a phone call that would change her life. She was invited to become the first woman passenger to cross the Atlantic Ocean in a plane. "The idea of just going as 'extra weight' did not appeal to me at all," she said, but she accepted the offer nonetheless. On June 17, after several delays due to bad weather, Amelia Earhart flew in a plane named Friendship with co-pilots Wilmer "Bill" Stultz and Louis "Slim" Gordon. The plane landed at Burry Port, South Wales, with just a small amount of fuel left. In 1928, Amelia Earhart was offered the opportunity to become the first woman passenger to fly across the Atlantic Ocean. Although this was an extraordinary honor, Amelia said, "The idea of just going as 'extra weight' did not appeal to me at all.” Why did Amelia choose the phrase “extra weight” to describe herself? A. She wanted to take other people with her.   B. She was only going to be a passenger on the plane. KEY C. She knew she was fat and weighed too much to fly on an airplane.  D. She was saying that women are silly and unimportant. TOO COMPLEX Now that the basic elements of selected response and constructed response items have been explored, let’s develop a better understanding of the characteristics of quality items by looking at some examples of faulty and exemplary selected response items. {+} This is an example of a poorly written selected response item for grade 3. There are several problems with this item. Let’s pause for thirty seconds so that you can review this item and think about issues that may make this item flawed. First, the stimulus text is too complex for grade 3 students. Second, the item is not aligned to the assessment target. The target requires students to determine the meaning of words but the item is more about understanding the figurative connotation of a phrase. The stem is wordy, confusing, and contains a lot of unnecessary information. Option A is not very plausible given the context of the passage. Finally, options C and D contain bias and sensitivity issues. POOR WORDING NOT PLAUSIBLE BIAS

Well Written Elementary Selected Response Example Item Grade: 4 Claim 1: Students can read closely and analytically to comprehend a range of increasingly complex literary and informational texts Target 14. LANGUAGE USE: Determine or interpret figurative language/literary devices or connotative meanings of words and phrases used in context and the impact of those word choices on meaning and tone. DOK: 2 DIFFERENT GRADE DIFFERENT TARGET Amelia Earhart Amelia Earhart Learns to Fly Amelia Earhart was born in Atchison, Kansas, on July 24, 1897. In those days, airplanes were not nearly as common as they are today. Earhart was 12 years old before she ever saw an airplane, and she did not take her first flight until 1920. Amelia Earhart was so thrilled by her first airplane ride that she quickly began to take flying lessons. She wrote, "As soon as I left the ground, I knew I myself had to fly."   Earhart excelled as a pilot. Her first instructor was Neta Snook, one of the first women to graduate from the Curtiss School of Aviation. Earhart borrowed money from her mother to buy a two-seat plane. She got her U.S. flying license in December 1921, and by October 1922, she set an altitude record for women of 14,000 feet. In 1923, Earhart received her international pilot's license - only the 16th woman to do so. At the same time, she was becoming famous for her aviation achievements. Amelia Earhart Flies Across the Atlantic In 1928, Amelia Earhart received a phone call that would change her life. She was invited to become the first woman passenger to cross the Atlantic Ocean in a plane. "The idea of just going as 'extra weight' did not appeal to me at all," she said, but she accepted the offer nonetheless. On June 17, after several delays due to bad weather, Amelia Earhart flew in a plane named Friendship with co-pilots Wilmer "Bill" Stultz and Louis "Slim" Gordon. The plane landed at Burry Port, South Wales, with just a small amount of fuel left. Amelia said, "The idea of just going as 'extra weight' did not appeal to me at all.” What does the phrase ‘extra weight’ refer to? A. Her fame as an international pilot B. Her role as a passenger on the plane C. Her understanding of how heavy she was D. Her awareness of how she was making history Now let’s examine how this item could be revised to be a well written item. Let’s pause for thirty seconds so that you can review this item and think about why it is a better item. {+} The complexity and readability of the stimulus text is more appropriate for 4th grade students. The item correlates well to target 14, which requires students to interpret the figurative meanings of text. The wording of the stem is clear, and concise. The answer options are plausible and elicit information about student misconceptions. The answer options have also been revised to be free of bias and sensitivity issues. BETTER WORDING PLAUSIBLE AND BIAS FREE

Flawed Middle School Selected Response Example Item Grade: 6 Claim 2: Students can produce effective writing for a range of purposes and audiences. Target 9. EDIT/CLARIFY: Apply or edit grade-appropriate grammar usage and mechanics to clarify a message and edit narrative, informational, and argumentative texts DOK: 3 BELOW TESTED GRADE TOO HIGH Uncle Sam Uncle Sam has long been an iconic image symbolic of the United States of America. His lean frame, flowing ivory beard, top hat, and patriotic ensemble are known throughout the world. Some people adhere to the belief that a gentleman by the name of Samuel Wilson is responsible for Uncle Sam. Wilson lived in Troy New York, in the early 1800s and he ran an extremely successful business that delivered meat to the United States Army. The meat was delivered in barrels stamped with “U.S.” to indicate that they belonged to the United States government. Sam Wilson’s employees quipped to the soldiers that the emblem stood for “Uncle Sam.” The farce spread and soon, throughout the country, “Uncle Sam” became a synonym for “United States.” This paragraph has a sentence that contains errors. Which sentence needs to be edited? A. the last sentence B. the fourth sentence KEY C. the second sentence D. the sixth sentence HIGH READING LEVEL TOO CONFUSING Here is an example of a poorly written middle school item. Let’s pause for thirty seconds so that you can review this item and think about issues that may make this item flawed. {+} In this item, the reading level of the passage is above the tested grade. Claim 2 passages should be below the tested grade. The item is too simple for a level 3 depth of knowledge. The format of the stem and answer options is confusing and students will waste too much time searching for a mistake The answer options are not presented in a logical order. In an item like this the options should be presented in consecutive order. ILLOGICAL ORDER

Well Written Middle School Selected Response Example Item Grade: 6 Claim 2: Students can produce effective writing for a range of purpose and audiences. Target 9. EDIT/CLARIFY: Apply or edit grade-appropriate grammar usage and mechanics to clarify a message and edit narrative, informational, and argumentative texts DOK: 2 NEW DOK Uncle Sam Have you seen pictures of Uncle Sam? He is a skinny man with a long white beard and a top hat. He usually wears the colors of the American flag: red, white, and blue. He has been a symbol of the United States for about two hundred years. Some people think that there was actually a real Uncle Sam, named Samuel Wilson. He lived in Troy, New York, in the early 1800s, he ran a business that delivered meat to the United States Army. The meat was delivered in barrels stamped “U.S.” The “U.S.” stamp was to show that it belonged to the United States government. The people delivering the meat knew that it came from Sam Wilson. They joked to the soldiers that the “U.S.” on the meat barrels stood for “Uncle Sam.” The joke spread. Soon, all over the country, “Uncle Sam” became another way of saying “United States.” Select the correct way to revise the highlighted sentence. He lived in Troy, New York, in the early 1800s. There he had ran a business that delivered meat to the United States Army. He lived in Troy, New York, in the early 1800s he ran a business that delivers meat to the United States Army. He lived in Troy, New York in the early 1800s, and he ran a business that delivered meat to the United States Army. In the early 1800s, Sam Wilson lived in Troy, New York, and ran a business that delivered meat to the United States Army. CORRECT READABILITY CLEAR WORDING Here is an example of the same item revised to be a well written middle school selected-response item. Let’s pause for thirty seconds so that you can review this item and think about why it is a better item. {+} The stimulus text has been rewritten to be below grade level as required by claim 2 The depth of knowledge level has been changed to one that is more appropriate for the task. Finally, the format of the stem and options is clear and concise.

Flawed Middle School Constructed Response Example Item Grade: 7 Claim 1: Students can read closely and analytically to comprehend a range of increasingly complex literary and informational texts. Target 7. LANGUAGE USE: Interpret impact or intent of figurative language use (e.g., alliteration, onomatopoeia, imagery), literary devices (e.g., flashback, foreshadowing), or connotative meanings of words and phrases used in context and their impact on reader interpretation DOK: 2 DOES NOT ALIGN The Fox and the Goat A Fox fell into a well, and though it was not very deep, he found that he could not get out again. After he had been in the well a long time, a thirsty Goat came by. The Goat thought the Fox had gone down to drink, so he asked if the water was good. "The finest in the whole country," said the crafty Fox, "jump in and try it. There is more than enough for both of us." The thirsty Goat immediately jumped in and began to drink. The Fox just as quickly jumped on the Goat's back. Then he leaped from the tip of the Goat's horns out of the well. The foolish Goat now saw what he had gotten into. He begged the Fox to help him out. What does the word crafty suggest about the Fox? NOT COMPLEX TOO EASY This is an example of a flawed constructed response item for grade 7. Let’s pause for thirty seconds so that you can review this item and think about issues that may make this item flawed. {+} In this item the stimulus text is not appropriate for 7th grade students. Both the context and the readability are too far below grade level. The item does not require students to interpret figurative language and therefore does not align with the assessment target. The stem does not ask students to support their response with information from the stimulus or specify what action the student is supposed to take. The item is not complex and does not require analytical reasoning or thinking. This item is more appropriate as a selected response item than as a constructed response item. POOR CR WORDING

Well Written Middle School Constructed Response Example Item Grade: 7 Claim 1: Students can read closely and analytically to comprehend a range of increasingly complex literary and informational texts. Target 11. REASONING & EVIDENCE: Use supporting evidence to justify interpretations of information presented or how it is integrated (author’s reasoning; interactions among events, concepts, people, or development of ideas) DOK: 3 PROPER ALIGNMENT What is Mold? Mold is the common name for many kinds of tiny organisms called fungi. There are thousands of types of molds that can be found indoors and outdoors. Different molds grow in colonies. They live on dead organisms such as decaying plants and animals. They can also live on wood, brick, stone, food, fabric and books. Some molds even thrive on living organisms as parasites!   Spores from mold growth can pose health risks. Some people are very sensitive to mold. Molds may cause a runny nose or itchy, watery eyes. Or mold may cause major concerns such as difficulty breathing, asthma attacks, infections, fever and major skin irritations. The best way to reduce and prevent mold growth is to control moisture. It is important to keep humidity levels low. It is important to reduce excess moisture that molds need to grow. People should repair leaks, completely remove any existing mold growth, and ventilate rooms that are prone to damp conditions. Outside, molds may grow in damp, shaded areas. People who are sensitive to molds should avoid such places. Explain why people who live in moist climates work harder to prevent mold than people who live in dry climates. Include details from the passage about how they prevent mold. APPROPRIATE COMPLEXITY Here is an example of a well written constructed-response item for grade 7. Let’s pause for thirty seconds so that you can review this item and think about why it is a better item. {+} The stimulus and prompt are appropriately complex for the grade level. The prompt aligns well with the assessment target. The prompt is well written and requires a complex answer involving analytical reasoning and thinking. PROPER CR WORDING

Flawed High School Constructed Response Example Item Grade: 9 Claim 1: Students can read closely and analytically to comprehend a range of increasingly complex literary and informational texts. Target 6. TEXT STRUCTURES/FEATURES: Analyze text structures, genre-specific features, or formats (visual/graphic/auditory effects) of texts and the impact of those choices on meaning or presentation DOK: 3 Frost wrote this poem using blank verse (unrhymed lines of iambic pentameter). Using this poem as an example, describe the general effect of this poetic form and explain how it contributes to the meaning of this particular poem. Mending Wall by Robert Frost There where it is we do not need the wall: SOMETHING there is that doesn't love a wall, He is all pine and I am apple orchard. That sends the frozen-ground-swell under it, My apple trees will never get across And spills the upper boulders in the sun; And eat the cones under his pines, I tell him. And makes gaps even two can pass abreast. He only says, "Good fences make good neighbours." The work of hunters is another thing: Spring is the mischief in me, and I wonder I have come after them and made repair If I could put a notion in his head: Where they have left not one stone on a stone, "Why do they make good neighbours? Isn't it But they would have the rabbit out of hiding, Where there are cows? But here there are no cows. To please the yelping dogs. The gaps I mean, Before I built a wall I'd ask to know No one has seen them made or heard them made, What I was walling in or walling out, But at spring mending-time we find them there. And to whom I was like to give offence. I let my neighbour know beyond the hill; Something there is that doesn't love a wall, And on a day we meet to walk the line That wants it down." I could say "Elves" to him, And set the wall between us once again. But it's not elves exactly, and I'd rather We keep the wall between us as we go. He said it for himself. I see him there To each the boulders that have fallen to each. Bringing a stone grasped firmly by the top And some are loaves and some so nearly balls In each hand, like an old-stone savage armed. We have to use a spell to make them balance: He moves in darkness as it seems to me, "Stay where you are until our backs are turned!" Not of woods only and the shade of trees. We wear our fingers rough with handling them. He will not go behind his father's saying, Oh, just another kind of out-door game, And he likes having thought of it so well One on a side. It comes to little more: He says again, "Good fences make good neighbours." Here is an example of a flawed constructed response item written for grade 9. Let’s pause for thirty seconds so that you can review this item and think about issues that may make this item flawed. {+} This is an example of an item that is overly complex and therefore inappropriate for the designated grade level. Many college students would have difficulty answering this item and a full analysis of the prompt would result in a complete essay. In addition, the use of unnecessary jargon, such as “blank verse” and “iambic pentameter”, introduces bias and violates universal design principles. NOT UD COMPLIANT TOO COMPLEX

Well Written High School Constructed Response Example Item Grade: 11 Claim 1: Students can read closely and analytically to comprehend a range of increasingly complex literary and informational texts. Target 12. ANALYSIS WITHIN OR ACROSS TEXTS: Analyze texts to determine how connections are made in development of complex ideas or events; or in development of topics, themes, rhetorical features DOK: 4 APPROPRIATE FOR GRADE Passage 1 The following excerpt comes from a speech written in 1872 by women’s rights pioneer Susan B. Anthony. Anthony was arrested after attempting to vote in the 1872 presidential election. After her conviction Anthony wrote this speech to make a constitutional argument for giving women the right to vote.   Our democratic-republican government is based on the idea of the natural right of every individual member thereof to a voice and a vote in making and executing the laws. We assert the province of government to be to secure the people in the enjoyment of their inalienable right. We throw to the winds the old dogma that government can give rights. No one denies that before governments were organized each individual possessed the right to protect his own life, liberty and property. When 100 to 1,000,000 people enter into a free government, they do not barter away their natural rights; they simply pledge themselves to protect each other in the enjoyment of them through prescribed judicial and legislative tribunals. They agree to abandon the methods of brute force in the adjustment of their differences and adopt those of civilization . . .The Declaration of Independence, the United States Constitution, the constitutions of the several States and the organic laws of the Territories, all alike propose to protect the people in the exercise of their God-given rights. Not one of them pretends to bestow rights. All men are created equal, and endowed by their Creator with certain inalienable rights. Among these are life, liberty and the pursuit of happiness. To secure these, governments are instituted among men, deriving their just powers from the consent of the governed. Passage 2 The following excerpt comes from the Second Treatise of Government by John Locke, published in 1690. Sect. 22. The natural liberty of man is to be free from any superior power on earth, and not to be under the will or legislative authority of man, but to have only the law of nature for his rule. The liberty of man, in society, is to be under no other legislative power, but that established, by consent, in the commonwealth; nor under the dominion of any will, or restraint of any law, but what that legislative shall enact, according to the trust put in it. Freedom then is not what Sir Robert Filmer tells us, Observations, A. 55. a liberty for every one to do what he lists, to live as he pleases, and not to be tied by any laws: but freedom of men under government is, to have a standing rule to live by, common to every one of that society, and made by the legislative power erected in it; a liberty to follow my own will in all things, where the rule prescribes not; and not to be subject to the inconstant, uncertain, unknown, arbitrary will of another man: as freedom of nature is, to be under no other restraint but the law of nature. Identify the main idea of each passage and explain how Locke’s treatise supports Anthony’s argument. APPROPRIATELY COMPLEX TEXTS Now let’s take a look at an example of a well written high school constructed response item. Let’s pause for thirty seconds so that you can review this item and think about why it is a better item. {+} This item is rigorous but appropriately complex for 11th grade. The stimuli are grade-appropriate historical documents and the brief introduction to the first passage sets the historical context for students who may lack background knowledge on the topic. The prompt is clear and concise and requires a reasonable amount of analytical thinking. CLEAR PROMPT

Selected Response and Constructed Response Essential Elements Benefits and Limitations Faulty and Exemplary Items Continue with this module to learn about technology-enhanced items. Thus far, this module has explored several issues specific to selected response and constructed response items. {+} Specifically, this module examined the essential elements of each item type, including the concept of a stem, stimuli, options, and scoring information. This module also explored the benefits and limitations of each item type. Finally, the module explored several characteristics of faulty items and described elements of exemplary items. The information presented in this module should be used to guide the development of selected response and constructed response items that are specified by the task models for each assessment target measured by the Smarter Balanced Assessment System. For those participants who will be writing technology-enhanced items, please continue with this module. For those participants who will not be writing technology-enhanced items, you may end the module now.

Capitalize on Technology Technology-Enabled Technology-Enhanced In addition to developing traditional selected and constructed response items, Smarter Balanced aims to capitalize on technology to improve the measure of student learning. To this end, the Consortium aims to develop two types of items that capitalize on technology. {+} These items are known as Technology Enabled and Technology Enhanced.

Technology-Enabled Items Digital Media Video Animation Sound Response Types Selected Response Constructed Response Example: Listen to President Kennedy’s 1961 inaugural address and then write an essay analyzing metaphors used regarding foreign policy. Technology-enabled items use {+} digital media as the stimulus, but do not require specialized interactions to produce response. Possible stimuli for English Language Arts technology-enabled items could include short video clips, audio recordings of lectures or speeches, or dramatic readings of prose or poetry. {+} Despite the use of these media types, a technology enabled item requires a student to provide either a selected response or a constructed response that consists of text. An English language arts technology-enabled item might require students to listen to President Kennedy’s 1961 inaugural address and answer a constructed response item analyzing the metaphors he uses regarding foreign policy. Another possible item might have students view a brief video about the proper way to brush and floss your teeth and then write a summary explaining the steps. For ELA assessments, most technology enabled items will be part of performance tasks that use non-text stimuli and Claim 3 items that involve listening to and/or viewing a stimulus. Example: View video and write a summary explaining steps in a process.

Technology-Enhanced Items Specialized interaction May have digital media for stimulus Same requirements as selected and constructed response items Students manipulate information Defined responses Technology-enhanced items are computer delivered items that require {+} specialized interactions students must perform to produce a response. Responses produced by a technology-enhanced item require students to do something other than write text or select from among a set of options. These items may also include digital media as the stimulus. Technology-enhanced items should conform to the same essential requirements that have already been discussed for writing quality selected response and constructed response items. The only difference is that they allow students to manipulate information in ways that are not possible with traditional selected response and constructed response items. Like selected-response items, technology-enhanced items have defined responses that can be scored in an automated manner.

Technology-Enhanced Example Item Below is a poem, a sonnet, in which the speaker discusses her feelings about a relationship. Read the poem and answer the question that follows. Remember When you can no more hold me by the hand, Nor I half turn to go yet turning stay. Remember me when no more day by day You tell me of our future that you plann’d: Only remember me; you understand It will be late to counsel then or pray. Yet if you should forget me for a while And afterwards remember, do not grieve. For if the darkness and corruption leave A vestige* of the thoughts that once I had Better by far you should forget and smile Than that you should remember and be sad. This is an example of an ELA technology-enhanced item for grade 8. For this item, students begin by reading a poem. Next, students find and {+} highlight lines in the text that reveal a change in the speaker’s message. Depending on how the item writer designed the item, students are able to select one or more blocks of text in the poem. *vestige: a mark, trace, or visible evidence of something that is no longer present or evident. In the sonnet “Remember,” which two lines reveal a change in the speaker’s message to her subject?

Key Components of a Technology-Enhanced Item INTERACTION SPACE Remember When you can no more hold me by the hand, Nor I half turn to go yet turning stay. Remember me when no more day by day You tell me of our future that you plann’d: Only remember me; you understand It will be late to counsel then or pray. Yet if you should forget me for a while And afterwards remember, do not grieve. For if the darkness and corruption leave A vestige* of the thoughts that once I had Better by far you should forget and smile Than that you should remember and be sad. The notes for a summary need to be arranged correctly in the order in which the events occurred in the passage. Click on each sentence and move it to arrange the sentence into correct chronological order. Summary of Events: Maria laughs with the old women. The guest and family eat dinner. Maria’s mother asks the guests for a story. Maria’s guests arrive. Maria becomes sad. The guests take turn telling stories. There are several terms that are important to understand when developing technology enhanced items. The first term is Interaction Space. The interaction space is the area in which students interact with an item to produce a response. {+} This is the interaction spaces for an item that presents students with a poem and asks them to select blocks of text within the poem. The interaction space is the area in which students select text blocks. This is a different item that presented a passage and is now asking students to order sentences that describe events in the passage. The interaction space allows students to select the sentences and then change their order.

Key Components of a Technology-Enhanced Item Below is a poem, a sonnet, in which the speaker discusses her feelings about a relationship. Read the poem and answer the question that follows. Remember When you can no more hold me by the hand, Nor I half turn to go yet turning stay. Remember me when no more day by day You tell me of our future that you plann’d: Only remember me; you understand It will be late to counsel then or pray. Yet if you should forget me for a while And afterwards remember, do not grieve. For if the darkness and corruption leave A vestige* of the thoughts that once I had Better by far you should forget and smile Than that you should remember and be sad. *vestige: a mark, trace, or visible evidence of something that is no longer present or evident. In the sonnet “Remember,” which two lines reveals a change in the speaker’s message to her subject? When developing a technology-enhanced item, the task model will indicate which type of interaction space should be used to allow students to produce a response. {+} As an example, the task model for this item specifies the use of the “select text” interaction space. TYPE OF INTERACTION SPACE

Key Components of a Technology-Enhanced Item Parameter: A variable provides input to a computer program Below is a poem, a sonnet, in which the speaker discusses her feelings about a relationship. Read the poem and answer the question that follows. Remember When you can no more hold me by the hand, Nor I half turn to go yet turning stay. Remember me when no more day by day You tell me of our future that you plann’d: Only remember me; you understand It will be late to counsel then or pray. Yet if you should forget me for a while And afterwards remember, do not grieve. For if the darkness and corruption leave A vestige* of the thoughts that once I had Better by far you should forget and smile Than that you should remember and be sad. *vestige: a mark, trace, or visible evidence of something that is no longer present or evident. In the sonnet “Remember,” which two lines reveals a change in the speaker’s message to her subject? Interaction Space Parameters Block Size: Custom blocks Valid Blocks: All Limit Selection: True Max Selections: 1 Interaction Space Parameters Block Size: Whole word Valid Blocks: All Limit Selection: False Max Selections: NA Below is a poem, a sonnet, in which the speaker discusses her feelings about a relationship. Read the poem and answer the question that follows. Remember When you can no more hold me by the hand, Nor I half turn to go yet turning stay. Remember me when no ore day by day You tell me of our future that you plann’d: Only remember me; you understand It will be late to counsel then or pray. Yet if you should forget me for a while And afterwards remember, do not grieve. For if the darkness and corruption leave A vestige* of the thoughts that once I had Better by far you should forget and smile Than that you should remember and be sad. *vestige: a mark, trace, or visible evidence of something that is no longer present or evident. Select all verbs that suggest the author is no longer alive. A parameter is a variable that allows a user to provide input to a computer program. As an example, when printing a document, the user is asked how many copies to print. The number entered for this parameter is used to tell the computer how many times to repeat the print function or routine for the document. Many interaction spaces contain parameters that allow an item writer to specify additional information like the number of responses the student can create or specific characteristics of the response. {+} As an example, the select text interaction space allows an item writer to define the size of the blocks of text that are to be selected. The text block size can be defined as whole words, whole sentences, whole paragraphs, or custom defined blocks. {+} For this item, custom blocks have been defined. A second parameter, named Limit Selection, allows the item author to indicate whether there is a limit to the number of selections that a student may make. In this example, the number of selections is limited. A related parameter allows the item writer to then indicate what the limit is. In this case, the limit is 1 selection. {+} By adjusting the values for each interaction space parameter, a single technology-enhanced item type can be used to create a wide variety of items. As an example, for this second item the same poem is used as the stimulus, but the block size is set to whole word and limit selection is set to false, which allows the student to make as many selections as he or she feels is necessary.

Scoring Rule: Logic used to score student response Who is the protagonist in the story? A. Joe B. Sue C. Maria D. The father Scoring Rule: If student response = C, then correct Otherwise, incorrect Scoring Rule: If object 1 = B, 2 = A, 3 = B, 4 = A, 5 = A, then correct Otherwise, incorrect Classify each word below based on whether it is a verb or a noun. Verbs Nouns Doll Run Dog Swim Eat All technology enhanced items have a scoring rule associated with them. {+} The scoring rule provides directions to the computer about the logic to be followed to score a response. In some cases, an item will have only one correct response. In other cases, there may be many possible correct responses. {+} In reality, all items have a scoring rule. The scoring rule for a selected response item simply states that if the student’s response is identical to the correct response, then the student receives one point, otherwise the student receives zero points. {+} Similarly, for a drag and drop item, the scoring rule compares the location of each object with its correct location. The student is then awarded one or more points based on the number of correct categorizations made.

Boolean: A variable that has a value of True or False Boolean Variables Boolean: A variable that has a value of True or False Below is a poem, a sonnet, in which the speaker discusses her feelings about a relationship. Read the poem and answer the question that follows. Remember When you can no more hold me by the hand, Nor I half turn to go yet turning stay. Remember me when no more day by day You tell me of our future that you plann’d: Only remember me; you understand It will be late to counsel then or pray. Yet if you should forget me for a while And afterwards remember, do not grieve. For if the darkness and corruption leave A vestige* of the thoughts that once I had Better by far you should forget and smile Than that you should remember and be sad. *vestige: a mark, trace, or visible evidence of something that is no longer present or evident. In the sonnet “Remember,” which two lines reveals a change in the speaker’s message to her subject? Interaction Space Parameters Block Size: Custom blocks Valid Blocks: All Limit Selection: True Max Selections: 1 Interaction Space Parameters Block Size: Whole word Valid Blocks: All Limit Selection: False Max Selections: NA Below is a poem, a sonnet, in which the speaker discusses her feelings about a relationship. Read the poem and answer the question that follows. Remember When you can no more hold me by the hand, Nor I half turn to go yet turning stay. Remember me when no ore day by day You tell me of our future that you plann’d: Only remember me; you understand It will be late to counsel then or pray. Yet if you should forget me for a while And afterwards remember, do not grieve. For if the darkness and corruption leave A vestige* of the thoughts that once I had Better by far you should forget and smile Than that you should remember and be sad. *vestige: a mark, trace, or visible evidence of something that is no longer present or evident. Select all verbs that suggest the author is no longer alive. Many technology enhanced items require an item writer to set a value for a special type of variable called a Boolean. {+} A Boolean is a variable that has one of two possible values, True or False. Many technology enhanced items use a Boolean to indicate whether or not a specific feature should be turned on or off. {+} As an example, the select text item type allows the item writer to indicate whether or not there is a limit to the number of selections a student can make. {+} In most cases, Boolean variables are used to turn a feature on or off and allow for the production of a wider variety of items from a given technology enhanced item type.

Technology-Enhanced Item Types Common English Language Arts Technology- Enhanced item types Dropdowns Classification Reorder text Select and order Select text Below is a poem, a sonnet, in which the speaker discusses her feelings about a relationship. Read the poem and answer the question that follows. Remember When you can no more hold me by the hand, Nor I half turn to go yet turning stay. Remember me when no more day by day You tell me of our future that you plann’d: Only remember me; you understand It will be late to counsel then or pray. Yet if you should forget me for a while And afterwards remember, do not grieve. For if the darkness and corruption leave A vestige* of the thoughts that once I had Better by far you should forget and smile Than that you should remember and be sad. *vestige: a mark, trace, or visible evidence of something that is no longer present or evident. In the sonnet “Remember,” which two lines reveals a change in the speaker’s message to her subject? The notes for a summary need to be arranged correctly in the order in which the events occurred in the passage. Click on each sentence and move it to arrange the sentence into correct chronological order. Summary of Events: Maria laughs with the old women. The guest and family eat dinner. Maria’s mother asks the guests for a story. Maria’s guests arrive. Maria becomes sad. The guests take turn telling stories. Classify each word below based on whether it is a verb or a noun. Verbs Nouns Doll Run Dog Swim Eat There are many different types of technology-enhanced item types that can be used to write an item. {+} Most ELA items will use one of the following types; dropdowns, classification, reorder text, select and order, and select text. As just three examples, the TEI poem item discussed earlier is an example of a select text item. Students selected a block of text from a poem as their answer. The item that asked students to order events in a story is an example of a reorder text item. And the item asking students to classify words as verbs or nouns is an example of a classification item. Additional examples are available in the Technology-Enhanced Item specifications.

Technology-Enhanced Item Template Select Type of Interaction Below is a poem, a sonnet, in which the speaker discusses her feelings about a relationship. Read the poem and answer the question that follows. Remember When you can no more hold me by the hand, Nor I half turn to go yet turning stay. Remember me when no more day by day You tell me of our future that you plann’d: Only remember me; you understand It will be late to counsel then or pray. Yet if you should forget me for a while And afterwards remember, do not grieve. For if the darkness and corruption leave A vestige* of the thoughts that once I had Better by far you should forget and smile Than that you should remember and be sad. *vestige: a mark, trace, or visible evidence of something that is no longer present or evident. In the sonnet “Remember,” which two lines reveals a change in the speaker’s message to her subject? Enter Content for Item Define Parameter Values Interaction Space Parameters Block Size: Custom blocks Valid Blocks: All Limit Selection: True Max Selections: 1 To guide the development of technology enhanced items, a template has been created for each type of technology enhanced item. The template for each item type is divided into four steps. {+} The first step in creating an item is to select the type of interaction the student must make to produce a response. The type of interaction selected will determine the template that will be used to produce an item. {+} As an example, to create this item the item writer selected the Select Text interaction type. {+} Once a template has been selected, the content that forms the prompt and other directions for the item can be entered in the content section of the template. {+} As an example, for the prompt of this item, the item writer entered “In the sonnet “Remember,” which lines reveal a change in the speaker’s message to her subject?” {+} The third step focuses on setting values for the variables associated with the interaction type selected. {+} As was described earlier, for this item the item writer indicated that the size of text blocks that students could select was defined as custom blocks. In addition, the item writer allowed all custom blocks to be selectable, but limited the number of selections to one. {+} The final component allows an item writer to enter information about how to score a student’s response. This information may include the number of points awarded for the item, and the responses for each point value. {+} As an example, for this item, the item writer indicated that text block number 4 is the correct response and the value for a correct response is one point. A later module will introduce the item writing system that will be used to create and record information for each of these four components of a technology enhanced item template. Specify Scoring Information Scoring Rule: Correct Block: 4 Correct Value: 1 Incorrect Value: 0

Technology-Enhanced Items Essential elements Steps required to write an item Type of item templates This portion of the module has explored several issues specific to technology-enhanced items. {+} Specifically, this portion of the module examined the essential elements of technology-enhanced items, including the concept of an interaction space, parameter values, and scoring rules. This module also explored the steps required to write a technology-enhanced item. Finally, the module described the types of technology-enhanced items that are currently available for development. To learn more about technology-enhanced items, please see the Technology-Enhanced Item Specifications developed by the Smarter Balanced Assessment Consortium.