1. Introduction to Evidence-Centered Design
Modern approach to designing items and tasks
High quality assessment requires careful and thoughtful development of items and performance tasks. The most important characteristic of a quality item or task is its ability to elicit evidence from a student that can support decisions about their learning. This module will examine Evidence-Centered Design and describes how it is used to guide the development of the Smarter Balanced Assessment System.

2. Traditional Item Development versus Evidence-Centered Design
Introduction to Evidence-Centered Design Modern Approach to Designing Items and Tasks
Traditional Item Development
versus
Evidence-Centered Design
Keys to Evidence- Centered Design
Claims
Assessment Targets
Task Models
Evidence
To guide the development of high quality items and tasks, the Smarter Balanced Assessment Consortium is committed to using Evidence-Centered Design to guide the development of its assessment system.
{+} This module will explore several concepts that provide a foundation for Evidence-Centered Design. It will begin with contrasting a traditional approach of developing items and tasks to an Evidenced-Centered approach. It will also present the following concepts: {+} claims about student learning, {+} assessment targets, {+} evidence of student learning, and {+} task models. {+}
This presentation will explore how these concepts are used to design high-quality items and performance tasks. Let’s begin by examining a traditional approach to item development and then contrast it with an Evidence-Centered Design approach.

3. Traditional Approach to Item Development
1.2.1 Apply reference skills to determine word meanings.
1.2.2 Apply vocabulary strategies in grade level text.
1.3.1 Understand and apply new vocabulary.
1.3.2 Understand and apply content/academic vocabulary.
1.4.1 Know common sight words appropriate to grade-level.
1.4.2 Apply fluency to enhance comprehension.
1.4.3 Apply different reading rates to match text.
2.1.1 Understand how to use questioning when reading.
2.1.2 Understand how to create mental imagery.
2.1.3 Understand and determine important or main ideas and important details in text.
2.1.4 Apply comprehension monitoring strategies before, during, and after reading: use prior knowledge/schema.
2.1.5 Apply comprehension monitoring strategies before, during, and after reading: predict and infer.
2.1.6 Apply comprehension monitoring strategies before, during, and after reading: monitor for meaning, create mental images.
2.1.7 Apply comprehension monitoring strategies during and after reading: summarize informational/expository text and literary/narrative text.
Educators have been creating items and tasks for decades using a traditional approach to item development that measures learning outcomes. For state testing programs, desired learning outcomes are often called Content Standards. Content Standards define what students are expected to know and be able to do.
When developing a test to measure achievement of content standards using a traditional approach, emphasis is placed on adequately sampling the full range of standards.

4. Traditional Approach to Item Development
1.2.2
1.3.2
2.1.3
2.1.5
1.2.2 Apply vocabulary strategies in grade level text.
1.3.2 Understand and apply content/academic vocabulary.
2.1.3 Understand and determine important or main ideas and important details in text.
2.1.5 Apply comprehension monitoring strategies before, during, and after reading: predict and infer.
2.1.7 Apply comprehension monitoring strategies during and after reading: summarize informational/expository text and literary/narrative text.
1.2.1 Apply reference skills to determine word meanings.
1.3.1 Understand and apply new vocabulary.
1.4.1 Know common sight words appropriate to grade-level.
1.4.2 Apply fluency to enhance comprehension.
1.4.3 Apply different reading rates to match text.
2.1.1 Understand how to use questioning when reading.
2.1.2 Understand how to create mental imagery.
2.1.4 Apply comprehension monitoring strategies before, during, and after reading: use prior knowledge/schema.
2.1.6 Apply comprehension monitoring strategies before, during, and after reading: monitor for meaning, create mental images.
To sample the standards, {+} a small number of items or tasks were developed to measure a sub-set of the content standards.

5. Traditional Approach to Item Development
1.2.2
1.3.2
2.1.3
2.1.5
2.1.7
One characteristic of a good test was the extent to which the items on the test represented the full range of standards {+} and the extent to which each item appeared to measure a single standard. The primary focus was on the alignment between the content of the item and the content standard.
While alignment is important, there is a key weakness in this traditional approach: While the content of an item might be aligned with a content standard, the information elicited by the item may not say anything meaningful about whether or not the student has achieved the standard.

6. Traditional Approach to Item Development
Content Standard 2.2.3: Perform addition accurately for single and two digit numbers.
Item:
Beth says that = 6.
Explain why Beth is correct.
As an example, the content of this item appears aligned with a content standard that focuses on addition. The information elicited from the student’s response, however, does not say much about a student’s ability to add. This happens, in part, because the item requires the student to describe in words how he or she knows two plus four is equivalent to six. In this example, the student may be able to add the two digits, but he or she may have difficulty expressing his or her understanding in writing.

7. Evidence-Centered Design
Item/Task
Evidence
Assessment Target
Claim
Content Standard
Evidence-Centered Design corrects this shortcoming. While alignment between standards and items or tasks remains important, strong emphasis is also placed on ensuring that
{+} each item and task {+} elicits evidence {+} about the target of assessment {+}
that can be used to support a claim about students’ development of the knowledge, skill, and/or ability {+} contained in a content standard. Central to Evidence-Centered Design is the idea of collecting evidence through a student’s response to an item or task that supports a claim about the extent to which a student has developed the knowledge, skill, and ability that is contained in a content standard.

8. Applying Evidence-Centered Design to Item and Task Development
Beth says that = 6.
Explain why Beth is correct.
Content Standard 2.2.3: Perform addition accurately for single and two digit numbers.
Weak Evidence
As an example, let’s return to the item that asks students to explain how they know two plus four equals six. As shown a few moments ago, this item appears aligned with the content standard because it focuses on addition. {+} However, the evidence elicited by the item may not allow a claim to be made about whether or not the student can actually add. In order to provide appropriate evidence about a student’s ability to add, the item must be redesigned.

9. Applying Evidence-Centered Design to Item and Task Development
2 + 4 = ____
Content Standard 2.2.3: Perform addition accurately for single and two digit numbers.
Stronger Evidence
One possible redesign is to simply have the student construct a response that indicates the result of adding two digits.
{+}
To be clear, an Evidence-Centered Design approach does not suggest that items and tasks should avoid having students explain their reasoning. Rather, an Evidence-Centered Design approach aims to ensure that the information elicited by an item is appropriate for the claims Smarter Balanced seeks to make about the achievement of content standards.

10. Applying Evidence-Centered Design to Item and Task Development
Beth says that = 6.
Explain why Beth is correct.
2 + 4 = ____
Content Standard 2.2.4: Perform mathematical operations and justify solutions.
Content Standard 2.2.3: Perform addition accurately for single and two digit numbers.
In this example, the original item would provide {+} appropriate evidence for a content standard that focuses on {+} a student’s ability to perform mathematical operations and justify his or her solutions. {+} The revised version elicits appropriate evidence about a student’s ability to add.

11. 6 Key Concepts of Evidence-Centered Design
1. Define the domain
Common Core Standards Math/ELA
2. Define claims to be made
4 ELA & 4 Math Claims Content Specifications
The process of designing items and tasks using an Evidence-Centered Design approach involves six key steps.
{+} The first step focuses on clearly defining the content area or domain that is to be measured. For Smarter Balanced, the content areas to be assessed are defined by the Common Core State Standards for Mathematics and the Common Core State Standards for English Language Arts.
{+} Once the domains to be measured are defined, the next step is to define the claims that will be made about the domains. A claim is a statement about what a student knows or is able to do. After carefully analyzing the Common Core State Standards and thinking about what students must know and be able to do in order to be prepared for college and career paths, Smarter Balanced has identified four claims specific to English Language Arts and four claims for mathematics that focus on what students are expected to be able to do at each grade level.
Let’s take a moment to look at an example of a claim.

12. English Language Arts Literacy Claim #1
Example of a Claim
English Language Arts Literacy Claim #1
Students can read closely and analytically to comprehend a range of increasingly complex literary and informational texts.
As an example, one English Language Arts claim focuses on student’s ability to read closely and analytically to comprehend literary and informational texts. The full set of claims are found in the Content Specifications and will be examined in greater detail in a future module that focuses on these specifications.

13. 6 Key Concepts of Evidence-Centered Design
1. Define the domain
Common Core Standards Math/ELA
2. Define claims to be made
4 ELA & 4 Math Claims Content Specifications
3. Define assessment targets
Knowledge, Skills, & Abilities
Once the domain is defined and the claims to be made are defined, {+} the third step is to clearly define the knowledge, skills, and abilities that form the domain. For Smarter Balanced, the knowledge, skills, and abilities that are intended to be measured within each domain are called “assessment targets”. An assessment target defines the specific knowledge, skill, or ability that students should be able to demonstrate within the domain. The content specifications define a large number of assessment targets that will be measured by the Smarter Balanced assessment system.

14. Example of an Assessment Target
Grade 11 – Assessment Target
Analyze the figurative (e.g., euphemism, oxymoron, hyperbole, paradox) or connotative meanings of words and phrases used in context and the impact of these word choices on meaning and tone.
As an example, one English Language Arts assessment target focuses on the analysis of figurative and connotative meanings of words and phrases and the impact of word choice on meaning and tone. Assessment targets were developed by carefully examining the Common Core State Standards and identifying the specific knowledge, skills, and abilities that form the standards. Assessment targets will be examined in greater detail in a separate module that focuses on the Content Specifications.

15. 6 Key Concepts of Evidence-Centered Design
1. Define the domain
Common Core Standards Math/ELA
2. Define claims to be made
4 ELA & 4 Math Claims Content Specifications
3. Define assessment targets
Knowledge, Skills, & Abilities
4. Define evidence required
Evidence to be Elicited from Student
Once assessment targets are defined, {+} the fourth step focuses on identifying the types of information that need to be collected from students to allow educators to say something meaningful about the student’s achievement of the assessment targets. The information Smarter Balanced aims to elicit from students is considered to be evidence that can be used to support or refute a claim about the student’s achievement of the assessment target.

16. Example of an Evidence Description
Evidence for Assessment Target 1
When reading informational or argumentative texts, students analyze the figurative or implied meanings of words or phrases as they are used in a text and analyze how the choice of these particular words affects meaning and tone.
As an example, for the English Language Arts grade eleven assessment target, one type of evidence comes from student responses in which they analyze the figurative and implied meanings of words or phrases and the effects that word choice has on meaning and tone. The evidence required for each assessment target is defined in the Item and Task Specifications. A separate module will explore evidence required to support claims for mathematics and for English language arts.

17. 6 Key Concepts of Evidence-Centered Design
1. Define the domain
Common Core Standards Math/ELA
2. Define claims to be made
4 ELA & 4 Math Claims Content Specifications
3. Define assessment targets
Knowledge, Skills, & Abilities
4. Define evidence required
Evidence to be Elicited from Student
Once types of evidence to collect are known, {+} the next step focuses on describing the characteristics of items and tasks that will elicit that evidence. The characteristics of an item or task are presented as a task model. A task model is a general description of an item or task. A task model can be thought of as a set of instructions that can be used to develop different versions of an item or task. What is important about a task model is that it is designed to collect evidence about the knowledge, skill, or ability that is contained in the standards.
5. Develop Task Models
Methods for Eliciting Evidence

18. Example of a Task Model Task Model 1
A constructed response for which the student is prompted to identify an example of figurative language, explain the meaning, and describe how it affects meaning and tone. Stimulus text should be on grade level.
A task model is a narrative description of key features of an item or task. Each task model must be designed to elicit a specific piece of evidence about an assessment target through the student’s response. From one task model, several items can be created that share key features and allow for eliciting similar evidence but in different ways.
As an example, this task model describes a situation in which students are presented with a text passage that is on grade level and are prompted to construct a response through which they identify an example of figurative language, explain the meaning of that language, and describe how the language affects meaning and tone. There will be more about task models in modules that focus on specific item types and item specifications.

19. 6 Key Concepts of Evidence-Centered Design
1. Define the domain
Common Core Standards Math/ELA
2. Define claims to be made
4 ELA & 4 Math Claims Content Specifications
3. Define assessment targets
Knowledge, Skills, & Abilities
4. Define evidence required
Evidence to be Elicited from Student
The final step {+} is to develop one or more items based on the task model. A task model might be thought of as a parent from which many items are developed, with common characteristics across these items.
5. Develop Task Models
Methods for Eliciting Evidence
6. Develop Items or Performance Tasks

20. Building a Logical Argument
Common Core State Standards
Claim
Assessment Target
Evidence
When applied properly, the use of Evidence-Centered Design to guide item development allows a logical argument to be built about student learning based on information provided through responses to carefully designed items and performance tasks. This argument takes the following form: {+} The student’s response to the item or task {+} provides evidence {+} about the assessment target {+} that supports a claim about the student’s development of the knowledge, skill, or ability {+} that is at the heart of the standards. Based on this evidence, a statement can be made about whether or not the student has achieved the standards.
Student Response

21. 6 Key Components of Evidence-Centered Design
1. Define the domain
Common Core Standards Math/ELA
2. Define claims to be made
4 ELA & 4 Math Claims Content Specifications
3. Define assessment targets
Knowledge, Skills, & Abilities
4. Define evidence required
Evidence to be Elicited from Student
To review, there are six steps in the Evidence-Centered Design process. {+] Step one is defining the domain, or identifying what is to be measured. For Smarter Balanced, the domains of interest include the Common Core State Standards for Mathematics and English Language arts. {+} Step two is establishing the claims to be made about the achievement of standards. {+} Step three is identifying the assessment targets. These are the knowledge, skilsl, and abilities that make up the Mathematics and English Language Arts domains. {+} Step four is determining the information that will provide evidence that a student has achieved the assessment targets {+} Step five is creating a task model that describes the characteristics of items and tasks that elicit evidence about the achievement of an assessment target. In essence, a task model is a set of instructions on how to build items and tasks. {+} Finally, step six is using the instructions specified in the task model to write items and tasks.
5. Develop Task Models
Methods for Eliciting Evidence
6. Develop Items or Performance Tasks

22. Examining an Item Through the Lens of Evidence-Centered Design
Use the line buttons to separate the rectangle into 6 equal sections. Then click on the sections to shade the area of the rectangle that represents 2/3 of the area of the whole rectangle.
Assessment Target:
Develop understanding of fractions as numbers.
Claim:
Students can explain and apply mathematical concepts and carry out mathematical procedures with fluency.
Evidence:
Student can create visual representations of fractions that range from one-half to four-fifths.
Let’s take a look at an item to explore how the concept of Evidence-Centered Design informs thinking about the item. {+} Here is an item that asks students to partition a rectangle into six parts and highlight the equivalent of two-thirds. To evaluate the quality of this item, the first step is to consider whether the item reflects its task model.
{+} The task model specifies the following item features: student is presented with quadrilaterals and is asked to partition the shapes into up to 20 sections and to highlight fractions that represent one-half, one or two thirds, one two or three fourths, or one two three or four fifths. Given this task model, it is clear that the item contains the features specified by the model. It is also clear that the task model is designed to produce evidence that the student can create visual representations of fractions that range from one-half to four-fifths.
{+} Finally, the evidence is intended to support a claim that a student can explain and apply mathematical concepts and carry out mathematical procedures with fluency to demonstrate an understanding of fractions as numbers. Given the alignment between the claim about the assessment target, the evidence required, the task model, and finally the item, it can be said that this item is designed appropriately to collect evidence from students that will support the intended claim. For students who produce a response that accurately partitions the quadrilateral into six parts and highlights four sections, the item provides evidence that supports the claim. For students who produce an inaccurate response, the item provides evidence that supports a claim that the student has not yet developed full understanding of fractions.
Task Model:
Partition tool is used to divide a given quadrilateral into up to 20 equal sections and to highlight fractions of the quadrilateral that represent halves, thirds, quarters, or fifths.

23. Using Evidence-Centered Design to Guide Item Design
What evidence is required given the assessment target I am measuring?
What are the key features that must be included in the item?
Will this item allow for the production of the evidence I am seeking?
Is there anything about this item that may make it more difficult to collect evidence from some students?
Evidence-Centered Design is a powerful tool for informing item development to help ensure that evidence is being collected about the claims made through an assessment. As an item is developed and reviewed, there are a few key questions to ask. {+} First, it is important to understand the assessment target being measured and to recognize the evidence that is to be collected through the item. {+} Second, by examining the task model, it is important to understand the key features that must appear in the item being developed or reviewed. Once the evidence being sought and the features of an item that help create the context in which that evidence can be collected are understood, a new item can be developed or an existing item can be reviewed. {+} Once an item is developed, it is important to revisit whether the resulting item is likely to produce the evidence being elicited. This is done by reviewing the task model and comparing the key features with the item of interest. If the item is not likely to produce the evidence of interest, then the item is in need of revision. {+} Finally, once it is determined that the item is well suited for eliciting the evidence sought, it is important to ask whether there are any features of an item, like the name of a person, an object, or situation that may be unfamiliar to some students or may otherwise make it difficult to collect accurate evidence from some students. For example, if an essay question asks students to describe what they did during a snow day, any student who has not experienced a snow day may have difficulty answering, thereby making it difficult to collect evidence about his or her ability to write. Any features of an item that may be unfamiliar to some students can create bias and, therefore, are in need of further revision. If the item is bias-free, it is ready to be shared with others.

24. Review of Key Concepts Standard Claim Evidence Task Model Item
This module demonstrated that Evidence-Centered Design is a process that helps item writers and reviewers focus on the evidence required to support a claim about an assessment target, and that this information is used as a guide to designing items that are likely to provide this evidence.
It also included information about the relationship between standards, claims about assessment targets, evidence, task models, and finally items. There will be more about claims, assessment targets, evidence, and task models in future modules that focus on the Mathematics and English Language Arts item and performance task specifications.