1. An Introduction to Writing 3D-Science Assessment Entities Dr. Joshua B
An Introduction to Writing 3D-Science Assessment Entities Dr. Joshua B. Smith American Institutes for Research

2. Items and Alignment
Test forms (the sequence of test questions (items) that are seen by a given student) must comply with a blueprint (the distribution of test content and other constraints (e.g., item formats; DOK requirements) that are desired by the entity giving the test).
Test blueprints for any given state’s high stakes summative tests are usually published documents that are often legislatively mandated.

3. Form Blueprints DECAS Science K-5 Adaptive Blueprint to Strand
MC (1pt)
MSCR (1 - 4 pts)
Items on Test
Min
Max
Total Test Form 42 48 2 50
Draft: 17 May 2010
Summary of Reporting Category 54 6 60
Reporting Category: Physical Science
DOK Minimums (to MC item min): DOK 1 = 5; DOK 2 = 5; DOK 3 = 3
15 content statements (50% = 7); 12 (80%) are E 15 19 21
5 strands (4 are E)
Standard 2: Materials and Their Properties
Strand 1: Properties and structure of materials (E-2) 13 1
Strand 2: Mixtures and solutions (E-1)
Strand 3: Conservation of matter (E-3)
Standard 3: Energy and Its Effects
Strand 1: Forms and sources of energy (I-3)
Strand 2: Forces and the transfer of energy (E-1)

4. Items and Alignment
To create a test form that complies with a blueprint, the items on that form need to have an alignment.
The items must be written such that they address a particular aspect of the science requirements that appear on the blueprint.
In order to align items, we take guidance from published content standards that outline the specific pieces (or aspects) of science that are intended to be taught and assessed.

5. Items and Alignment
Most standards have been traditionally organized into a nested hierarchy of increasing specificity.
In this Delaware Science example, we have:
a. Reporting Category (Earth & Space Science)
b. Standard (4: Earth in Space)
c. Strand (1: The Earth/Moon/Sun System)
d. Content Statement (omitted from this blueprint)
Reporting Category: Earth and Space Science
DOK Minimums (to MC item min): DOK 1 = 4; DOK 2 = 4; DOK 3 = 2
MC (1pt)
MSCR (1 - 4 pts)
Items on Test
12 content statements (50% = 6); 9 (75%) are E
Min
Max
4 strands (2 are E) 10 14 2 16
Standard 4: Earth in Space
Strand 1: The Earth/Moon/Sun systems (E-2) 9 1
Strand 2: The Solar System (I-3)

6. Items and Alignment
Standards are usually pretty straightforward and by themselves provide much of what we need to craft an aligned item.

7. 3-D Science Standards are rather more complex

8. Three-Dimensional Science Standards
The folks who developed the framework behind the 3-D standards recognized that science aligns with the same basic philosophy regardless of discipline, and that the work of “doing science” refers to a finite number of activities and concepts/themes that, like the philosophy of science itself, cross disciplines.

9. Three-Dimensional Science Standards
Science and Engineering Practice (SEP): the activities
Scientific and Engineering Practices
1. Asking questions (for science) and defining problems (for engineering)
2. Developing and using models
3. Planning and carrying out investigations
4. Analyzing and interpreting data
5. Using mathematics and computational thinking
6. Constructing explanations (for science) and designing solutions (for engineering)
7. Engaging in argument from evidence
8. Obtaining, evaluating, and communicating information

10. Three-Dimensional Science Standards
3-D standards, therefore, focus on the “doing” of science more than on what we think of as traditional science content - i.e., the observations of the world/universe at large and the hypotheses and theories that we devise to explain those observations, which is what 1-D standards usually focus on.

11. Three-Dimensional Science Standards
The philosophy of instruction based on 3-D standards is that if we can provide students with some basic “tools” of how to do some “actual science” tasks (using specific science content as examples), then students should be able to use those tools to help better understand nature (essentially stuff they see in their daily travels).

12. How we Apply the Practices

13. Three-Dimensional Science Standards
But of course we cannot just have the practices (the activities of science) alone.
We cannot consider someone scientifically literate if they spend all of their time thinking about the process of making observations but ignore the observations themselves.

14. Three-Dimensional Science Standards
Disciplinary Core Ideas (DCIs): big ideas
Physical Sciences
PS1: Matter and its interactions
PS2: Motion and stability: Forces and interactions
PS3: Energy
PS4: Waves and their applications in technologies for information transfer
Life Sciences
LS1: From molecules to organisms: Structures and processes
LS2: Ecosystems: Interactions, energy, and dynamics
LS3: Heredity: Inheritance and variation of traits
LS4: Biological evolution: Unity and diversity
Earth and Space Sciences
ESS1: Earth’s place in the universe
ESS2: Earth’s systems
ESS3: Earth and human activity
Engineering, Technology, and Applications of Science
ETS1: Engineering design
ETS2: Links among engineering, technology, science, and society

15. Three-Dimensional Science Standards
So the 3-D standards work to weave together these activities of making/understanding observations about the natural world with the observations themselves (or at least with specific observations that were determined to be good examples to use for the science practice in question).

16. Three-Dimensional Science Standards
This really isn’t a new concept – 1-D legacy standards have often included science process strands within the overall package of covered content – but for a number of reasons, application of those strands has rarely resulted in a smooth weave of science process with science content, especially on assessments.

17. Three-Dimensional Science Standards
In this spirit of trying to achieve that seamless weave then, each 3-D “standard” is a blend of one or two ”big ideas” from a science discipline, one of these scientific activities that are common to the doing of all sciences, and one of a number of broad themes that are found to cross scientific disciplinary boundaries.

18. Three-Dimensional Science Standards
Cross-Cutting Concepts (CCCs): the themes
Crosscutting Concepts
1. Patterns
2. Cause and effect: Mechanism and explanation
3. Scale, proportion, and quantity
4. Systems and system models
5. Energy and matter: Flows, cycles, and conservation
6. Structure and function
7. Stability and change

19. Aligning to the CCC

20. Recap: Legacy Standards
Legacy standards are one-dimensional.
They usually address facts or process (you have to dually-align items to address both).
Most often these standards address facts.

21. Recap: Three-D Standards
3-D standards address facts and process at the same time.
Each 3-D “standard” is built from one or two ”big ideas” from a science discipline, one of these scientific activities that are common to the doing of all sciences, and one of a number of broad themes that are found to cross scientific disciplinary boundaries. .

22. Three-Dimensional Science Standards
You cannot divide up a 3-D standard into its components, teach and/or assess them separately, and truthfully assert that what you are doing is aligned to that 3-D standard.
The intent is for the standard to be dealt with as a unit.
So, any classroom lesson or test question that is successfully built around a 3-D standard is effectively dually-aligned to science process and content.

23. The Structure of the Standards
Because of this weave of three dimensions to generate each standard, and because the central dimension is the practice, each 3-D “standard” is really a statement that describes an expectation of performance.
These Performance Expectations (PEs) state what a student should know and be able to do at the end of a period of instruction, related to the blend of dimensions that make up that particular PE.

24. Performance Expectation:
what students should know and be able to do at the end of the period of instruction
Dimensions/Foundations
Each PE represents the integration of three dimensions of science education: scientific and engineering practices (SEPs), disciplinary core ideas (DCIs), and crosscutting concepts (CCCs)

25. The Structure of the Standards
A set of Evidence Statements have been written that attempt to pull out and organize, from the blended instruction of a PE’s dimensions, the specific pieces of observable evidence that could be assessed or scored.

26. Evidence Statements:
should describe observable evidence that a scorer or assessor could actually see and measure
If you write items that cover each numbered evidence statement (i.e., 1.a, 2.a, and 3.a and/or 3.b), you should find that you’ve sufficiently covered the dimensions and thus the performance expectation

27. AIR 3-D – aligned entities
We are not writing to the evidence statements.
We are writing to Task Demands, which are derived from the PE and/or the evidence statements.
Task demands are designed to provide more concrete guidance for an author than is provided by either the PE or the evidence statements, aiding them in their task of crafting properly aligned task statements in interactions.

28. AIR Task Demands Task Demands
Make simple calculations using given data to calculate or estimate the total weight of a substance after heating, cooling, or mixing.
Measure or graph data that can be used to calculate or estimate the total weight of a substance after heating, cooling, or mixing.
Describe and/or summarize data (e.g., using illustrations and/or labels) to identify/highlight trends, patterns, or correlations concerning the weight of the substances being investigated at the beginning and end of an investigation.
Compile and/or select, from given information, the particular data needed for a specific inference about the total weight of substances. This can include sorting out the relevant data from the overall body of given information.
Select, describe, or illustrate a prediction made by applying the findings from measurements or an investigation.
Use relationships identified in the data to explain that regardless of the type of change, the total weight of matter is conserved.

29. Task Demands
Select or identify from a collection of potential model components, including distractors, components needed for a model that can explain lunar phases, eclipses of the sun, eclipses of the moon, or seasons on Earth. Components might include the sun, moon, Earth, solar energy, the moon’s orbital trace, Earth’s orbital trace, the angle of the moon’s orbital trace, the angle of Earth’s orbital trace, Earth’s axis, or the tilt of Earth’s axis.
Assemble or complete, from a collection of potential model components, an illustration or flow chart that is capable of representing the causes of lunar phases, eclipses of the sun, eclipses of the moon, or seasons on Earth. This does not include labeling a simple diagram of the Earth-sun-moon system.
Describe, select, or identify the relationships among components of a model that can explain lunar phases, eclipses of the sun, eclipses of the moon, or seasons on Earth. Components might include the sun, moon, Earth, solar energy, the moon’s orbital trace, Earth’s orbital trace, the angle of the moon’s orbital trace, the angle of Earth’s orbital trace, Earth’s axis, or the tilt of Earth’s axis.
Manipulate the components of a model to demonstrate how the relationships among the sun, the moon, Earth, and solar energy change to result in lunar phases, eclipses of the sun, eclipses of the moon, or seasons on Earth. * (SEP/DCI/CCC)
Make predictions about the effects of changes in the relationships among the sun, the moon, Earth, and solar energy as they relate to lunar phases, eclipses of the sun, eclipses of the moon, or seasons on Earth. Predictions can be made by manipulating model components, completing illustrations, or selecting from lists with distractors. * (SEP/DCI/CCC)
Identify missing components, relationships, or other limitations of a model that can explain lunar phases, eclipses of the sun, eclipses of the moon, or seasons on Earth.

30. Task Demands
Select or identify from a collection of potential model components, including distractors, components needed for a model that can explain lunar phases, eclipses of the sun, eclipses of the moon, or seasons on Earth. Components might include the sun, moon, Earth, solar energy, the moon’s orbital trace, Earth’s orbital trace, the angle of the moon’s orbital trace, the angle of Earth’s orbital trace, Earth’s axis, or the tilt of Earth’s axis.
Assemble or complete, from a collection of potential model components, an illustration or flow chart that is capable of representing the causes of lunar phases, eclipses of the sun, eclipses of the moon, or seasons on Earth. This does not include labeling a simple diagram of the Earth-sun-moon system.
Describe, select, or identify the relationships among components of a model that can explain lunar phases, eclipses of the sun, eclipses of the moon, or seasons on Earth. Components might include the sun, moon, Earth, solar energy, the moon’s orbital trace, Earth’s orbital trace, the angle of the moon’s orbital trace, the angle of Earth’s orbital trace, Earth’s axis, or the tilt of Earth’s axis.
Manipulate the components of a model to demonstrate how the relationships among the sun, the moon, Earth, and solar energy change to result in lunar phases, eclipses of the sun, eclipses of the moon, or seasons on Earth. * (SEP/DCI/CCC)
Make predictions about the effects of changes in the relationships among the sun, the moon, Earth, and solar energy as they relate to lunar phases, eclipses of the sun, eclipses of the moon, or seasons on Earth. Predictions can be made by manipulating model components, completing illustrations, or selecting from lists with distractors. * (SEP/DCI/CCC)
Identify missing components, relationships, or other limitations of a model that can explain lunar phases, eclipses of the sun, eclipses of the moon, or seasons on Earth.

31. 3-D Learning = Science Performance at the Intersection of the Dimensions
Phenomenon- Driven 3D Student Performances
1. Standards
2. Instruction
3. Assessment
4. Instructional Materials
5. Professional Development
Science and Engineering Practices
Crosscutting
Concepts
Disciplinary Core Ideas

32. 3-D Learning = Science Performance at the Intersection of the Dimensions
Phenomenon- Driven 3D Student Performances
1. Standards
2. Instruction
3. Assessment
4. Instructional Materials
5. Professional Development
Science and Engineering Practices
Crosscutting
Concepts
Disciplinary Core Ideas

33. The Structure of the Standards
Phenomenon: a discrete observation/event about the world/universe that can anchor a 3D-aligned classroom lesson or assessment entity. A phenomenon is essentially something interesting about the world that relates to one or two big ideas, and which can be studied using one of the scientific practices around which the PEs are built.
Phenomena are intended to anchor classroom discussions; the phenomenon sparks a question which kicks off the discussion – but - the entire discussion should be focused on addressing or explaining the phenomenon.

34. Phenomena
Phenomena (e.g., a sunburn) are specific examples of something that is happening in the world/universe—an event or a specific example of a general process.
Phenomena are not the explanations or the scientific terminology behind that specific example of what is happening.
They are the start-point of the conversation and the focus to be explained. They aren’t the explanation (that’s what the discussion about the phenomenon is for).

35. Phenomena in Student Learning
The desire is that there is a shift from: I’m going to provided you with the definition of evaporation. Then we’re going to discuss how evaporation operates.
The desire is that there is a shift to: In the early morning on a sunny day, there is a large puddle of rainwater in the middle of a sidewalk in town. After noon, the puddle has disappeared.
Essentially, here is a puzzling event or process that is related to evaporation, and which can spark questions about the underlying cause: How and why did this happen?

36. Phenomena

37. 3-D Standards in the Classroom
A classroom lesson that is built around 3-D standards (as they were designed to be implemented) is anchored on a phenomenon that is related to one or two core ideas, and which can be explained by students using a specific activity(s). The students conduct the activity to address the phenomenon.
Any assessment entities attempting to parallel this kind of classroom instruction need to be similarly crafted…

38. What is the Phenomenon Here?
NOT A PHENOMENA

39. What is the Phenomenon Here?
PHENOMENA

40. The Structure of the Clusters
AIR clusters aligned to 3-D Standards all include the same basic elements:
One phenomenon (usually in a stimulus on the left side of the screen)
Materials which augment the phenomenon (in the stimulus)
A cluster task statement (short statement at the end of the stimulus materials which provides the student information about the the overall goal of the cluster)
Individual interactions (usually on the right side)

41. The Structure of the 3-D Clusters
Each cluster is designed to engage the examinee in a grade-appropriate, meaningful scientific activity aligned to a specific standard. This task should be explicitly stated in the stimulus materials in a clear statement.
Each cluster begins with a phenomenon, which anchors the entire cluster. The interactions within the cluster all address the phenomenon. The phenomenon and materials that augment it are usually set apart from the items in a stimulus.
The interactions are engaged in sequence.
Each interaction in the cluster is aligned to at least two of the three dimensions (SEP, DCI, CCC) and if possible all three.

42. AIR 3-D – aligned entities
Clusters and Stand-Alone Items
Stimuli include text/imagery/animations/simulations
phenomenon & text
phenomenon
& text; graphics
interaction
several interactions
are aligned to the
stimulus, perhaps
related to different
parts
stimulus
stem text
prompts
text & animation
Item
cluster

43. The Structure of the 3-D Clusters
Stimulus
Performance
Expectation
Cluster
SEP
Interaction 1
Part A, B, etc.
Task Demand 1
DCI
DCI
SEP
DCI
CCC
Interaction 2
Part A, B, etc.
Task Demand 2
CCC
Interaction 3
Part A, B, etc.
SEP
Task Demands
3 & 4
DCI
Interaction 4
Part A, B, etc.
SEP
Task Demands
2 & 4
DCI
CCC

44. The Structure of the 3-D Clusters

45. The Structure of the 3-D Clusters

46. Cluster Specifications
Cluster specifications are documents that are designed to guide the work of writers as they craft linked groups of test questions, as well as to guide the subsequent reviews of those questions.
The specifications should serve as a roadmap for writers, facilitating the creation of interactions that will properly align to the three-dimensions and which will function together properly as coherent clusters.

47. The Structure of the Specifications
The text of the PE and the relevant practice, core idea, and cross cutting concept from which the standard/PE is derived.
Content limits, which refine the intent of the standards and provide limits of what may be asked of examinees. For example, they may identify the specific formulae that students are expected to know or not know for a specific PE.

49. The Structure of the Specifications
Vocabulary, which identifies the relevant technical words that students are expected to know, and related words that they are explicitly not expected to know. Of course, the latter category should not be considered exhaustive, since the boundaries of relevance are ambiguous, and the list is essentially limited by the imagination of the writers.

51. The Structure of the Specifications
Sample phenomena, which provide some examples of the sort of phenomena that would support effective item clusters related to the standard in question. In general, these should be guideposts, and item writers should seek comparable phenomena, rather than drawing on those within the documents. Novelty is valued when applying scientific practices.

53. The Structure of the Specifications
Task demands are the heart of the specifications. They identify the types of interactions and simulations that item writers should employ.
Each interaction should be clearly linked to one or more task demands. The verbs (e.g., select, illustrate, describe) guide the types of interactions writers might employ to elicit the student response.
We avoid explicitly identifying desired interaction types to accommodate future innovations and to avoid discouraging imaginative work by the writers.

55. The Structure of the Specifications
At the end of the specification documents we will be appended screen shots of a sample cluster, anchored on one of the sample phenomena that are listed in the specification.
The screen shots show the stimulus first, which includes the phenomenon and any context-providing materials that support/augment it.
The individual student interactions follow.

59. Resources A Framework for K-12 Science Education NGSS Standards Book
NGSS with Evidence Statements
\\Dc1fs\dc1ehd\share\Science Team Folder\NGSS
\\Dc1fs\dc1ehd\share\Science Team Folder\Training Materials\NGSS Training