Designing a Pool of Items

Workshop Flow The construct of MKT –Gain familiarity with the construct of MKT –Examine available MKT instruments in the field Assessment Design –Gain familiarity with the Evidence-Centered Design approach –Begin to design a framework for your own assessment Assessment Development –Begin to create your own assessment items in line with your framework Assessment Validation –Learn basic tools for how to refine and validate an assessment Plan next steps for using assessments

Domain Modeling (Design Pattern) (Define Test Specs) Domain Analysis Define item Template Define item Specs Develop Pool of items Collect/ Analyze Validity Data Refine items Refine items Assemble Test Document Technical Info Assessment Development Process

From Design to Development Use information from the Design Pattern to generate items (Use the process of generating items to refine the Design Pattern) What the test-taker should know Focal Knowledge, Skills, and Abilities Evidence we can collect to show they know it Potential observations Potential work products Potential rubrics Kinds of situations that can evoke this evidence Characteristic features Variable features

Item Development Steps Based on the foundational design work… 1.Content experts develop draft items 2.Refine draft items – “hygiene” 3.Collect validity data on draft items 4.Use validity data to refine items 5.Assemble and document the assessment

Assessment Item Anatomy Essential parts of a butterfly are… A.Wings, legs, teeth B.Eyes, wings, legs C.Nose, knees, ears D.Fingers, frontal lobes, hair Butterflies are members of the _______ phylum. What are the closest relatives of the butterfly, and why? Stem Choices Distracters Prompt Constructed Response (Closed-Ended) Constructed Response (Open-Ended) Multiple Choice

Conducting Hygiene Review is essential Clarity and precision Accuracy Grammar No ambiguity No unintended cues Distracters believable, appropriate, same length Placement of correct response varied

Not a Trivial Process! SimCalc

Other Resources for Hunters and Gatherers Standardized tests –International PISA ( ) TIMSS ( ) –National NAEP ( ) –State TAKS released items

Other Resources for Hunters and Gatherers Other resources –Established curricular materials Approved textbooks TPD materials Supplemental teacher resources –SRI’s online resources Performance Assessment Links in Math (PALM): Integrated Performance Assessments in Technology: Online Evaluation Resource Library (OERL): GLOBE assessment tools: –Research literature (e.g., theoretical papers that provide example items) –Other research-based assessments –Mental Measurements Yearbook from the Buros Institute of Mental Measurements

Conducting an “Item Camp” Content experts are trained in how to write items –Illustrative examples provided for each –Trainer demonstrates how to use the templates Content experts develop items –Use a form to help structure refinement –Iterative creation, sharing, feedback –Helps to work in groups –Give them a goal and a timeline –Remind them of their charge –Make them aware of their tools –Don’t throw away work Aim for 2-3x more items than you expect to use on the test form

Welcome to (mini) Item Camp!

Goals Develop 1 to 3 draft items for your (or your neighbor’s) assessment Have an experience of participating in an item camp

Starting with Item Templates What are templates? –Templates are industry standard in assessment development –Templates outline the structure of an item, you fill in the content –Guidelines to systematically produce high-quality items What are these templates? –Created based on the SimCalc design pattern to help seed thinking about the SimCalc items –Templates might be quite different for your design pattern –They are based on our observations about the structure of items that fit into types and “worked” in field testing –Heuristic tool, not theoretical framework In this context, recommended but not required

Template Types Based on Domain Analysis of MKT  Unconventional forms or representations  Choosing problems and examples that can illustrate key curricular ideas  Differentiating between colloquial and mathematical uses of language  Linking precise aspects of representations  Understanding implications of models and representations  Evaluating mathematical statements

Structure of the Templates All multiple choice, with the following elements: –Context  Sets up a teaching task  Presents information necessary to answer the question –Question/Prompt –Distracters

Review Templates

Guidelines for distracters Usually 4 or 5 distracters (may be more) May use “Choose one” or “Choose all that apply.” Responses may include “All of the above” and “None of the above.” Whenever possible, distracters will reflect common errors or misunderstandings, naïve pre-conceptions, or other misconceptions. Teachers should not be able to rule out a distracter or identify the answer simply because of superficial or trivial characteristics, syntactic complexity, or concept complexity. Distracters will not be partially correct responses nor will they be designed to “trick” teachers in responding incorrectly. Be judicious!

Other Item Types True/false Matching Open-ended –Calculations –Short responses –Generating representations –Generating explanations, stories, etc. Require a rubric for scoring

Filling Out the Item Form Name Connection to standard or curriculum KSA assessed Item Correct answer and/or rubric Notes

“Item Hygiene” Consider this a brainstorm, so do not be overly concerned about IH However, keep in mind that this is the next part of the pipeline IH includes: –Clarity, precision, correctness –Grammar –Non-ambiguity –No unintended cues –Distracters believable, appropriate, same length –Placement of correct response varied

Draw on Resources Available Your Design Pattern Your curriculum materials MKT resources (e.g., prior assessments) NCTM and Texas standards

Activity #3 Mini-Item Camp Find Activity #3 in Tab 5 Work on your own, with a partner, or with a small group Develop some MKT items (1 to 3) Prepare 1 or 2 items for the Item Hygiene Review (see page 7) On the item form, include: –Name –Connection to standard or curriculum –KSA assessed –Item –Correct answer and/or rubric –Notes

Time for Activity: 9:30-10:15 On the item form, include: –Name –Connection to standard or curriculum –KSA assessed –Item –Correct answer and/or rubric –Notes

Activity #4 Find Activity #4 on Tab 5 Sit in groups of 3 for Hygiene Review Use the Item Hygiene Guide to edit your teammates items Fix your 1 or 2 items and prepare a clean copy for this afternoon’s activity Discussion to follow –Show-and-tell of 2 or 3 items –Your insights, questions, challenges

Activity #4 Conduct Item Hygiene Review Conduct item hygiene on your set of items, using the Item Hygiene Guide. We suggest writing corrections directly on the item. Following, we will discuss –Insights about development of assessment items –Questions and challenges

Some Useful Resources Anderson, J. R. (2000). Cognitive psychology and its implications (3 rd Ed.). New York: W. H. Freeman and Company. Baxter, G. P., & Glaser, R. (1998). The cognitive complexity of science performance assessments. Educational Measurement: Issues and Practice, 17, 3, Pellegrino, J., Chudowsky, N., Glaser, R. (Eds.). (2001). Knowing what students know: The science and design of educational assessment. Washington, DC: National Academy Press. Quellmalz, E., Hinojosa, T., & Rosenquist, A. (2001). Design of student assessment tools for the Global Learning and Observations to Benefit the Environment (GLOBE) program. Presentation at the annual GLOBE International Conference, Blaine, WA. Snow, R. E., Federico, P. A., & Montague, W. E. (Eds.). (1980). Aptitude, learning, and instruction: Volume 2. Cognitive process analyses of learning and problem-solving. Hillsdale, NJ: Erlbaum.