ETS Confidential & Proprietary Diagnostic testing that just might make a difference Dylan Wiliam Institute of Education, University of London www.dylanwiliam.net.

Slides:



Advertisements
Similar presentations
Response to Intervention (RtI) in Primary Grades
Advertisements

Feedback to students: What do we know about feedback and learning?
WELCOME to Kindergarten
WELCOME TQ SUMMER 2011 WORKSHOP: PROPORTIONALITY JUNE 6-17, 2011.
Supporting Rigorous Mathematics Teaching and Learning
Webb’s Depth of Knowledge
SVMI Concept Development Lesson Common Core State Standards- Mathematics (CCSS-M) Conference Mariana Alwell Professional Development Provider Silicon Valley.
Curriculum & Instruction Webinar October 18, 2013.
Re-viewing the CMI Framework (Re-view: Take another look, see anew)
Improving the learning of numeracy through formative assessment Dylan Wiliam National Numeracy Conference Edinburgh, March
Integrating assessment with instruction: what will it take to make it work? Dylan Wiliam.
Expedition Atlantis Using the engaging nature of robotics exploration to teach students skills in proportional reasoning.
Once you know what they’ve learned, what do you do next? Designing curriculum and assessment for growth Dylan Wiliam Institute of Education, University.
Learning Goals, Scales and Learning Activities
Science PCK Workshop March 24, 2013 Dr. Martina Nieswandt UMass Amherst
1 Formative Assessment Debbie Owens AMSP University of Kentucky Kathy Strunk AMSP University of Tennessee.
Lesson Objectives Summer Content Institute “The quality of one’s thinking about objectives during planning directly accounts for the effectiveness.
Fred Gross Education Development Center, Inc.
MATHEMATICS KLA Years 1 to 10 Understanding the syllabus MATHEMATICS.
Day 3: Rubrics as an Assessment Tool. "There are only two good reasons to ask questions in class: to cause thinking and to provide information for the.
“We will lead the nation in improving student achievement.” CLASS Keys TM Module 1: Content and Structure Spring 2010 Teacher and Leader Quality Education.
Building Effective Assessments. Agenda  Brief overview of Assess2Know content development  Assessment building pre-planning  Cognitive factors  Building.
Engaging Learners and Realizing the Development of Mathematical Practices ALM Conference July 15, 2015 Trena L. Wilkerson Professor, Mathematics Education.
Math Liaison Meeting September 2014 Presenter: Simi Minhas, Math Achievement Coach Network 204.
What is the Affective Domain and where does it come from? Considering the Affective Domain in Geoscience education Jenefer Husman Arizona State University.
Chapter 11 Helping Students Construct Usable Knowledge.
Math rigor facilitating student understanding through process goals
5E Learning cycle – with sample lesson
 Participants will teach Mathematics II or are responsible for the delivery of Mathematics II instruction  Participants attended Days 1, 2, and 3 of.
Indicators of Multiplicative Reasoning among Fourth Grade Students Carrier, J. (2010) Under the direction of Dr. Sarah B. Berenson and Dr. Kerri Richardson.
Quick Glance At ACTASPIRE Math
1. Principles Equity Curriculum Teaching 3 Assessment Technology Principles The principles describe particular features of high-quality mathematics programs.
© 2013 UNIVERSITY OF PITTSBURGH LEARNING RESEARCH AND DEVELOPMENT CENTER Study Group 7 - High School Math (Algebra 1 & 2, Geometry) Welcome Back! Let’s.
Company LOGO Embedding Mathematics into Advanced Careers 1 New Jersey Entrepreneurship Summer Teacher Training Institute July 31, 2013 Facilitator: Kenna.
RIGOR IN ASSESSMENTS Nov 14 – Teach for America Break-Out Session.
Planning and Integrating Curriculum: Unit 4, Key Topic 1http://facultyinitiative.wested.org/1.
Modified Achievement Tests for Students with Disabilities: Distractor Analysis Michael C. Rodriguez University of Minnesota CCSSO’s National Conference.
IES Evaluations and Data Collection Instruments Lauren Angelo National Center for Education Evaluation and Sally Atkins-Burnett Mathematica Policy Research.
Classroom Evaluation & Grading Chapter 15. Intelligence and Achievement Intelligence and achievement are not the same Intelligence and achievement are.
CT 854: Assessment and Evaluation in Science & Mathematics
Record Keeping and Using Data to Determine Report Card Markings.
Pennsylvania Standard B Algebra and Functions Discover and generalize patterns, including linear, exponential and simple quadratic relationships.
EEPD 2009: The Collaborative Inquiry Process Farmington High School November 3, 2009.
1. Designing an assessment system Presentation to the Scottish Qualifications Authority, August 2007 Dylan Wiliam Institute of Education, University of.
Pennsylvania Standard J. Geometry Standard
Formative Assessment Follow up #1: Planning for Assessment Jeanette Grisham Adrienne Somera NWESD.
P.R.I.D.E. School Professional Day :45 am- 3:30 pm.
MH502: Developing Mathematical Proficiency, Geometry and Measurement, K-5 Seminar 1 September 28, 2010.
Assessment Power! Pamela Cantrell, Ph.D. Director, Raggio Research Center for STEM Education College of Education University of Nevada, Reno.
State Support for Classroom Assessment Fen Chou, Ph.D. Louisiana Department of Education National Conference on Student Assessment June 27, 2012.
Early Childhood Outcomes Center New Tools in the Tool Box: What We Need in the Next Generation of Early Childhood Assessments Kathy Hebbeler ECO at SRI.
The Value in Formative Assessment Prepared By: Jen Ramos.
A Signature Tool of The Institute for Learning
Prof. Dr. Christine Garbe (University of Cologne)
High Quality Items for High Stake Tests Delhi, 20 November, 2015.
Write your personal definition of “cognitive rigor” What do rigorous academic environments look and sound like?
Knowledge is fixed and need only to transfer from teacher to students is based on constructive and transformation process through learning process Learning.
SBAC-Mathematics November 26, Outcomes Further understand DOK in the area of Mathematics Understand how the new SBAC assessments will measure student.
PASS Criteria F Construction of Knowledge F Disciplined Inquiry F Value Beyond School.
Implementing Formative Assessment Processes: What's Working in Schools and Why it is Working Sophie Snell & Mary Jenatscheck.
How Does the Common Core Affect the Work of Teaching? Records of Practice from University of Michigan-Dearborn Interview Task Analyses Rheta Rubenstein.
Day Two: February 25, :30-3:00. Series Goals Participants will have the opportunity to:  Work collaboratively to:  Deepen their knowledge of the.
Using Cognitive Science To Inform Instructional Design
Oleh: Beni Setiawan, Wahyu Budi Sabtiawan
On the relevance of (more) data-based decision making in education
Designing an assessment system
Quality in formative assessment
Designing an assessment system
EDUC 2130 Quiz #10 W. Huitt.
Pedagogical Content Knowledge – Elementary Mathematics
Presentation transcript:

ETS Confidential & Proprietary Diagnostic testing that just might make a difference Dylan Wiliam Institute of Education, University of London

2 Diagnostic Items in Mathematics and Science: Project rationale Traditional testing deals with individuals, but teachers mostly deal with groups Data-Push vs. Decision-Pull –Data-push Quality control at end of an instructional sequence Monitoring assessment Identifies that remediation is required, but not what Requires new routines to utilize the information –Decision-Pull Starts with the decisions teacher make daily Supports teachers “on-the-fly” decisions If a 30-item test provides useful information on an individual, then responses from 30 individuals on a single item might provide useful information on a class

3 Premises of the DIMS project Single, well-developed, questions provide a reasonably strong basis for real-time instructional decision-making Questions with principled and interpretable incorrect answers can be used to elicit evidence of student thinking Such questions, when used with all-student response systems, and when followed with discussion, can advance student learning Such questions can deepen teacher knowledge (both content and content-pedagogical)

4 The DIMS project in practice Focuses on 4 th & 8 th grade, mathematics & science Includes a bank of approximately 150 diagnostic questions per subject/grade level Question bank is a complementary resource for any curriculum, not a curriculum replacement Items to be used one at a time, within the flow of day- to-day lessons Focuses on collecting evidence about student learning in order to adapt instruction to meet the learning needs of students in real time

5 Item development Review of state standards Review of relevant literature on student conceptions Identification of significant (mis-)conceptions Item construction –appears to be very difficult for many item-writers –a distractor-stem-key approach worked for some Item review and editing –Expert review –Piloting –Final review and editing

6 DIMS meets traditional psychometrics Concept-based distractors increase difficulty Most IRT models assume –Monotonicity –Equivalence of incorrect responses Need for analysis of single items makes most theories difficult to apply, or irrelevant

7 Semi-dense items (Bart, Post, Lesh & Behr, 1994) Five properties Response interpretability Response discrimination Rule discrimination Exhaustive set usage Semi-density

8 Response interpretability Each response interpretable by at least one cognitive rule

9 Response discrimination Each response interpretable by exactly one cognitive rule

10 Rule discrimination I Response discrimination + uniqueness of cognitive rules that interpret a response

11 Exhaustive rule set usage Response discrimination + every cognitive rule interprets at least one response to the item.

12 Semi-density Exactly one cognitive rule interprets each response to the item and each cognitive rule interprets exactly one response

13 Discriminating incorrect cognitive rules (Hart, 1981) Version 1 If e+f = 8, then e+f+g = a) 9 b)12 c)15 d)8+g Version 2 If f+g = 8, then f+g+h = a)9 b)12 c)15 d) 16 e) 8+h

14 Discriminating correct cognitive rules (Bart et al., 1994) Version 1 a)12 b)24 Version 2 a)24 cents, because 8×3=24 or 8 pieces × 3 cents per piece=24 cents. b)24 cents, because 4×6=24. c)24 cents, because 2/6=8/x and 2x=48 for which x=24. d)24 cents, because 2/6=8/x and 2/6x4/4=8/24. e)24 cents, because 2/6=4/12=6/18=8/24. f)24 cents, became 2/6=4/12=8/24. g)12 cents, because 2+4=6 implies that 8+4=12 Ann and Kathy each bought the same kind of bubble gum at the same store. Ann bought two pieces of gum for six cents. If Kathy bought eight pieces of gum, how much did she pay?

15 Discriminating between incorrect and correct cognitive rules Version 1 There are two flights per day from Newtown to Oldtown. The first flight leaves Newtown each day at 9:20 and arrives in Oldtown at 10:55. The second flight from Newtown leaves at 2:15. At what time does the second flight arrive in Oldtown? Show your work. Version 2 There are two flights per day from Newtown to Oldtown. The first flight leaves Newtown each day at 9:05 and arrives in Oldtown at 10:55. The second flight from Newtown leaves at 2:15. At what time does the second flight arrive in Oldtown? Show your work.

16 Discriminating between incorrect and correct cognitive rules (2) BCDABCDA

17 Conclusion Correct Incorrect

18 Conclusion (2) For an item to support instructional decision- making, the key requirement is that in no case do incorrect and correct cognitive rules map on to the same response. If this property is met, then the semi-density properties are less important. If this property is not met, then the semi-density properties are irrelevant. The discovery of new incorrect cognitive rules that interpret item keys leads to item improvement

19 Conclusion (3) Data on impact on student achievement not yet available Evidence of impact on –Teachers’ practice –Student engagement –Student affect

20 Questions? Comments?