UCLA Graduate School of Education & Information Studies National Center for Research on Evaluation, Standards, and Student Testing Online Assessment within an Ontological Framework Jia Wang & Terry Vendlinski American Educational Research Association Annual Meeting San Francisco, CA - April 7-11, 2006

2/28 Overview Section 1: Project Overview Section 2: Description of ADDS Section: Findings The section titles here should be reflected in the blue section dividers throughout your presentation.

Section 1: Project Overview The section title here should be reflected in the OVERVIEW slide of your presentation.

4/28 For many years… The theme of CRESST work has been: Assessment to improve learning

5/28 IERI/NSF Online Assessment Design Project Project goal: Based on an ontology of science knowledge, the project is to create an Assessment Design and Delivery System (ADDS) for middle school science teachers.

6/28 An Ontology: is a map of expert knowledge and skills represents concepts and the relationships among them

7/28 Building the Ontology for Physical Science: Physicists identify organizing concepts and principles (“big ideas”) Determine related ideas and skills: facts, problems, situations, etc. Map all ideas and skills

8/28 ADDS Ontology

9/28 Ontology (Hugin) allows us to: associate assessment items with the “Big Idea(s)” they assess rank the assessment items by student mastery probability when overlaid with Bayesian Network (Protégé) continually update the conditional probabilities based on students’ later performance

Section 2: Description of ADDS

11/28 ADDS has: an assessment authoring system an assessment delivery system reporting and interpretation tools When pasting text from another document, do the following: 1.Highlight the text you want to replace 2.Go to the EDIT menu and select PASTE SPECIAL 3.Select “Paste as: UNFORMATTED TEXT”

12/28 Teachers can use ADDS to: Design new assessment Select and/or revise assessments from database Upload assessments into the database When pasting text from another document, do the following: 1.Highlight the text you want to replace 2.Go to the EDIT menu and select PASTE SPECIAL 3.Select “Paste as: UNFORMATTED TEXT”

13/28 To begin: One topic per slide max Keep it clean, clear, consistent Fewer points, more slides 6 bullets per slide, 6 words per bullet Printouts can have notes When pasting text from another document, do the following: 1.Highlight the text you want to replace 2.Go to the EDIT menu and select PASTE SPECIAL 3.Select “Paste as: UNFORMATTED TEXT”

14/28 From design to use When pasting text from another document, do the following: 1.Highlight the text you want to replace 2.Go to the EDIT menu and select PASTE SPECIAL 3.Select “Paste as: UNFORMATTED TEXT”

15/28 Design to a topic : When pasting text from another document, do the following: 1.Highlight the text you want to replace 2.Go to the EDIT menu and select PASTE SPECIAL 3.Select “Paste as: UNFORMATTED TEXT”

16/28 Design to a standard : When pasting text from another document, do the following: 1.Highlight the text you want to replace 2.Go to the EDIT menu and select PASTE SPECIAL 3.Select “Paste as: UNFORMATTED TEXT”

17/28 Pre-made assessment :

Section 3: Findings The section title here should be reflected in the OVERVIEW slide of your presentation.

19/28 Findings – Big Ideas The teachers in the treatment group were much more likely to begin the assessment development process by noting the broad idea that they were trying to assess, and have the students address these “big ideas” rather than merely recalling specific facts. No teachers in the control group and no teacher prior to treatment apparently used “big ideas” as a basis for test development. We believe that using the “Big Ideas” as a basis for test development will encourage teachers to develop assessments that allow better inferences about how deeply students understand the important concepts in a field of study.

20/28 Findings – Rubrics Only teachers in the treatment group developed rubrics or detailed the responses they expected to receive back from students. Our experience suggests that the very process of rubric development encourages test writers to clarify or refine the test question, even as they evaluate student work. Based on research and our own experience, we believe the capability of ADDS to associate assessment items with rubrics has the potential to significantly improve both instruction and student learning.

21/28 Findings – Teachers Teachers in the non-treatment group seldom used any information sources when designing assessments. In general, we found that assessments that included information sources are far more likely to ask students for higher order thinking that those that do not include such resources. Teachers using the ADDS produced fewer assessment questions during the same time period than those control teachers.

22/28 Conclusions Scaffolding the assessment development process for teachers and providing a means whereby assessments can be continually “polished” should improve the quality of classroom formative assessments. The technology itself increased the time necessary to develop assessments, but the assessments were often probing student thinking at a deeper level (the “big ideas” of a knowledge domain), included expected student responses and scoring rubrics, and situated the tasks for students in a context where the student could apply knowledge or a concept being assessed. ADDS has the potential to positively effect assessment practice and student learning in classrooms where it is regularly used by teachers.

23/28 next presentation Jia Wang voice fax ©2006 Regents of the University of California Terry Vendlinski voice fax