Gary W. Phillips Vice President & Institute Fellow American Institutes for Research Next Generation Achievement Standard Setting Symposium CCSSO NCSA New.

Slides:

Advertisements

Similar presentations

1 North Carolina National Assessment of Education (NAEP) 2009 LEA Training--Session A December 2008 Iris L. Garner, Ph.D. NAEP.

Advertisements

Kalispell’s 4 th Grade Reading International Comparison On the th grade PIRLS international Reading assessment, Florida’s performance was compared.

Comparing State Reading and Math Performance Standards Using NAEP Don McLaughlin Victor Bandeira de Mello National Conference on Large-Scale Assessment.

Computing Leadership Summit STEM Education Steve Robinson U.S. Department of Education White House Domestic Policy Council February 22, 2010.

Jack Buckley Commissioner National Center for Education Statistics October 3, 2013.

Dissemination of TIMSS 2006/07 as a trend study in New Zealand Presentation to the IEA 50 th General Assembly 2009 Dr Jit Cheung Manager Comparative Education.

Design of the 2011 NAEP- TIMSS Linking Study Andrew Kolstad Senior Technical Advisor to the Assessment Division National Center for Education Statistics.

Improving Education Quality in Thailand Five Stylized Facts Kevin Macdonald, World Bank

Reliability and Linking of Assessments. Figure 1 Differences Between Percentages Proficient or Above on State Assessments and on NAEP: Grade 8 Mathematics,

The State of the State TOTOM Conference September 10, 2010 Jim Leigh Office of Assessment and Information Services Oregon Department of Education.

WHAT DO SMARTER QUESTIONS LOOK LIKE? COMPUTER ADAPTIVE TEST: ENGLISH.

Multnomah County Student Achievement Presented to the Leaders Roundtable November 25, 2008 Source: Oregon Department of Education, Dr. Patrick.

Gary W. Phillips Vice President and Chief Scientist American Institutes for Research August 3, 2015.

Introduction to plausible values National Research Coordinators Meeting Madrid, February 2010.

High School Mathematics: Where Are We Headed? W. Gary Martin Auburn University.

Mathematics The Bigger Picture. Mathematics is not arithmetic… Arithmetic is about –math facts –computation –algorithms –word problems Mathematics is.

September 11,  Measurements of Student Progress (MSP) ◦ Reading and math were tested every year in grades 3 through 8. ◦ Writing was tested in.

Assessment toolkit of primary school subject competences Peter Nezhnov, PhD degree in psychology, Leading Researcher at the Russian Academy of Education.

Mathematics Initiative Office of Superintendent of Public Instruction 2006 Mathematics WASL Released Items for Grades 3–8 December 6, 2006 Slide

Setting Performance Standards for the Hawaii State Alternate Assessments: Reading, Mathematics, and Science Presentation for the Hawaii State Board of.

Measuring Learning and Improving Education Quality: International Experiences in Assessment John Ainley South Asia Regional Conference on Quality Education.

Benchmarking with National and International Assessments Larry V. Hedges Northwestern University This paper is intended to promote the exchange of ideas.

International Assessments September 5, 2012 DAC Annual Meeting

Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved. 10/7/2015 A Model for Scaling, Linking, and Reporting.

Slide 1 Estimating Performance Below the National Level Applying Simulation Methods to TIMSS Fourth Annual IES Research Conference Dan Sherman, Ph.D. American.

Dubai’s 15 year old students Results from PISA 2009 Fatma Al Janahi Head of International Assessments Knowledge & Human Development Authority GCES Symposium,

1 STANDARD ERRORS ON DIFFERENCES. How to compute the standard error of the difference between : – Two countries; – An OECD country and the OECD total.

Understanding The PLAN. What Do My Scores Mean?  Your scores range between 1 and 32.  The PLAN takes the number of correct responses on each test and.

Jackson County School District A overview of test scores and cumulative data from 2001 – 2006 relative to the following: Mississippi Curriculum Test Writing.

1 Comparing International Assessments to NAEP May 30, 2008 Eugene Owen Senior Advisor, NCES International Activities Program This paper is intended to.

35th Annual National Conference on Large-Scale Assessment June 18, 2005 How to compare NAEP and State Assessment Results NAEP State Analysis Project Don.

 Any high stakes testing report  AMRT Reports  Science Assessments Today we will go over an ARMT report.

What Use Are International Assessments for States? 30 May 2008 Jack Buckley Deputy Commissioner National Center for Education Statistics Institute of Education.

Future Ready Schools National Assessment of Educational Progress (NAEP) in North Carolina Wednesday, February 13, 2008 Auditorium III 8:30 – 9:30 a.m.

A Statistical Linkage Between NAEP and ECLS-K Grade Eight Reading Assessments Enis Dogan Burhan Ogut Young Yee Kim Sharyn Rosenberg NAEP Education Statistics.

Gary W. Phillips American Institutes for Research United States Department of Education Public Hearings December 1, 2009, Denver, Colorado.

School-level Correlates of Achievement: Linking NAEP, State Assessments, and SASS NAEP State Analysis Project Sami Kitmitto CCSSO National Conference on.

KAREN TOHINAKA SAGE Overview. What is SAGE? SAGE (Student Assessment of Growth and Excellence)  is Utah’s computer adaptive assessment system aligned.

Describing the Academic Content of International Mathematics and Science Assessments & Their Relationship To State Academic Content Standards & Assessments.

Gary W. Phillips American Institutes for Research CCSSO 2014 National Conference on Student Assessment (NCSA) New Orleans June 25-27, 2014 Multi State.

USING THE OECD TEST FOR SCHOOLS (BASED ON PISA) FOR COMPARISON OF AMERICAN STUDENT PERFORMANCE AGAINST INTERNATIONAL BENCHMARKS Frank Rijmen, CTB.

Provincial Assessment Results Anglophone West School District November 26, 2015.

Obtaining International Benchmarks for States Through Statistical Linking: Presentation at the Institute of Education Sciences (IES) National Center for.

Ming Lei American Institutes for Research Okan Bulut Center for Research in Applied Measurement and Evaluation University of Alberta Item Parameter and.

NAEP-TIMSS Linking Study In 2011, both NAEP and TIMSS assessed grade 8 students in Mathematics and Science 2011provided an opportunity to examine the linkage.

Scale Scoring A Revised Format for Provincial Assessment Reports.

OECD EMPLOYER BRAND Playbook 1 What makes schools and school systems successful? Programme for International Student Assessment (PISA) Organisation for.

Connecticut Mastery Test Fourth Generation Grades 3, 4, 5, 6, 7 and 8 Connecticut Academic Performance Test Third Generation Grade 10 Presented to the.

1 Main achievement outcomes continued.... Performance on mathematics and reading (minor domains) in PISA 2006, including performance by gender Performance.

Student Achievement Data Mount Olive Township Public Schools.

1. 2 National Assessment of Educational Progress (NAEP) Survey of academic achievement for the nation and the states Assesses various subjects at grades.

Policy Definitions, Achievement Level Descriptors, and Math Standards.

STUDENT ASSESSMENT FOR INDEPENDENTS SCHOOLS: BRIEF REMARKS. J. Enrique Froemel, Ph.D. Director, Student Assessment Office.

Provincial Assessment Results

NAEP and TIMSS Mathematics 2015

Metropolitan Nashville Public Schools

Comparability of Assessment Results in the Era of Flexibility

PISA • PIRLS • TIMSS Program for International Student Assessment

NECAP PRESENTATION.

MME Reading Score Analysis

Delaware Department of Education

Sampling Distribution

Sampling Distribution

AdvancED Priority Targets

PARCC Math Evidence Statements.

Brian Gong Center for Assessment

NAEP and International Assessments

Solving Equations 3x+7 –7 13 –7 =.

Example Make x the subject of the formula

Innovative Approaches for Examining Alignment

Presentation transcript:

Gary W. Phillips Vice President & Institute Fellow American Institutes for Research Next Generation Achievement Standard Setting Symposium CCSSO NCSA New Orleans LA June 25, 2014 Using International Benchmarking to Help Establish Internationally Competitive Performance Standards

National and International Benchmarking 2 NAEP TIMSS PIRLS

3 National Benchmarking using NAEP ( simulated data ) Math, Grade 8NAEP-equivalents to State Score Page inEstimatedNAEP-StandardProportion BookmarkStateProportionEquivalentError ofin U.S. OrderedScaledAboveScaledNAEP-Above Item BookletScoresCut-ScoreScoreEquivalentNAEP-Equiv Ad = Proficient = Basic = Below Basic

4 National and International Benchmarking using NAEP and TIMSS ( simulated data ) Math, Grade 8NAEP-equivalents to State ScoreTIMSS-equivalents to State Score Page inEstimatedNAEP-StandardProportionTIMSS-StandardProportion BookmarkStateProportionEquivalentError ofin U.S.EquivalentError ofin Korea OrderedScaledAboveScaledNAEP-AboveScaledTIMSS-Above Item BookletScoresCut-ScoreScoreEquivalentNAEP-EquivScoreEquivalentTIMSS-Equiv Ad = Ad= High = Proficient = Intermediate = Basic = Low = Below Basic Below Low

5 National and International Benchmarking using NAEP and TIMSS ( simulated data ) Math, Grade 8NAEP-equivalents to State ScoreTIMSS-equivalents to State Score Page inEstimatedNAEP-StandardProportionTIMSS-StandardProportion BookmarkStateProportionEquivalentError ofin U.S.EquivalentError ofin Korea OrderedScaledAboveScaledNAEP-AboveScaledTIMSS-Above Item BookletScoresCut-ScoreScoreEquivalentNAEP-EquivScoreEquivalentTIMSS-Equiv Ad = Ad= High = Proficient = Intermediate = Basic = Low = Below Basic Below Low

6 National and International Benchmarking using NAEP and TIMSS ( simulated data ) Math, Grade 8NAEP-equivalents to State ScoreTIMSS-equivalents to State Score Page inEstimatedNAEP-StandardProportionTIMSS-StandardProportion BookmarkStateProportionEquivalentError ofin U.S.EquivalentError ofin Korea OrderedScaledAboveScaledNAEP-AboveScaledTIMSS-Above Item BookletScoresCut-ScoreScoreEquivalentNAEP-EquivScoreEquivalentTIMSS-Equiv Ad = Ad= High = Proficient = Intermediate = Basic = Low = Below Basic Below Low

7 National and International Benchmarking using NAEP and TIMSS ( simulated data ) Math, Grade 8NAEP-equivalents to State ScoreTIMSS-equivalents to State Score Page inEstimatedNAEP-StandardProportionTIMSS-Standard BookmarkStateProportionEquivalentError ofin U.S.EquivalentError ofTIMSS- OrderedScaledAboveScaledNAEP-AboveScaledTIMSS-Equivalent Item BookletScoresCut-ScoreScoreEquivalentNAEP-EquivScoreEquivalentRange Ad = Ad= High = Proficient = Intermediate = Basic = Low = Below Basic Below Low

International Benchmarking with PISA 8 The international benchmarking with PISA was replicated in in 2010 in three states (Hawaii, Oregon and Delaware) in three subjects (reading, mathematics and science). The statistical linking with PISA was accomplished by embedding secure PISA items in the state assessment through a legally binding Memo of Understanding and Confidentiality Agreement between AIR and the OECD.

Common item Equating for PISA 9 About 30 items are embedded as field test items in the computer adaptive test in the state. The computer adaptive field test algorithm randomly administered the PISA items. Number items per student - If the state had F = 400 items for field testing, P = 30 PISA items, and the test length was L = 40, then the expected number of PISA items administered to each student was E( p ) = 1/F * L * P = 3. Number students per item - If the student population size was N = 10,000, then the expected sample size per PISA item was E( n ) = 1/F * L * N = 1,000. Number of items per student does not provide enough information per student to get estimates of PISA scores (i.e., distributions of PISA- plausible values). Number students per item provides enough information per item to get good state-based estimates of the PISA item parameters.

PISA Grade 10 Mathematics 10 Results for State Assessment OrderedTheta Standard Error of ThetaPercentPISA-Standard Percent Reaching Percent Reaching ItemAssociated ReachingequivalentError ofOECDFinland BookletItemWith StateScaledPISA- PageParameterRP67 Cut-scoreScoreequivalent PISA Level V = 607 (Top Performers)

PISA Grade 10 Mathematics 11 Results for State Assessment OrderedTheta Standard Error of ThetaPercentPISA-Standard Percent Reaching Percent Reaching ItemAssociated ReachingequivalentError ofOECDFinland BookletItemWith StateScaledPISA- PageParameterRP67 Cut-scoreScoreequivalent PISA Level IV = 545 (Strong Performers)

PISA Grade 10 Mathematics 12 Results for State Assessment OrderedTheta Standard Error of ThetaPercentPISA-Standard Percent Reaching Percent Reaching ItemAssociated ReachingequivalentError ofOECDFinland BookletItemWith StateScaledPISA- PageParameterRP67 Cut-scoreScoreequivalent PISA Level III = 482 (Moderate Performers)

PISA Grade 10 Mathematics 13 Results for State Assessment OrderedTheta Standard Error of ThetaPercentPISA-Standard Percent Reaching Percent Reaching ItemAssociated ReachingequivalentError ofOECDFinland BookletItemWith StateScaledPISA- PageParameterRP67 Cut-scoreScoreequivalent

PISA Grade 10 Mathematics 14 Results for State Assessment OrderedTheta Standard Error of ThetaPercentPISA-Standard Percent Reaching Percent Reaching ItemAssociated ReachingequivalentError ofOECDFinland BookletItemWith StateScaledPISA- PageParameterRP67 Cut-scoreScoreequivalent PISA Level II = 420 (Moderate Performers)