Cornelia Orr, Executive Director National Assessment Governing Board 12 th Grade NAEP Preparedness Research Plans and Progress

The Board’s Work on Preparedness Has Been Underway for Nearly a Decade th Grade Commission Established 2005 Board Resolution on Reporting 12 th Grade Preparedness 2006 Grade 12 Reading & Mathematics Frameworks Revised 2009 Board Approves Preparedness Research Plan Conduct Research Studies and Develop Preparedness Report  The Governing Board has demonstrated a rigorous, comprehensive, and methodical approach to research and has not presumed the equivalence of college and career preparedness. This research will help inform the definition of preparedness in terms of NAEP. 2 July 26, 2011 Cornelia Orr, NAGB

Key Considerations for the Governing Board Statements in NAEP reports must be shaped and supported by validity evidence (validity framework) Recognize NAEP has limitations Reporting on NAEP 12 th grade preparedness may or may not prove to be feasible Consider additional Grade 12 academic areas in the future 3 July 26, 2011 Cornelia Orr, NAGB

Purpose: Technical Panel on 12 th Grade Preparedness Research Identify technically appropriate and valid ways to use grade 12 NAEP for reporting preparedness Assist in planning research and validity studies Recommend research designs and priorities for implementation in 2008, 2009, and thereafter Focus on 12 th grade reading and mathematics 4 July 26, 2011 Cornelia Orr, NAGB

Technical Panel on 12 th Grade Preparedness Research: Members Michael Kirst (Chair) Stanford University John Campbell University of Minnesota David T. Conley University of Oregon Michael Kane National Conference of Bar Examiners Mark David Milliron Catalyze Learning International Robert Mislevy University of Maryland George C. Thornton, III Colorado State University 5 July 26, 2011 Cornelia Orr, NAGB

Panel’s Key Conclusions Reporting on 12 th grade preparedness seems feasible Use multipronged approach to collect validity information Avoid representing 12 th grade NAEP as the single authoritative definition or conception of preparedness Consider the national context related to preparedness 6 July 26, 2011 Cornelia Orr, NAGB

Preparedness: Working Definitions Preparedness for College: reading and mathematics knowledge and skills to qualify for placement into entry level college credit courses that meet general education requirements, without the need for remedial coursework in math or reading Preparedness for Workplace Training: reading and mathematics knowledge and skills needed to qualify for a job training program without remediation in math or reading “Preparedness” versus “Readiness” 7 July 26, 2011 Cornelia Orr, NAGB

Types of Research Studies Content alignment Judgmental standard-setting Statistical relationship Survey of Postsecondary Institutions Benchmarking Studies 8 July 26, 2011 Cornelia Orr, NAGB

Study Interrelationships 9 July 26, 2011 Cornelia Orr, NAGB

Data for Preparedness Research 2009 NAEP 12 th grade Reading and Mathematics  National and state level data  11 states voluntarily participated Linkages to state data systems (only 1 in 2009)  Advance planning  Confidentiality assurance processes  Follow-up for post secondary college/career status Planning for July 26, 2011 Cornelia Orr, NAGB

Grade 12 NAEP Results for 11 states 11 July 26, 2011 Cornelia Orr, NAGB

Research Studies Preparedness StudyStatus Content Comparisons (SAT, ACT, Accuplacer, Work Keys) Complete Judgmental Standard SettingUnderway Statistical Relationships /Linking (SAT, Accuplacer, ACT) Underway Survey of Higher Ed InstitutionsUnderway BenchmarkingPlanned 12 July 26, 2011 Cornelia Orr, NAGB

Reference Points on the NAEP Scale (Scenario 1) July 26, Cornelia Orr, NAGB

Reference Points on the NAEP Scale (Scenario 2) July 26, Cornelia Orr, NAGB

Reference Points on the NAEP Scale (Scenario 3) July 26, Cornelia Orr, NAGB

Next Steps Develop summary report of validity evidence  Validity Framework – external reviews Develop preparedness reports  Based on national data analyses  Research-focused report  Public-focused report Issue Preparedness Report in July 26, 2011 Cornelia Orr, NAGB

Questions and Discussion What are the results from the preparedness research so far?  The content of NAEP is similar to the content assessed on common indicators of college preparedness, that is, the ACT, SAT, and ACCUPLACER.  Some content on NAEP is similar to WorkKeys, though the content on NAEP is generally broader than the content on WorkKeys and WorkKeys measures some content that is not in NAEP.  The linking of NAEP to SAT: NAEP proficient is about at 500 on the SAT for both reading and mathematics, the college board readiness benchmark. How can states use the NAEP preparedness research?  As a resource for state and/or consortia standard setting  To provide reference points for state readiness/preparedness definitions 17 July 26, 2011 Cornelia Orr, NAGB

Other Questions? Dr. Cornelia S. Orr Executive Director National Assessment governing Board 800 North Capitol Street NW, Suite 825 Washington, DC July 26, 2011 Cornelia Orr, NAGB