1. Cornelia Orr, Executive Director National Assessment Governing Board 12 th Grade NAEP Preparedness Research Plans and Progress

2. The Board’s Work on Preparedness Has Been Underway for Nearly a Decade 2002 12 th Grade Commission Established 2005 Board Resolution on Reporting 12 th Grade Preparedness 2006 Grade 12 Reading & Mathematics Frameworks Revised 2009 Board Approves Preparedness Research Plan Conduct Research Studies and Develop Preparedness Report 2010-2012  The Governing Board has demonstrated a rigorous, comprehensive, and methodical approach to research and has not presumed the equivalence of college and career preparedness. This research will help inform the definition of preparedness in terms of NAEP. 2 July 26, 2011 Cornelia Orr, NAGB

3. Key Considerations for the Governing Board Statements in NAEP reports must be shaped and supported by validity evidence (validity framework) Recognize NAEP has limitations Reporting on NAEP 12 th grade preparedness may or may not prove to be feasible Consider additional Grade 12 academic areas in the future 3 July 26, 2011 Cornelia Orr, NAGB

4. Purpose: Technical Panel on 12 th Grade Preparedness Research Identify technically appropriate and valid ways to use grade 12 NAEP for reporting preparedness Assist in planning research and validity studies Recommend research designs and priorities for implementation in 2008, 2009, and thereafter Focus on 12 th grade reading and mathematics 4 July 26, 2011 Cornelia Orr, NAGB

5. Technical Panel on 12 th Grade Preparedness Research: Members Michael Kirst (Chair) Stanford University John Campbell University of Minnesota David T. Conley University of Oregon Michael Kane National Conference of Bar Examiners Mark David Milliron Catalyze Learning International Robert Mislevy University of Maryland George C. Thornton, III Colorado State University 5 July 26, 2011 Cornelia Orr, NAGB

6. Panel’s Key Conclusions Reporting on 12 th grade preparedness seems feasible Use multipronged approach to collect validity information Avoid representing 12 th grade NAEP as the single authoritative definition or conception of preparedness Consider the national context related to preparedness 6 July 26, 2011 Cornelia Orr, NAGB

7. Preparedness: Working Definitions Preparedness for College: reading and mathematics knowledge and skills to qualify for placement into entry level college credit courses that meet general education requirements, without the need for remedial coursework in math or reading Preparedness for Workplace Training: reading and mathematics knowledge and skills needed to qualify for a job training program without remediation in math or reading “Preparedness” versus “Readiness” 7 July 26, 2011 Cornelia Orr, NAGB

8. Types of Research Studies Content alignment Judgmental standard-setting Statistical relationship Survey of Postsecondary Institutions Benchmarking Studies 8 July 26, 2011 Cornelia Orr, NAGB

9. Study Interrelationships 9 July 26, 2011 Cornelia Orr, NAGB

10. Data for Preparedness Research 2009 NAEP 12 th grade Reading and Mathematics  National and state level data  11 states voluntarily participated Linkages to state data systems (only 1 in 2009)  Advance planning  Confidentiality assurance processes  Follow-up for post secondary college/career status Planning for 2013 10 July 26, 2011 Cornelia Orr, NAGB

11. Grade 12 NAEP Results for 11 states 11 July 26, 2011 Cornelia Orr, NAGB

12. Research Studies Preparedness StudyStatus Content Comparisons (SAT, ACT, Accuplacer, Work Keys) Complete Judgmental Standard SettingUnderway Statistical Relationships /Linking (SAT, Accuplacer, ACT) Underway Survey of Higher Ed InstitutionsUnderway BenchmarkingPlanned 12 July 26, 2011 Cornelia Orr, NAGB

13. Reference Points on the NAEP Scale (Scenario 1) July 26, 2011 13Cornelia Orr, NAGB

14. Reference Points on the NAEP Scale (Scenario 2) July 26, 2011 14Cornelia Orr, NAGB

15. Reference Points on the NAEP Scale (Scenario 3) July 26, 2011 15Cornelia Orr, NAGB

16. Next Steps Develop summary report of validity evidence  Validity Framework – external reviews Develop preparedness reports  Based on national data analyses  Research-focused report  Public-focused report Issue Preparedness Report in 2012 16 July 26, 2011 Cornelia Orr, NAGB

17. Questions and Discussion What are the results from the preparedness research so far?  The content of NAEP is similar to the content assessed on common indicators of college preparedness, that is, the ACT, SAT, and ACCUPLACER.  Some content on NAEP is similar to WorkKeys, though the content on NAEP is generally broader than the content on WorkKeys and WorkKeys measures some content that is not in NAEP.  The linking of NAEP to SAT: NAEP proficient is about at 500 on the SAT for both reading and mathematics, the college board readiness benchmark. How can states use the NAEP preparedness research?  As a resource for state and/or consortia standard setting  To provide reference points for state readiness/preparedness definitions 17 July 26, 2011 Cornelia Orr, NAGB

18. Other Questions? Dr. Cornelia S. Orr Executive Director National Assessment governing Board 800 North Capitol Street NW, Suite 825 Washington, DC 20002-4233 www.nagb.org 18 July 26, 2011 Cornelia Orr, NAGB