1. NAEP Achievement Levels Michael Ward, Chair of COSDAM Susan Loomis, Assistant Director NAGB Christina Peterson, Project Director ACT

2. Why Define Achievement Levels and Set Standards? Cut Points on Scale of Assessment Scores Below Basic Basic Proficient Advanced

3. Why Define Achievement Levels and Set Standards? Congressional mandate. Provide context and meaning to various scores achieved by students on an assessment

4. Science and Judgment Statistics, psychometrics, and science clearly play significant roles in standard setting... So does judgment—and yes, even perspectives on things like politics, justice, and resulting costs. The relative impact of these variables can depend on the nature of the stakes attached to a given assessment.

5. NAEP Achievement Levels Congress authorized the creation of the National Assessment Governing Board to set policy for the NAEP and to set performance levels for the assessment in 1988.

6. Achievement Levels Three levels: Basic, Proficient, and Advanced Policy definitions are general statements to give meaning to the levels. Achievement levels definitions are specific statements of what students should know and be able to do in a specific subject at a specific grade.

7. NAEP Policy Definitions Proficient: This level represents solid academic performance for each grade assessed. Students reaching this level have demonstrated competency over challenging subject matter, including subject-matter knowledge, applications of such knowledge to real world situations, and analytical skills appropriate to the subject matter.

8. NAEP Policy Definitions Basic: This level denotes partial mastery of prerequisite knowledge and skills fundamental for proficient work at each grade.

9. NAEP Policy Definitions Advanced: This level signifies superior performance beyond proficient.

10. NAEP Achievement Level Definition for Grade 8 Proficient Performance in Mathematics Eighth-grade students performing at the Proficient level should apply mathematical concepts and procedures consistently to complex problems in the five NAEP content strands.

11. 8 th grade Proficient continued Eighth graders performing at the Proficient level should be able to conjecture, defend their ideas, and give supporting examples. They should understand the connections among fractions, percents,, decimals, and other mathematical topics such as algebra and functions. Students at this level are expected to have a thorough understanding of basic level arithmetic operations—an understanding sufficient for problem solving in practical situations.

12. 8 th grade Proficient continued Quantity and spatial relationships in problem solving and reasoning should be familiar to them, and they should be able to convey underlying reasoning skills beyond the level of arithmetic. They should be able to compare and contrast mathematical ideas and generate their own examples. These students should make inferences from data and graphs; apply properties of informal geometry; and accurately use the tools of

13. 8 th grade Proficient continued technology. Students at this level should understand the process of gathering and organizing data and be able to calculate, evaluate, and communicate results within the domain of statistics and probability.

14. Uses of Achievement Levels Achievement levels are to be the primary means of reporting student performance on NAEP. Cut scores represent what students at each of the three levels of achievement know and can do. Performance is reported as the percentage of students who score at or above each achievement level cut score or within the score range of each achievement level. – Show changes over time in performance.

15. What is a Cut Score? NAEP Score Scale Below Basic Basic Proficient Advanced

16. How are results reported? % of Students Scoring at or above Each Achievement Level Cut Score

17. Basic Below Basic Proficient % Advanced % of Students Performing Within Each Achievement Level

18. What is Standard Setting? Standard setting (achievement levels-setting) is a judgment. Judgments must be well informed. Standard setting procedures are designed to inform panelists regarding the achievement levels definitions (the standards), the assessment, standard setting methodology, and student behavior in assessment situations.

19. Who Develops NAEP Achievement Levels ? Board policy requires that panels represent educators (70%) and non-educators (30%) – Educators shall inclu ‒ de classroom teachers for the grade and subject (55%) – Other educators with knowledge of the subject matter and students at the grade level of the assessment (15%) – Non-educators should have knowledge of the subject matter and students at the grade level of the assessment (30%)

20. NAEP Achievement Levels Panels Panels should be representative – Gender – Race/ethnicity – Geographical regions

21. Evaluations Throughout the Process

22. Round 1 Round 1 Feedback Round 2 Round 2 Feedback Round 3 Round 3 Feedback & Consequences Data Recommendations for Final Cutpoints Final Cutpoints Item Selection Exemplar Items Framework Item Pool Achievement Level Descriptions The Achievement Level Setting Process

23. NAEP Standard Setting Methodology Modified Angoff : Mathematics, Reading, and Writing (1992) Modified 2 Angoff: Geography, U.S. History, and Science (1994; 1996) ACT/NAGB Method: Civics and Writing (1998) Modified Bookmark: Grade 12 Mathematics 2005 Modified Bookmark: Grade 12 Economics 2006

24. Advantages of Mapmark Bookmark allows multiple choice and constructed response items to be judged in the same way. Bookmark is relatively fast and easy to implement. Mapmark adds a spatial dimension to the item maps to show actual placement of items on the score scale. Mapmark adds a holistic component to feedback for greater reliability.

25. Remaining Issues A response probability (RP) is needed to locate the bookmarks on the score scale. – Cutscore depends on RP. – Items may change location on score scale with different RP values. Relationship between mastery and lower borderline performance relative to achievement levels definitions appears cognitively complex.

26. Item Description

27. Grade 12 Economics Achievement Levels Setting Project August 2006-July 2007 TM

28. Proposed Methods Mapmark with Domains Knowledge, Skills and Abilities (KSA) Review Feedback and Domain Information Feedback and Consequences Data Mapmark with Whole Booklet Feedback Knowledge, Skills and Abilities (KSA) Review Feedback and Student Performance Information Feedback and Consequences Data

29. The Standard Setting Project Process: A series of studies aimed at identifying the best method for standard setting in the content area Studies to Date: Domain Development (Mapmark with Domains) Field trial (Mapmark with Whole Booklet Feedback)

30. Methods Methods – Mapmark with Domains Coherent and well-defined Ordered in difficulty Potential panelists are able to consistently classify items into domains – Mapmark with Whole Booklet Feedback Whole booklet feedback was clear to panelists and well-received

31. Project Schedule to Date EventDatePurpose Domain Development Meeting September 17-19, 2006Develop domains for use in Mapmark with domains standard setting method. Domain Item Classification October 13, 2006Evaluate domains. Field TrialOctober 20-21, 2006Evaluate and refine Mapmark with whole booklet feedback. COSDAM MeetingNovember 16-18, 2006Review results of domain development and field trial. Determine two methods for use in the pilot study.

32. Project Schedule: Tasks Remaining Pilot StudyDecember 6-9, 2006Compare two methods. Validity StudiesJanuary 11-13, 2007Assess the validity of the ALS cut scores. COSDAM MeetingFebruary 2, 2007 (Tentative) Review results of pilot and decide on ALS method. NAGB MeetingMarch 1-3, 2007Review results of pilot study and ALS method. EventDatePurpose

33. Project Schedule: Tasks Remaining ALS MeetingMarch 7-10, 2007Set new Grade 12 cut scores and select exemplar items. COSDAM MeetingMay 8, 2007Review ALS and Validity Studies results. Prepare recommendation of cut scores for the full board. NAGB MeetingMay 17-19, 2007Review ALS and Validity Studies results. Action on cut scores. EventDatePurpose

34. Questions? How is the response probability criterion determined? What is the Governing Board’s role in the achievement level setting process? What is the Governing Board’s role in the achievement level setting process? How are the achievement level descriptions developed?

35. NAEP Related Resources For further information on NAEP achievement levels, visit these Web sites:  www.nces.ed.gov/nationsreportcard/set-achievement-lvls.asp?id=rd www.nces.ed.gov/nationsreportcard/set-achievement-lvls.asp?id=rd  www.nces.ed.gov/nationsreportcard/achlevdev.asp www.nces.ed.gov/nationsreportcard/achlevdev.asp  www.nces.ed.gov/nationsreportcard/researchcenter/statemapping.asp www.nces.ed.gov/nationsreportcard/researchcenter/statemapping.asp  www.nces.ed.gov/nationsreportcard/tdw/analysis/2000_2001/describing _achiev_developmental.asp www.nces.ed.gov/nationsreportcard/tdw/analysis/2000_2001/describing _achiev_developmental.asp  www.nagb.org/who-we-are/20-anniversary/bourque-achievement- levels-formatted.pdf www.nagb.org/who-we-are/20-anniversary/bourque-achievement- levels-formatted.pdf