0 Michele Sonnenfeld NAEP State Coordinator Florida Department of Education October 2006 Florida Association of Science Supervisors

1 National Assessment of Educational Progress (NAEP) “ The Nation’s Report Card” NAEP’s two major goals are to: 1.Measure and compare student achievement in states and other jurisdictions 2.Report changes in achievement of 4 th, 8 th, and 12 th graders over time

2 NAEP Has produced National results for grades 4, 8, and 12 since 1969 Has produced state-level results at grades 4 and 8 since 1990 Has produced results for Trial Urban Districts (TUDA) since 2002 Does not produce student-, school-, or district-level results, except for the 10 TUDAs

3 Current Organizational Structure of NAEP US Department of Education Institute of Education Services National Center for Educational Statistics Assessment Division National Assessment of Educational Progress National Assessment Governing Board (NAGB) Contractors: ETS, Westat, AIR, NCS Pearson, GMRI, Hager Sharp, HumRRO, And ESSI

4 Federal Legislative Reference No Child Left Behind Act of 2001 (NCLB) Section 111(c)(2) of NCLB requires states/districts who receive Title I funding to participate in biennial State NAEP assessments of 4 th and 8 th graders in Reading and Mathematics, beginning with the school year.

5 State Legislative Reference Added in 1990 Florida State Statute (2) “The Commissioner of Education shall direct Florida school districts to participate in the administration of NAEP... both for the national sample and for any state-by-state comparison programs.” –State legislation requires all selected schools to participate in all NAEP assessments.

6 NAEP Components STATE (Public) (Grades 4 & 8) National Assessment of Educational Progress Main NAEP Long Term Trend NAEP TUDA (Public) (Grades 4 & 8) NATIONAL (Public & Private) (9-, 13-, & 17-year olds) NATIONAL (Public & Private) (Grades 4, 8, & 12)

7 NAEP Frameworks The American Institute of Research (AIR), with the help of teachers, develops the NAEP frameworks. The National Assessment Governing Board (NAGB) approves and adopts the frameworks after they have been reviewed and critiqued by teachers. Each framework describes the content and format of the 4 th, 8 th, and 12 th grade assessments. There are frameworks for each of the content areas assessed by NAEP.

8 NAEP Science Frameworks Current science framework was used to guide the 1996, 2000, and 2005 assessments A new framework, approved in 2005, will be used to direct future assessments.

9 NAEP Science Frameworks Current science framework requires assessment in –Earth science –Physical science –Life science

10 NAEP Design: Item Development Items are based on the subject area frameworks approved by the National Assessment Governing Board (NAGB). Items are developed by teachers, subject area specialists, and assessment experts. Items meet professional standards for testing. Experts review items for technical/statistical quality, content quality, bias, and sensitivity.

11 How Florida’s School Sample is Selected Step 1: Lists of schools are obtained from the National Center for Education Statistics Common Core of Data (CCD). Step 2: Schools are grouped into strata by two primary characteristics: –Type of location (central city, urban fringe, rural) –Minority enrollment (white, African American, Hispanic) Step 3: Within strata, schools are ordered by a measure related to achievement. Step 4: A proportional sample is selected systematically from the ordered list.

12 Florida’s Measure of Achievement Grade 4 Mathematics average scale scores for all students, grade 4, in 2005 Grade 8 Median household income (because the match rate for achievement scores for 8 th grade was less than 70%)

13 How is the Student Sample Selected? A list of all 4 th or 8 th grade students enrolled at the sampled schools is submitted electronically to Westat from the FLDOE, based on the October FTE data. 90% of selected students at each school must participate in the assessment or make- up sessions are scheduled at schools not meeting this benchmark. Students exempted by their parents are not replaced and are not used to calculate the assessment participation rate.

14 NAEP 2007 NAEP assessments will be administered in Florida from January 22 through February 21, Each student takes only a portion of the questions, answering two 25-minute sections of subject-area questions. Students also answer questions related to their background and their home or school experiences in the subject of their assessment. Assessment is administered by a team from Westat, the NAEP administration contractor for the US DOE, and National Center for Education Statistics (NCES). Parents must be notified prior to the assessment. Refusals must be in writing.

15 Sampling Schools and Students ~350 Florida public schools in 49 districts will participate in some type of NAEP assessment in Grade 4 – Reading and Mathematics: Florida’s sample will include 167 schools. Schools can elect to test all students (up to 120). If there are more than th graders, NAEP will select a sample of 62 students to assess. Grade 8 – Reading, Mathematics, and Writing: NAEP will select a sample of 93 students at each of the 161 Florida schools included in the sample. Grade 12 – Reading, Mathematics, and Writing: Florida’s sample will include 31 high schools, and 75 students from each school will be selected to participate.

16 NAEP Inclusions and Accommodations NAEP has included students needing accommodations since Students with disabilities (SWD) and English language learners (ELL) are to be included in NAEP based on an IEP/504 Accommodations Plan/ELL Plan team decision. Most frequent accommodations with NAEP: 1. Extra testing time 2. Individual or small group administrations 3. Large Print booklets Does not include reading aloud of passages or questions on the NAEP reading assessment. Assessment must be completed in one day. Assessment is not available in Braille unless the district or state provides it.

17 Analysis and Reporting National, State, and Trial Urban District Assessment (TUDA) results –NAEP scores are reported in two ways – by Scale Scores and by Achievement Levels –No student, classroom, or school results are provided –No district results are provided except for the TUDAs –Subgroup, state, and national comparisons can be made –Relationships between student proficiency and certain background variables are available

18 NAEP Scale Scores Scale scores provide a comparison of student performance with regard to a group (e.g., the nation) or a subgroup (e.g., ELL) on a set of items. Scale scores: Reading and Mathematics scale score range is Science and Writing scale score range is Scale scores are reported as an average for groups of students and may obscure progress or problems related to student performance across distribution of scores.

National NAEP Science Average Scale Scores Grades 4 and 8 National NAEP average science scale scores were 4 points higher in 2005 than in 1996 at Grade 4. At Grade 12, scale scores were 3 points lower. There has been no change at grade 8. * indicates change is significant.

20 NAEP 2005 Hands-On Science Tasks Approximately half of the students assessed in science participated in a hands-on task (kits). Grade 4 – Seeds, Unknown Powder, Markers Grade 8 – Markers, Powders, Soil Tests Grade 12 – Soil Tests, Antacid, Pendulum

21 NAEP 2005 Hands-On Science Tasks Students use the kits to conduct an investigation and record their findings in their booklet Kits are self-contained, including whatever safety equipment may be needed Students are instructed how to clean up when they are finished with their kit Certain pieces of the kits are left at the school for their use

22 Florida Compared to the Nation NAEP 2005 Science Scale Scores Grade 4 Of the 44 states and one jurisdiction that participated in the th grade science assessment, students’ average scale scores in Florida were higher than that of 10 jurisdictions, not significantly different from that of 12 jurisdictions, and lower than that of 22 jurisdictions.

23 Florida Compared to the Nation NAEP 2005 Science Scale Scores Grade 8 Of the 44 states and one jurisdiction that participated in the th grade science assessment, students’ average scale scores in Florida were higher than that of 3 jurisdictions, not significantly different from that of 8 jurisdictions, and lower than that of 33 jurisdictions.

24 NAEP Achievement Levels NAEP Achievement Levels (ALs) try to identify what students should know and be able to do at various achievement levels. The NAEP achievement levels are Advanced, Proficient, Basic, and below Basic. Each is determined by cutpoints established by professional educators and NAGB. The number or percentage of students in the population at large or in subpopulations can be reported by the four achievement levels. Results show how different groups are performing in relation to each other and over time.

25 Florida Compared to the Nation NAEP 2005 Science at or Above Proficient Grade 4 Of the 44 states and one jurisdiction that participated in the 2005 Grade 4 science assessment, the percentage of students scoring at or above Proficient in Florida was higher than that of 8 jurisdictions, not significantly different from that of 15 jurisdictions, and lower than that of 21 jurisdictions.

26 Florida Compared to the Nation NAEP 2005 Science at or Above Proficient Grade 8 Of the 44 states and one jurisdiction that participated in the 2005 Grade 8 science assessment, the percentage of students scoring at or above Proficient in Florida was higher than that of 3 jurisdictions, not significantly different from that of 11 jurisdictions, and lower than that of 30 jurisdictions.

27 Florida NAEP Achievement Levels Grade 8 Science 1996 and 2005 Florida’s scores have remained unchanged since 1996.

28 Florida’s Performance Compared with the Nation NAEP 2005 Science Grades 4 and 8 Percentage of States Florida Outperformed Grades 4 and 8 in 2005 Based on Race/Ethnicity

29 Florida’s Performance Compared with the Nation NAEP 2005 Grade 8 Science Percentage of States for which Florida’s Change from 1996 to 2005 was Greater by Race/Ethnicity

NAEP Grade 4 Science Race/Ethnicity Grade 4 Average Scale Scores for Florida and the Nation by Race/Ethnicity

NAEP Grade 4 Science Race/Ethnicity In 2005, the average scale score of African American students was 31 points lower than the average scale score of white students. In the nation, the average scale score for African American students was 33 points lower than the average scale score of white students. In 2005, the average scale score of Hispanic students was 17 points lower than the average scale score of white students. This performance gap was narrower than that of the nation (29 points).

NAEP Grade 8 Science Race/Ethnicity Grade 8 Average Scale Scores for Florida and the Nation by Race/Ethnicity

33 NAEP Grade 8 Science 1996 and 2005 Race/Ethnicity Grade 8 Science Average Scale Scores for 1996 and 2005 for Florida and the Nation by Race/Ethnicity

34 NAEP Grade 8 Science 1996 and 2005 Race/Ethnicity In 2005 and 1996, the average scale score of Florida’s African American students was 37 points lower than the average scale score of white students. The National gap declined from a 39 point difference in 1996 to a 36 point difference in In 2005 and 1996, the average scale score of Hispanic students was 24 points lower than the average scale score of white students. The National gap increased from 30 points to 32 points.

35 NAEP AND FCAT FCAT – measures student performance on selected benchmarks, as defined by Florida’s SSS NAEP – reports on the performance of groups of students at a given time - and across time - without specifying how a subject should be taught

FCAT, Florida NAEP, and National NAEP Science Grades 4 and 8 Achievement Levels FCAT – Level 3 and Above NAEP – Proficient and Above Grade 4 Grade 8

FCAT, Florida NAEP, and National NAEP Science Grades 4 and 8 Percentage of students performing at Proficient and above in science is very similar for FCAT Grade 5 and Grade 4 Florida NAEP and National NAEP. Percentage of students scoring at Proficient and above on the Grade 8 science FCAT is greater than on the Grade 8 Florida NAEP and National NAEP.

FCAT Grade 5 and Florida NAEP and National NAEP Grade 4 Science Race/Ethnicity Achievement Levels FCAT – Grade 5, Level 3 and Above NAEP – Grade 4, Proficient and Above

FCAT Grade 5 and Florida NAEP and National NAEP Grade 4 Science Grade 4/5 students in all three racial/ethnic groups had higher average scale scores in science and a greater percentage achieving Proficient and above on the FCAT than they did on Florida NAEP and National NAEP

FCAT, Florida NAEP, and National NAEP Grade 8 Science Race/Ethnicity Achievement Levels FCAT – Grade 8, Level 3 and Above NAEP – Grade 8, Proficient and Above

FCAT, Florida NAEP, and National NAEP Grade 8 Science A greater percentage of Grade 8 white, African American, and Hispanic students scored Level 3 or above on the FCAT than scored Proficient and above on Florida NAEP and National NAEP

42 The Nation’s Report Card

43 NAEP Websites Florida’s NAEP website: NAEP website: NAEP Data Explorer: NAEP Questions Tool:

44 NAEP Coordinator Contact Information Michele Sonnenfeld NAEP State Coordinator Florida Department of Education (850) SC FAX (850) SC