Sharon Ford Schattgen, Ph.D. November 7, 2012 Presentation Prepared for the Missouri Charter School Association’s Seminar for Governing Board Members A Close Look at the Indices that Governing Board Members Should Use to Evaluate Charter School Quality

Types of Performance Data Used to Evaluate School Quality  Missouri Assessment Program Data  MAP Grade-Level Assessment & End-of-Course Assessment Proficiency Rates  TerraNova Median National Percentile Ranks  MAP Scale Scores Averages, Arrayed by Cohort  MAP Index Scores  MAP Growth Estimates  Other Summative Standardized Assessment Data  Additional Standardized Achievement Test Scores (e.g., Stanford, ITBS, etc.)  College Readiness/Placement Data (e.g., ACT/SAT Scores, Advanced Placement Scores)  Local Interim Assessment Data (e.g., NWEA Assessment Data)  Additional Performance Measures  Attendance, Dropout, & Graduation Rates  Post-Secondary Success Data  Accountability Data  Missouri School Improvement Program Annual Performance Report (MSIP APR)  Information Associated with Special Designations 2

School X Demographic Data Enrollment/S chool Year Total Black270 (94.7%) 332 (94.9%) 407 (93.8%) 472 (93.7%) 540 (93.9%) Hispanic2 (.7%) 2 (.6%) 2 (.5%) 3 (.6%) Not reported White13 (4.6%) 16 (4.6%) 25 (5.8%) 28 (5.6%) 31 (5.4%) Free/Reduce d Price Lunch 227 (85%) 321 (96.1%) 393 (92%) 467 (92.3%) 514 (89.4%) Free/Reduced Price Lunch Percentages for Corresponding Years: Comparison District = 72.3%, 68.7%, 83.8%, 85.7%, 87.4%; MO = 42.1%, 43.7%, 46.9%, 47.8%, 49.5% 3

Note 1: Disaggregation of data by “subgroup” is required! 4 One of the goals of the No Child Left Behind Act is to eliminate discrepancies in achievement across “subgroups” of students, which means Governing Board members should examine data that are disaggregated to allow you to see any “gaps” in performance. In other words, you need to review data that are disaggregated by racial or ethnic group, free/reduced-price lunch status, special-education status, and limited English proficiency status, and, when appropriate, gender.

Note 2: Most of the data Governing Board members will receive will not be presented in its original format. Rather, it will likely be summarized and reformatted for the purposes of creating an annual report. School personnel access their data from the DESE MCDS Portal, and test publishers also provide electronic and print reports. School Report Cards are posted at 5

Note 3: We assess achievement for specific purposes; here are 5 such purposes. 1.Provide summary information about what students know and can do relative to course objectives or content/process standards 2.Yield information on an interim basis to inform policymaker and/or educator decisions at classroom, school, or district level 3.Collect student-centered information to guide minute-by- minute and/or day-to-day instruction 4.Place students in specific programs/services 5.Identify students who need further evaluation Purposes 3, 4, and 5 (listed in green, italic type) yield data that are not appropriate for inclusion in a report to a Governing Board. Data collected for these purposes have utility only for educators. 6

Frequency of administration Scope (curriculum focus) & duration of cycle (time frame) Summative Interim (instructional, evaluative, predictive) aka “common” or “benchmark” Formative (minute by minute and/or day-to-day integrated into lesson) aka “classroom” Tiers of Achievement Assessment Perie, Marion, & Gong, 2009 Increasing

A Reminder: Types of Performance Data Used to Evaluate School Quality  Missouri Assessment Program Data  MAP Grade-Level Assessment & End-of-Course Assessment Proficiency Rates  TerraNova Median National Percentile Ranks  MAP Scale Scores Averages, Arrayed by Cohort  MAP Index Scores  MAP Growth Estimates  Other Summative Standardized Assessment Data  Additional Standardized Achievement Test Scores (e.g., Stanford, ITBS, etc.)  College Readiness/Placement Data (e.g., ACT/SAT Scores, Advanced Placement Scores)  Local Interim Assessment Data (e.g., NWEA Assessment Data)  Additional Performance Measures  Attendance, Dropout, & Graduation Rates  Post-Secondary Success Data  Accountability Data  Missouri School Improvement Program Annual Performance Report (MSIP APR)  Information Associated with Special Designations 8

ASSESSMENT DATA Missouri Assessment Program (MAP) Grade-Level & End-of-Course (EOC) Assessment Proficiency Rates “MAP Proficiency Rate” is the percentage of students scoring in the top two MAP achievement levels—the Proficient Level and the Advanced Level. Following a brief introduction to the MAP, there is a sample line graph showing trends in MAP Grade-Level Proficiency Rates, a sample bar graph showing disaggregated Grade- Level Proficiency Rates for one year, and a sample table showing trends in End-of-Course Proficiency Rates. 9

Missouri Assessment Program (MAP) MAP includes required consecutive grade-level (3-8) assessments in Communication Arts and Mathematics and grade-span (5 and 8) assessments in Science. MAP also includes end-of-course (EOC) exams, including English II, Algebra I, and Biology I. See DESE website for testing requirements. MAP results are analyzed and reported in a variety of ways in order to inform policy and practice and to satisfy federal and state accountability requirements. Performance on each MAP assessment is reported in terms of four achievement levels: Below Basic, Basic, Proficient, and Advanced. The TerraNova tests, which yield National Percentile Ranks, are components of the MAP grade-level assessments. MAP results are also reported using Scale Scores (calculated by test publisher). DESE also calculates MAP Index Scores to use in the MO School Improvement Program (state accountability system). 10

MAP Mathematics Proficiency Rates Grade 7 11

12

MAP End-of-Course (EOC) Exam Proficiency Rates Algebra I School X22.2% N= % N= % N= % N=50 Comparison School 22.7%26.3%41.7%44.8% Missouri52.6%57.3%59.7%56.7% 13

ASSESSMENT DATA TerraNova Median National Percentile Ranks TerraNova Tests are embedded within the MAP Grade-Level Assessments. Each TerraNova yields a National Percentile Rank, which allows us to compare the performance of our students to that of their national grade-level peers. To report group performance, we calculate the Median National Percentile Rank. TerraNova Median National Percentile Ranks for multiple years are shown on the following slide. 14

TerraNova Median National Percentile Ranks School X Communication Arts Grade Grade Grade School X Mathematics Grade Grade Grade School X Science (Grade 8) 2008: : : : : 37 NATIONAL Terra Nova Median Percentile Rank = 50; Range = 1 to 99 15

ASSESSMENT DATA MAP Scale Score Averages, Arrayed by Cohort When arrayed by cohort, MAP Scale Score Averages show the change over time for a given group of students as they move through the grade span (e.g., sixth grade to seventh grade; seventh grade to eighth grade). This type of analysis, shown in the following table, indicates, in a very general way, whether there is student growth in achievement. 16

MAP Scale Score Averages for Grade Cohorts School X Communication Arts Grade 6Grade 7Grade MAP Communication Arts Scale Scores range from 455 to 875 (grades 3-8). 17

ASSESSMENT DATA MAP Index Scores Each year, DESE calculates a MAP Index Score for each content area in each grade span. The Index is calculated by multiplying the percentage of reportable students scoring within a given achievement level by a constant: 6 x % in Below Basic; 7 x % in Basic; 8 x % in Proficient; and 9 x % in Advanced. The resultant products are then summed to produce the Index, which ranges from MAP Index Scores, arrayed over time, are used in the Missouri School Improvement Program as an indicator of whether there is longitudinal improvement in performance. The following table presents an example of MAP Index Scores over time. 18

School Y MAP Index Scores Grade Level Status Grades 3- 5 Math Source: Understanding Your Annual Performance Report, Missouri Department of Elementary & Secondary Education

ASSESSMENT DATA MAP Growth Estimates MAP Growth Estimates are calculated using complex statistical formulae, which utilize each student’s MAP Scale Scores over time—for example, from sixth grade to seventh grade, and from seventh grade to eighth grade. The Department of Elementary and Secondary Education first reported MAP growth data in 2011 The next slide presents an example of MAP growth data. 20

MAP Value-Added Growth Estimates Mathematics State Range: -3 to +3 State Average = 0 *Growth estimates were calculated using longitudinal MAP grade-level assessment data. 21

ASSESSMENT DATA Other Summative Standardized Assessment Data Students may take other summative standardized assessments, in addition to the MAP— assessments such as the Stanford Achievement Test or the Iowa Test of Basic Skills or ACT’s EXPLORE or PLAN. If so, these tests will likely be administered in the fall or the spring of the year. Results are used to inform instructional decisions. A subsequent slide presents an example of data from PLAN, a standardized test that may be administered to 10 th -grade students. 22

Data from Other Standardized Achievement Tests In addition to the MAP, students may take another summative assessment, such as Stanford Achievement Test, Iowa Test of Basic Skills, EXPLORE, or PLAN. Scores from these tests could be reported in terms of: – Median Percentile Rank – Percent of Students Scoring At or Above Grade – Scale Score Average 23

24 In , 67% of CPS students earned Composite Scores at or above the national average.

ASSESSMENT DATA College Readiness/Placement Data The ACT is a set of curriculum-referenced tests that measure students’ readiness for college. ACT Scale Scores range from 1 to 36, and the 2012 national average is A student whose Scale Score in a particular content area meets or exceeds the corresponding ACT College Readiness Benchmark has a high probability of being successful in an entry-level, credit-bearing college course in that subject. Scores from Advanced Placement Exams are used to award college credit or advanced placement in college coursework. Advanced Placement Exam cores range from 1 to 5. The following tables present example ACT data and Advanced Placement data. 25

ACT Scale Score Averages Class of 2012 EnglishMathReadingScienceComposite School X Comparison District Data not available16.5 Missouri National

ACT Data: Class of 2012 Percentage of Test-Takers Scoring at or Above National Average Composite Scale Score of 21 School X20.0% Comparison District 18.2% Percentage of Test-Takers Scoring at or Above College Readiness Benchmark School X Missouri English: 18% Math: 13% Reading: 15% Science: 12% All 4: 12% English: 73% Math: 46% Reading: 56% Science: 33% All 4: 27% Percentage of 2012 graduates taking ACT: School X = 85% Comparison District = 70% MO = 67-75% US = 52% 27

28

ASSESSMENT DATA NWEA Assessment Indices Some charter schools periodically administer classroom-based assessments (such as those published by NWEA, the Northwest Evaluation Association) in order to monitor student progress and to adjust instruction based on student needs. Pre- and post-test NWEA assessment indices for the school year are presented on the following table; however, these data must be interpreted with care because we have no evidence of their technical characteristics (e.g., reliability and validity). However, such data could be appropriate for inclusion in a report to a board because they have the potential to provide information about student learning. 29

Example of Local Interim Assessment: NWEA Indices ReadingMath Grade 6 Percent At/Above Grade Level, Fall %18% Percent At/Above Grade Level, Spring %46% Grade 7 Percent At/Above Grade Level, Fall %20% Percent At/Above Grade Level, Spring %54% Grade 8 Percent At/Above Grade Level, Fall %25% Percent At/Above Grade Level, Spring %42% Grade 9 Percent At/Above Grade Level, Fall %36% Percent At/Above Grade Level, Spring %54% 30

ADDITIONAL PERFORMANCE MEASURES Attendance, Dropout, & Graduation Rates; Post-Secondary Success Data Attendance rates, dropout rates, and graduation rates, along with data about post- secondary success, are shown on the following four slides. These measures of performance complement achievement data. 31

Attendance Rate Example School 91.2%92.6%93.4%94.7%93.9% Comparison District 88.6%91.5%92.0%92.5%93.1% Missouri94.0%94.4%94.2%94.4%94.7% 32

Dropout Rate Example School 0%1.5%1.6%3.4%7.8% Comparison District 13.9%13.8%17.5%23.6%19.5% Missouri3.5% 3.2%3.4% 33

Graduation Rate Academic Year: Example School88.7% Comparison District74.5% Missouri87.7% Graduation Rate 34 Provide trend data when possible.

Graduation Rate Post-Secondary Placement Rate 35 Over 85% of School Y ’s 2010 graduates enrolled in post-secondary institutions, entered the job market in a field related to their career-education coursework, or joined the military.

ACCOUNTABILITY DATA Missouri School Improvement Program: Annual Performance Report The MSIP APR presents evaluative data about districts and charter schools in relation to 14 accountability standards. Schools are held accountable for only those standards pertaining to its grade span. In accordance with established criteria, the Missouri Department of Elementary & Secondary Education can issue designations based on a school’s or a district’s performance. The following two tables summarize School X’s and a comparison district’s 2012 MSIP APRs and provide data supporting a specific designation—in this case as a “focus school.” 36

MSIP Standard/IndicatorSchool XComparison District MAP Grades 6-8 MathNot Met MAP Grades 6-8 Comm ArtsNot Met EOC Algebra I MathNot MetMet EOC English II Comm ArtsNot Met [Bonus MAP Achievement]Not Met [MAP Grade 8 Science]Not Met 9.3 ACTNot Met Advanced CoursesNot MetMet Career Education CoursesNot MetMet College PlacementNot MetMet Career Education PlacementNot MetMet 9.5 Graduation RateMet 9.6 Attendance RateMet 9.7 Subgroup AchievementNot Met 37

School X “Focus School” Indices Percentage of Students Scoring Proficient or Advanced on MAP Assessments (i.e., MAP Proficiency Rate) Three-Year Cumulative Student Gap Group, English/Langua ge Arts 27.5%22.8%20.9%24.0% Student Gap Group, Mathematics 29.0%26.4%29.7%28.3% Combined English/Language Arts & Mathematics Proficiency Rate = 26.2% Targets: Communication Arts = 44.21%; Math = 44.78% 38

And, by the way... The tests, they are a changin’ In , new state assessments will become operational in Missouri—assessments that are referenced to the Common Core State Standards and are designed to measure students’ readiness for college and careers. Two consortia are developing such assessments: * Smarter Balanced * PARCC 39

In closing.... Questions & Comments.... For further information, go to these websites. MAP Assessments & MSIP: Assessment Consortia: