Exploring Alternate AYP Designs for Assessment and Accountability Systems 1 Dr. J.P. Beaudoin, CEO, Research in Action, Inc. Dr. Patricia Abeyta, Bureau of Indian Education NIEA 2008 Annual Conference Seattle, Washington

Session Goals 2 Frame the technical and resource considerations for decision-makers Provide clarification on the regulatory provisions associated with NCLB Share assessment and accountability design features

Guiding Questions 3 urpose How does the purpose for the change/action serve Native children? irection What direction will the change/action provide the educational community? otivation To what degree will the change/action motivate the Native peoples?

30,000ft. SYSTEM VIEW 4

School System Components 5 Academic Content Standards “What do students need to know and be able to demonstrate?” Instruction “How is content organized, provided, modeled, and expanded upon?” Assessment “To what degree is the content expressed accurately by the leaner?” Accountability “Which learning indicators are being reached, showing progress, or remaining unchanged?”

Integration 6 Things to Consider… Changes to one component will have some influence on the remaining components Improvements in one area often result in benefits throughout the system Isolated changes create bulkization, inefficiencies, and internal stressors

7 Leverage

Compliance If you change the academic content standards used for AYP… If you change the assessments used for AYP… If you change how AYP is defined… 8 Then you must provide evidence that addresses Section 1 of the USDE’s Peer Review Guidance Then you must provide evidence that addresses Sections 2-7 of the USDE’s Peer Review Guidance Then you must create an AYP Workbook that address 10 accountability principles.

Overview: Standards and Assessments Peer Review 9 Conditioned on what is changed, evidence must be presented to the BIE and USDE Peer Reviewers… Section 1-Content Standards Section 2-Achievement Standards Section 3-Coherent Assessment System Section 4-Technical Quality Section 5-Alignment Section 6-Inclusion Section 7-Reporting

Overview: AYP Workbook 10 Principle 1-Includes all schools and districts Principle 2-Includes all students Principle 3-Includes an allowable method for AYP determinations Principle 4-Includes annual accountability decisions Principle 5-Includes subgroup accountability Principle 6-Based on academic assessments Principle 7-Has additional indicators Principle 8-Separate decision for reading and mathematics Principle 9-System validity and reliability Principle 10-Includes participation rates Conditioned on what is changed, policy and empirical evidence must be submitted for review by the BIE and USDE…

11 Overview: Review Process

ASSESSMENT 12

Generic Assessment Life Cycle 13

Foundational Components  Academic content standards  Articulate what student are expected to know and be able to do  Grade level expectations in reading/ELA and mathematics  Grade span clusters in science  Contain skill and process knowledge 14

Foundational Components  Academic achievement standards  Performance level descriptors  No less than three achievement levels (with applicable labels)  “Cut-Score” established using a technically valid standard-setting approach (e.g, Modified Bookmark, Body of Work, Teacher Judgment)  Established by subject matter experts (teachers) and other panelists with applicable skills 15

Foundational Components  Assessments aligned to intended purposes  Assessments in grades 3-8 and high school:  Reading/ELA  Mathematics  Science (3 grade clusters)  Standardized administration and scoring procedures 16

Foundational Process  Typically month development process  Documented evidence  Reviewed by external (to USDE) experts  Evaluated evidence against Peer Review Guidance  Resubmissions typical 17

USDE PEER REVIEW GUIDANCE Standards and Assessments 18

Section 1: Academic Content Standards Students know and are able to…  Content standards are:  Adopted in mathematics; reading/ELA  Adopted in science (grade spans)  Coherent and rigorous  Developed by stakeholders 19

Section 2: Academic Achievement Standards The level students have mastered the content is…  Achievement standards are:  Adopted in mathematics, reading/ELA, and science  Assigned achievement levels and content- based descriptors  Established using an appropriate standard- setting procedure  Aligned to the content standards 20

Section 3: Coherent Assessment System The different assessment components within the larger assessment system contribute… A coherent system:  Is based on state and/or local assessments  Provides integrated achievement data  Aggregates only comparable measures  Utilizes multiple measures focused on higher order thinking skills  Implements alternate assessments 21

Section 4: Technical Quality The technical quality of the assessments is… High quality assessments are:  Valid measures based on their purpose  Constructed to measure the content standards  Accurate and consistent  Fair and accessible  Consistent across different forms and events  Standardized in their administration and scoring 22

Section 5: Alignment The assessments reflect the content standards characteristics by… Strong alignment is provided by using: Using procedures to improve alignment Techniques to match the content standards and patterns of emphasis Capture the range and depth of knowledge in the standards Pattern of emphasis Approaches to measure both content and process knowledge Content-based performance descriptors 23

Section 6: Inclusion The assessment system includes all students by… Students participate in the assessments as demonstrate by:  Impact data showing high rates of student participation  Affording students with disabilities participation under accommodated conditions  Accommodating students learning to speak English  Including migrant and other highly mobile students 24

Section 7: Reporting The assessment reports provide accurate and timely information about student achievement by… Student achievement data is:  Reported using valid and credible procedures  Based on all students from valid assessments  Reported at the individual student, subgroup, school, district, and state-levels  Disseminated in a timely manner  Easily understood by parents and the public 25

USDE REVIEW GUIDANCE AYP Workbook 26

Principal 1: Includes All Schools and Districts  Includes all schools and districts  Holds all schools to the same criteria  Incorporates the academic achievement standards  Provides information in a timely manner  Includes report cards  Includes rewards and sanctions 27

Principal 2: Includes All Students  Includes all students  Consistently defines full academic year (FAY)  Includes mobile students 28

Principal 3: Method of AYP Determinations  Applies universal proficiency by as the long-term goal  Uses a method to make subgroup, school, and district AYP determinations  Establishes a starting point with statewide annual measurable objectives (AMOs)  Establishes intermediate goals 29

Principal 4: Includes Annual Decisions  Determines annually the progress of schools and districts 30

Principal 5: Includes Subgroup Accountability  Includes all the required student subgroups  Holds schools and LEAs accountable for the progress of student subgroups  Includes students with disabilities (SWD) and limited English proficient (LEP) students  Includes an established minimum n-count  Includes safeguards to protect student privacy 31

Principal 6: Based on Academic Assessments  Based primarily on academic assessments 32

Principal 7: Includes Additional Indicators  Includes graduate rate for high schools  Includes an additional academic indicators for elementary and middle schools  Establishes valid and reliable additional indicators 33

Principal 8: Includes Separated Decisions for Reading/ELA and Math  Holds students, schools, and districts separately accountable for reading/ELA and mathematics 34

Principal 9: System Validity and Reliability  Produces reliable decisions  Produces valid decisions  Addresses changes in assessments and student populations 35

Principal 10: Includes Participation Rates  Includes a way to calculate the rate of participation on the statewide assessments  Applies the 95% participation criteria to student subgroups and small schools 36

37 Research in Action, Inc. World-Class Educational Quality™ Voice: Fax: Website: