Determining Validity For Oklahoma’s Educational Accountability System Prepared for the American Educational Research Association (AERA) Oklahoma State Department of Education April 15, 2004

Education Perspective for Oklahoma, All Local Education Agencies (LEAs) in the State of Oklahoma participate in Title I programs. There are 541 total LEAs 44 LEAs with a student enrollment of 2,500 or greater 201 LEAs with a student enrollment between 499 and 2, LEAs with a student enrollment of less than 500

Oklahoma Demographics, 2003 Total school districts: 541 Independent (K-12): 430 Elementary (K-8): 111 Total School Sites: 1791 Teachers: 47,259

Oklahoma Demographics, 2003 Average Daily Membership: 618,358 Special Education (K-12): 13.6%83,812 Alternative Education (K-12): 2.5%15,312

Oklahoma Demographics, 2003 American Indian/Alaskan: 17.9% Asian/Pacific Islander: 1.5% Black/Non-Hispanic: 10.9% Hispanic: 7.0% White Non-Hispanic/Other: 62.6%

State Standards, Assessments & Accountability Timeline Oklahoma Education Reform and Funding Act of Initial development of core curriculum with statutory requirement for review and revision every 3 years

Legislation mandated the development of Criterion-Referenced Tests (CRTs) in seven (7) content areas for Grades 5, 8, and First review and revision of core curriculum, Priority Academic Student Skills (PASS) Implementation of CRTs in Grades 5, 8, and 11 State Standards, Assessments & Accountability Timeline

PASS revisions – 1997, 2000, 2003 Revisions have provided more specific, detailed and clear standards PASS includes the Oklahoma School Testing Program (OSTP) “blueprints” for the CRTs State Standards, Assessments & Accountability Timeline

Low-performing, high challenge schools identification process adopted in state law; includes sanctions and technical assistance Academic Performance Index (API) adopted in state law; includes API Awards State Standards, Assessments & Accountability Timeline

Oklahoma’s Accountability System To guide instruction To measure achievement To promote a climate of change Proficiency for all Content and Performance Standards Assessments Additional Indicators Decision Rules Sanctions and Rewards

Validity Questions for Oklahoma’s Accountability System Do our standards help guide instruction? Do our assessments and performance indicators measure performance standards (i.e. achievement)? Do our decisions rules accurately identify schools? Do our sanctions and rewards promote change towards intended outcomes?

Key Elements in Determining Validity and Reliability 1.Ensure that the system has provided the intended outcomes. 2.Conduct research on additional information to corroborate findings. 3.Analyze design and implementation of each component of the system. 4.Conduct Analysis on several levels.

Sources of External Evidence Studies and research using additional indicators to corroborate findings or results of the system. Surveys on attitudes and opinions. Outside reviews for components of system.

Sources of Internal Evidence Ongoing review of policies and procedures. Data analysis for various levels such as State level, district level, and for particular types of schools. Evidence of Quality Control measures and Data Audits. Trend analysis over time.

PASS - external reviews have included: : Achieve, Inc : Southwest Educational Development Laboratory (SEDL) : Technical Issues in Large Scale Assessment (TILSA) : “Quality Counts” report cards by Education Week Oklahoma’s Steps Towards Validation Studies and Reviews

Studies and Reviews TILSA Review: Curriculum and Assessment Alignment Study ( ) with Dr. Norman Webb, University of Wisconsin Results of the study enabled Oklahoma to conduct more in-depth alignment of standards and assessments Provided common language to facilitate communication in building curriculum and designing aligned assessments

Research Study ( ) by Dr. John Poggio, University of Kansas Study to determine levels of cognitive complexity of multiple choice test items Studies and Reviews

Implementation Survey (January 2002) After initial implementation of the API an attitude and opinion survey was collected. Additional surveys specifically on Adequate Yearly Progress (AYP) will be collected in October Studies and Reviews

Study on School Improvement Schools –Collecting additional local LEA assessments from additional grades. Purpose: –To corroborate the results of the decision rules. –To look at trends of schools being identified as school improvement. Studies and Reviews Ongoing Study

Grants and Projects No Child Left Behind (NCLB) Enhanced Assessment Grant Oklahoma selected as lead state in collaborative curriculum/assessment alignment project with Council of Chief State School Officers (2003) Continues the alignment work done in 2001 Includes vertical alignment of curriculum and assessments (Grades 3-8) Includes alignment of special education alternate assessments

Grants and Projects NCLB English Language Development Enhanced Assessment Grant Collaborative multi-state project to develop assessment materials for English Language Proficiency Tests

Grants and Projects Project with Council of Chief State School Officers (SCASS) Develop survey tools to determine the level of implementation of a state’s standards-based language arts curriculum To provide support for teachers in meeting NCLB requirement to increase student achievement

Policies A few examples of changes or development as a result of reviewing the system. Developed policy for appeals of Adequate yearly progress. Developed a Technical Advisory Committee to review and offer expertise in regards to technical issues of the State testing program. Implemented new rules and regulations regarding data audits and security.

AYP Statistics 367 (20.5%) of Oklahoma Schools did not make AYP. 198 (36.5%) of Oklahoma School Districts did not make AYP. 1.7 is the average number of subgroups that did not make AYP by school. 1.5 is the average number of subgroups that did not make AYP by district.

AYP Statistics Total number of subgroups not meeting specific performance targets, ordered from greatest to least. –Reading target –Math target –Additional Indicator (attendance or graduation) –Participation target

AYP Statistics Total number of subgroups not meeting specific performance targets, ordered from greatest to least. –Reading target –Math target –Additional Indicator (attendance or graduation) –Participation target