Just Read Florida Leadership Conference, July 2006 Quest for Perfection “Steve Jobs” biography by Walter Isaacson Never satisfied with a proposed solution, always looking at it as science and art, always pushing harder for better, faster, more simple and elegant solutions. Florida Department of Education (c) 2006

Florida Comprehensive Assessment Tests and End-of-Course Assessments Just Read Florida Leadership Conference, July 2006 Florida Comprehensive Assessment Tests and End-of-Course Assessments FCAT 2.0 Reading Grades 3 – 10 FCAT 2.0 Mathematics Grades 3 – 8 FCAT 2.0 Science Grades 5 & 8 FCAT Writing Grades 4, 8 &10 Algebra 1 & Geometry Biology Under development: U.S. History, Civics High stakes but multiple measures approach (e.g., graduation, course credit, grade 3 reading, school grades, teacher evaluations). Latter: multiple years and combined with classroom observations. Florida Department of Education (c) 2006

Pervasive Quality Assurance Just Read Florida Leadership Conference, July 2006 Pervasive Quality Assurance Built into every process from planning and designing, to executing Process and outcome documentation Process and outcome audits Transparency Continuous refinement and redesign Discipline and vigilance; trust but verify Follow standard best practices. Mature systems run the risk of complacency as people implementing may not question or, worse, may go through motions without laser focus. Will focus on some challenges and somewhat unique approaches and lessons learned New staff position (replaced an “assistant director”) of process manager, who’s role is to ensure contract requirements, specifications, and all QA processes are being applied. Florida Department of Education (c) 2006

Pervasive Quality Assurance Just Read Florida Leadership Conference, July 2006 Pervasive Quality Assurance Design and Development Test Construction Document Quality Test Security and Administration Scoring Calibration/Equating Reporting Validity Follow Standards for educational and psychological testing, CCSSO TILSA QC Checklist (Jan 2003). Needs companion piece for CBT. Validity refers to the extent to which the inferences made from a test (i.e., that the student knows the material or not) is justified and accurate. Concurrent validity (NAEP and SAT9) Florida Department of Education (c) 2006

Design and Development Just Read Florida Leadership Conference, July 2006 Design and Development Test Development Center Contracted center, staffed by educators Ensures educator involvement & content quality Standing and Rotating Committees (e.g., Content Advisory, STEM Experts, College-ready Standards, …) Annual Report to District Superintendents District educators who served in past year District educators who are in database (statewide ~4200) Request for nominations A core part of system is a serious focus on educator involvement. Transparency, clear communication, presence in state organizations Challenge: involving enough (and sufficiently diverse) educators to permeate the system with real information about test content and processes. Keeping experienced core while continually refreshing involvement. In 2010-11 351 participants, 24 weeks of meetings, and 47 different meetings Florida Department of Education (c) 2006

Test and Item Content Quality Just Read Florida Leadership Conference, July 2006 Test and Item Content Quality HS Meet statistical criteria? Content Advisory Committees Item Review Committees Items are field tested Review and select field-test items Test and Item Specs HS Yes Appropriate? Standards Assessed Clarifications Item Types Examples Example of effort to communicate complex process and educators’ key roles to stakeholders. Used in committee meeting training to orient them to where their role contributes as well as to the extensive involvement of educators. Yes Key Educator Involvement HS Handscoring Florida Department of Education (c) 2006

Just Read Florida Leadership Conference, July 2006 Test Construction Annual training conducted for Department and contractor staff Test Construction Specifications Revised and enhanced annually Incorporates checklists Checklists follow documents during test construction Relentless focus on item quality, overall coherence of test content, statistical quality Example: location of reading passages compared to prior location is strictly controlled Improvement example: Changed reference curve from prior year’s curve to an “ideal curve” generated from 3 years of curves (avoid drift) Florida Department of Education (c) 2006

Just Read Florida Leadership Conference, July 2006 Document Quality Test Development Center Editors Accommodations specialist Editorial and content checklists Variety of staff take each test Intense and synchronized process Checklists and bookmaps circulate with documents, sign off for each reviewer required Florida Department of Education (c) 2006

Just Read Florida Leadership Conference, July 2006 Test Administration Annual “Test Administration Debrief” meeting Review of all products against specifications, including shipment samples Webinars for preparation and training on CBT Tests are administered and monitored by certified educators. Multiple equated core forms of CBT are randomly assigned within classrooms. Shipment samples: shrink wrap exposed writing prompt, packing slip doesn’t match contents Florida Department of Education (c) 2006

Just Read Florida Leadership Conference, July 2006 Test Security Extensive and enhanced test security requirements Test Security Statute, s. 1008.24, F.S. Prohibited Activities Agreement Seating charts, test group codes Test materials chain of custody form Student pledge Data forensics on all grades and tests Some student results automatically invalidated but have an appeals process. Investigations, including Department of Law Enforcement and FDOE Inspector General Must sign and provide certificate number Plege: “I agree that I will not give or receive unauthorized help during this test. I understand that giving or receiving such help during the test is cheating and will result in the invalidation of my test results.” Chain of custody: track test materials at all times, including their location, the dates and times they are handled, and the names of the persons performing various activities involving the materials.  Florida Department of Education (c) 2006

Scoring and Data Quality Just Read Florida Leadership Conference, July 2006 Scoring and Data Quality Specifications and checklists Extensive planning and dry runs with mock data Contractor site monitoring by FDOE Document processing and equating audits Triangulation with FDOE and multiple contractors Additional assistance from FSU for data checking Dry runs include handscoring “pseudoscoring” (test deck) Monitoring includes handscoring sites Florida Department of Education (c) 2006

Handscoring Quality Assurance Just Read Florida Leadership Conference, July 2006 Handscoring Quality Assurance Extensive quality assurance processes are utilized for FCAT Writing handscoring FDOE and educators review, revise, and approve all training and qualifying materials. FDOE staff provide on-site monitoring for training, qualifying, and first full week of scoring One week of “pseudoscoring” External audit and public reporting Budget cuts led to dropping of educator observations of HS Reporting includes provision of all students’ scored essays to schools Florida Department of Education (c) 2006

Just Read Florida Leadership Conference, July 2006 Reporting Results Statute (s.1008.22(c)14, F.S.) dictates testing and reporting timeframes. Cross-agency planning team meets weekly for several months prior to reporting. Hire seasonal data checkers to verify files/paper reports. Choreographed file transfers (contractor, FDOE, web team, data warehouse), version control, analyses, and briefings Reporting: test results for the FCAT must be made available no later than the week of June 8 Testing: FCAT Writing may not be administered earlier than the week of March 1, and a comprehensive statewide assessment of any other subject may not be administered earlier than the week of April 15. EOCs: an administration period for assessments that meets the intent of end-of-course assessments and provides student results prior to the end of the course. Florida Department of Education (c) 2006

Pervasive Quality Assurance Just Read Florida Leadership Conference, July 2006 Pervasive Quality Assurance Built into every process from planning and designing, to executing Process and outcome documentation Process and outcome audits Transparency Continuous refinement and redesign Discipline and vigilance; trust but verify Follow standard best practices. Mature systems run the risk of complacency as people implementing may not question or, worse, may go through motions without laser focus. Will focus on some challenges and somewhat unique approaches and lessons learned New staff position (replaced an “assistant director”) of process manager, who’s role is to ensure contract requirements, specifications, and all QA processes are being applied. Florida Department of Education (c) 2006

More information Websites: fcat.fldoe.org/default.asp www.floridapathtosuccess.org/ Technical: fcat.fldoe.org/fcatpub5.asp : @EducationFL Florida Department of Education FL Department of Education