1. Just Read Florida Leadership Conference, July 2006
Quest for Perfection
“Steve Jobs” biography by Walter Isaacson
Never satisfied with a proposed solution, always looking at it as science and art, always pushing harder for better, faster, more simple and elegant solutions.
Florida Department of Education (c) 2006

2. Florida Comprehensive Assessment Tests and End-of-Course Assessments
Just Read Florida Leadership Conference, July 2006
Florida Comprehensive Assessment Tests and End-of-Course Assessments
FCAT 2.0 Reading Grades 3 – 10
FCAT 2.0 Mathematics Grades 3 – 8
FCAT 2.0 Science Grades 5 & 8
FCAT Writing Grades 4, 8 &10
Algebra 1 & Geometry
Biology
Under development:
U.S. History, Civics
High stakes but multiple measures approach (e.g., graduation, course credit, grade 3 reading, school grades, teacher evaluations). Latter: multiple years and combined with classroom observations.
Florida Department of Education (c) 2006

3. Pervasive Quality Assurance
Just Read Florida Leadership Conference, July 2006
Pervasive Quality Assurance
Built into every process from planning and designing, to executing
Process and outcome documentation
Process and outcome audits
Transparency
Continuous refinement and redesign
Discipline and vigilance; trust but verify
Follow standard best practices.
Mature systems run the risk of complacency as people implementing may not question or, worse, may go through motions without laser focus.
Will focus on some challenges and somewhat unique approaches and lessons learned
New staff position (replaced an “assistant director”) of process manager, who’s role is to ensure contract requirements, specifications, and all QA processes are being applied.
Florida Department of Education (c) 2006

4. Pervasive Quality Assurance
Just Read Florida Leadership Conference, July 2006
Pervasive Quality Assurance
Design and Development
Test Construction
Document Quality
Test Security and Administration
Scoring
Calibration/Equating
Reporting
Validity
Follow Standards for educational and psychological testing, CCSSO TILSA QC Checklist (Jan 2003). Needs companion piece for CBT.
Validity refers to the extent to which the inferences made from a test (i.e., that the student knows the material or not) is justified and accurate. Concurrent validity (NAEP and SAT9)
Florida Department of Education (c) 2006

5. Design and Development
Just Read Florida Leadership Conference, July 2006
Design and Development
Test Development Center
Contracted center, staffed by educators
Ensures educator involvement & content quality
Standing and Rotating Committees (e.g., Content Advisory, STEM Experts, College-ready Standards, …)
Annual Report to District Superintendents
District educators who served in past year
District educators who are in database (statewide ~4200)
Request for nominations
A core part of system is a serious focus on educator involvement. Transparency, clear communication, presence in state organizations
Challenge: involving enough (and sufficiently diverse) educators to permeate the system with real information about test content and processes. Keeping experienced core while continually refreshing involvement.
In participants, 24 weeks of meetings, and 47 different meetings
Florida Department of Education (c) 2006

6. Test and Item Content Quality
Just Read Florida Leadership Conference, July 2006
Test and Item Content Quality HS
Meet statistical criteria?
Content Advisory Committees
Item Review Committees
Items are field tested
Review and select field-test items
Test and Item Specs HS
Yes
Appropriate?
Standards
Assessed
Clarifications
Item Types
Examples
Example of effort to communicate complex process and educators’ key roles to stakeholders. Used in committee meeting training to orient them to where their role contributes as well as to the extensive involvement of educators.
Yes
Key
Educator Involvement HS
Handscoring
Florida Department of Education (c) 2006

7. Just Read Florida Leadership Conference, July 2006
Test Construction
Annual training conducted for Department and contractor staff
Test Construction Specifications
Revised and enhanced annually
Incorporates checklists
Checklists follow documents during test construction
Relentless focus on item quality, overall coherence of test content, statistical quality
Example: location of reading passages compared to prior location is strictly controlled
Improvement example: Changed reference curve from prior year’s curve to an “ideal curve” generated from 3 years of curves (avoid drift)
Florida Department of Education (c) 2006

8. Just Read Florida Leadership Conference, July 2006
Document Quality
Test Development Center
Editors
Accommodations specialist
Editorial and content checklists
Variety of staff take each test
Intense and synchronized process
Checklists and bookmaps circulate with documents, sign off for each reviewer required
Florida Department of Education (c) 2006

9. Just Read Florida Leadership Conference, July 2006
Test Administration
Annual “Test Administration Debrief” meeting
Review of all products against specifications, including shipment samples
Webinars for preparation and training on CBT
Tests are administered and monitored by certified educators.
Multiple equated core forms of CBT are randomly assigned within classrooms.
Shipment samples: shrink wrap exposed writing prompt, packing slip doesn’t match contents
Florida Department of Education (c) 2006

10. Just Read Florida Leadership Conference, July 2006
Test Security
Extensive and enhanced test security requirements
Test Security Statute, s , F.S.
Prohibited Activities Agreement
Seating charts, test group codes
Test materials chain of custody form
Student pledge
Data forensics on all grades and tests
Some student results automatically invalidated but have an appeals process.
Investigations, including Department of Law Enforcement and FDOE Inspector General
Must sign and provide certificate number
Plege: “I agree that I will not give or receive unauthorized help during this test. I understand that giving or receiving such help during the test is cheating and will result in the invalidation of my test results.”
Chain of custody: track test materials at all times, including their location, the dates and times they are handled, and the names of the persons performing various activities involving the materials. 
Florida Department of Education (c) 2006

11. Scoring and Data Quality
Just Read Florida Leadership Conference, July 2006
Scoring and Data Quality
Specifications and checklists
Extensive planning and dry runs with mock data
Contractor site monitoring by FDOE
Document processing and equating audits
Triangulation with FDOE and multiple contractors
Additional assistance from FSU for data checking
Dry runs include handscoring “pseudoscoring” (test deck)
Monitoring includes handscoring sites
Florida Department of Education (c) 2006

12. Handscoring Quality Assurance
Just Read Florida Leadership Conference, July 2006
Handscoring Quality Assurance
Extensive quality assurance processes are utilized for FCAT Writing handscoring
FDOE and educators review, revise, and approve all training and qualifying materials.
FDOE staff provide on-site monitoring for training, qualifying, and first full week of scoring
One week of “pseudoscoring”
External audit and public reporting
Budget cuts led to dropping of educator observations of HS
Reporting includes provision of all students’ scored essays to schools
Florida Department of Education (c) 2006

13. Just Read Florida Leadership Conference, July 2006
Reporting Results
Statute (s (c)14, F.S.) dictates testing and reporting timeframes.
Cross-agency planning team meets weekly for several months prior to reporting.
Hire seasonal data checkers to verify files/paper reports.
Choreographed file transfers (contractor, FDOE, web team, data warehouse), version control, analyses, and briefings
Reporting: test results for the FCAT must be made available no later than the week of June 8
Testing: FCAT Writing may not be administered earlier than the week of March 1, and a comprehensive statewide assessment of any other subject may not be administered earlier than the week of April 15.
EOCs: an administration period for assessments that meets the intent of end-of-course assessments and provides student results prior to the end of the course.
Florida Department of Education (c) 2006

14. Pervasive Quality Assurance
Just Read Florida Leadership Conference, July 2006
Pervasive Quality Assurance
Built into every process from planning and designing, to executing
Process and outcome documentation
Process and outcome audits
Transparency
Continuous refinement and redesign
Discipline and vigilance; trust but verify
Follow standard best practices.
Mature systems run the risk of complacency as people implementing may not question or, worse, may go through motions without laser focus.
Will focus on some challenges and somewhat unique approaches and lessons learned
New staff position (replaced an “assistant director”) of process manager, who’s role is to ensure contract requirements, specifications, and all QA processes are being applied.
Florida Department of Education (c) 2006

15. More information
Websites: fcat.fldoe.org/default.asp Technical: fcat.fldoe.org/fcatpub5.asp Florida Department of Education FL Department of Education