New England Common Assessment Program Fall 2004 Pilot Test Administration Workshop September-October 2004

Workshop Program Welcome and introductions 1.Overview of the assessment program  Questions 2.Description of the fall 2004 pilot test  Questions 3.Pilot test logistics  Questions 4.What’s Next  Questions

An Emerging Vision Cabot School, Vermont, Web Project Artwork N ew E ngland C ommon A ssessment P rogram

How Did We Get Here? It began with No Child Left Behind – but it became a shared vision of high standards and quality assessment.

 Each state must assess students every year in each of grades 3-8, and one grade at high school beginning in the academic year.  Each state is responsible for developing expectations for student achievement in mathematics and reading/language arts in each of grades 3-8; To meet this challenge… January 8, 2002, No Child Left Behind was signed into law.

The New England Compact was instituted in 2002 by the Commissioners of Education of Maine, New Hampshire, Rhode Island, and Vermont. The New England Compact provides a forum for the states to explore idea, build a collective knowledge base, and establish cross-state activities that benefit each state. From this collaboration emerged the New England Common Assessment Program

What are the Advantages of Collaboration?  Developing a customized test at off the shelf prices  Expanded Resources and Improved Quality  Teacher Involvement X 3  Test CoordinatorsX 3  Content Experts X 3  Technical Advisory CommitteeX 3  Bias Review X 3  Commitment to and experience with Item Development and Review X 3

Key Challenges in the Design of the New England Common Assessments  Create a common set of Grade Level Expectations that fairly and validly represent the standards of all three states  Reach agreement and shared vision on how to measure the GLEs  Allow schools, districts and the states to maintain unique approaches to curriculum and instruction  Develop common test standards and cut points that will work in each state’s unique accountability system  Provide accessibility to the assessment for the maximum number of students possible

Based on a year and a half of work with Grade Level Expectation Teams, Content Teams, Item Review Teams and Bias Review Committees, we believe a large-scale common assessment can and should:  Be linked to state and local content standards  Provide information valued at the classroom level by teachers who use this data to change instruction - in other words be INSTRUCTIONALLY RELEVANT  Support the continuum of assessment from classroom to state levels  Meet tough standards of reliability and validity  Be maximally accessible Emerging Principles -

Overview of Test Design Who?  The assessment includes public school students in grades 3-8 in New Hampshire, Rhode Island, and Vermont.  Through explicit planning during test construction and the use of accommodations, the tests will be accessible to all but a very few students.  The common assessment does NOT include each state’s high school assessment, science assessment, alternate assessment or English language proficiency assessment programs.

Overview of Test Design What?  The content, skills, and depth of knowledge contained in the Grade Level Expectations (GLEs) developed jointly by the three states expressly for this assessment program.  Reading and Mathematics tests at grades 3 through 8. Writing tests at grades 5 and 8.  At each grade level, the tests will measure end-of- grade GLEs for the previous year.  Each test will be designed to measure a range of student achievement across four performance levels.

Overview of Test Design When?  A “full-scale” Pilot Test will be administered October 26, 27 and 28, 2004 for Reading and Mathematics, and January for Writing  Tests will be administered in the fall rather than the spring.  Operational testing will begin in October  Testing will occur during a 3-week window at the beginning of October.

Overview of Test Design Why - Why fall testing?  Assessments results will be returned in December (January 2006 in the initial year), followed later by accountability results.  Allows time for interpretation and use of the assessment results for curriculum and instruction improvement during the spring and summer  Allows us to get the results back to the teacher who gave the test  Minimizes impact on instructional time  Allows us to truly test end of grade standards  Provides measurement of long-term learning  Improves compliance with NCLB accountability requirements

WHAT ARE TEST SPECIFICATIONS?  Type of items: multiple-choice, extended response, etc.  Length of test: hours and sessions  Number of test items and points  Distribution of Emphasis  Depth of Knowledge

Overview of Test Specification Each test will include a variety of item types  Multiple-choice  Constructed-response  Short Answer (mathematics and writing)  Extended Writing (writing) Form Follows Function

The New England Common Assessment uses a mixed common and matrix design.  Common Items- Items that are the same for ALL. The scores are based on these items  Matrix Items- Items that are part of equating and field testing. Each form is different. These items don’t “count” on student scores. Overview of Test Specifications

What is Depth of Knowledge?  Levels are focused on the complexity of the item, not on how different students interact with the item  Descriptors in each discipline to guide item development and classification  Levels help define the upper limits and range of items that are “fair game” for an assessment for a given GLE Overview of Test Specifications

Depth of Knowledge Levels  Level 1Recall  Level 2Skill/Concept  Level 3Strategic Thinking  Level 4Extended Thinking From the work of Norman Webb Overview of Test Specifications

Example: Depth of Knowledge Applied to Math  Level 1 involves recall, or the use of a procedure, solving an equation, or applying an algorithm or formula.  Level 2 involves more than one step, demonstrating conceptual understanding through models and explanations, classifying information, and interpreting data from a simple graph. Overview of Test Specifications

Math Example (continued)  Level 3 involves reasoning, planning, or using evidence  Level 4 requires complex reasoning, planning, and thinking over extended periods of time. In mathematics, Level 4 Depth of Knowledge will not be assessed on the state grade level assessments. Overview of Test Specifications

MATHEMATICS TEST SPECIFICATIONS: ITEM TYPES:  multiple-choice (1 points)  Short answer (1 points)  Short answer (2 points)  Constructed response (4 points) NUMBER OF POINTS: 66 Total  Multiple-choice (32 points)  Short answer (18 points)  Constructed response (16 points)

MATHEMATICS TEST SPECIFICATIONS (cont.) Test Length:  Three testing sessions of approximately 1 hour each. Depth of Knowledge:  Levels 1 – 3 are measured on the assessment.

Distribution of Emphasis for Mathematics Assessment: 2(3) indicates end of grade 2 tested beginning of grade 3 Approximate Distribution of Emphasis 2 (3)3(4)4(5)5(6)6(7)7(8) Number and Operations 55%50% 45%30%20% Geometry and Measurement 15%20% 25% Algebra and Functions 15% 30%40% Data, Statistics, & Probability 15% 100%

READING TEST SPECIFICATIONS: ITEM TYPES:  2 long passages with 8 multiple-choice items and 2 constructed-response items  2 short passages with 4 multiple-choice items and 1 constructed-response item  4 stand alone multiple-choice items

READING TEST SPECIFICATIONS (cont.): Number Of Points: 52 Total  multiple-choice (28 points)  constructed response (24 points) Test Length:  Three sessions of about 1 hour each. Depth of Knowledge:  Levels 1 - 3

Distribution of Emphasis for Reading Assessment Reading Content Clusters 2/ Begin 3 3/ Begin 4 4/ Begin 5 5/ Begin 6 6/ Begin 7 7/ Begin 8 Balance of Representation Balance of Representation Balance of Representation Balance of Representation Balance of Representation Balance of Representation Word Identification 20%15%---- Vocabulary 20% Initial Understanding Lit. 20% 15% Initial Understanding Info 20% Analysis and Interpretation Literary 10%15%20% 25% Analysis and Interpretation Informational 10% 20% Reading Assessment TOTALS 100%

Overview of Test Design Improved Accessibility through Universal design

 Inclusive assessment population  Precisely defined constructs  Accessible, non-biased items  Amenable to accommodations  Simple, clear, and intuitive instructions and procedures  Maximum readability and comprehensibility  Maximum legibility Overview of Test Design Improved Accessibility through Universal design

Assessment Development Timeline ActivityWhenWho Develop GLEs and Test Specs 1/03 – 9/03DOEs, Committees, NCIEA Issue RFP and Select an Assessment Contractor 10/03 – 1/04DOEs Project Planning1/04 – 2/04DOEs, MP

Assessment Development Timeline ActivityWhenWho Preliminary Item Development 2/04 – 3/04MP, DOEs First Item Review Committee Meetings 3/1/04Committees, DOEs, MP Reading/Writing Preliminary Passage Selection 3/04MP, DOEs Bias/Sensitivity Committee Review of Passages 3/04DOEs, Committee, MP

Assessment Development Timeline ActivityWhenWho Second Item Review Committee Meetings 4/27/04 – 4/28/04 Committees, DOEs, MP, NCIEA Item Development5/04 – 6/04MP, DOEs Third Item Review Committee Meetings 7/6/04 – 7/9/04Committees, DOEs, MP, NCIEA Bias/Sensitivity Committee Review Meeting 7/14/04 – 7/15/04 Committee, DOEs,MP Final Item Review & Selection 7/04 – 8/04DOEs, MP

Assessment Development Timeline ActivityWhenWho Field Test Forms Construction/ Production 7/04 – 9/04DOEs, MP Ship Materials to Schools By 10/20MP Reading & Math Field Test Administration 10/26/04 – 10/28/04 All Schools, DOEs, MP Writing Field Test Administration January, 2005Schools, DOEs, MP

Fall 2004 NECAP Pilot Testing Purpose:  Provide an opportunity to field-test all of the items  Further refine the item sets  Guide selection of items for operational tests  Try out all the planned testing procedures, manuals, shipping/receiving, etc.  Give schools an opportunity to experience the assessment prior to the 1 st operational administration in October 2005  Give schools an opportunity to provide feedback (via student and teacher questionnaires)

Fall 2004 NECAP Pilot Testing When?  Reading and math will be piloted on October 26, 27 and 28, 2004  Writing will be piloted in January, 2005  Pilot testing materials will be shipped via UPS on October 15 th and delivered to schools by October 20 th  Completed tests and materials should be packed for UPS pick-up by 8:00 AM on Monday, November 1 st. Schools DO NOT need to contact UPS

Fall 2004 NECAP Pilot Testing Who?  All public schools in NH, RI and VT with any of the grades 3 through 8 will participate  Each grade in a school will be assigned one content area test  Schools selected to pilot the writing test WILL NOT administer a test at grades 5 and 8 in October

Fall 2004 NECAP Pilot Testing Who? Because the results of the pilot will be used to judge the accessibility of the assessment for all students, it is important to include as many students as possible in the NECAP Pilot Assessment, including students with disabilities and English Language Learners

Fall 2004 NECAP Pilot Testing Who?  All students enrolled in grades 3 through 8 as of October 26 must participate, with the following exceptions:  Students who would normally participate in an alternate assessment  English Language Learners enrolled in a US school for less than one year  Students who are absent for the entire pilot testing window  Students whose extraordinary personal circumstances prevent them from participating  DO NOT contact the DOE to report or request exceptions

Fall 2004 NECAP Pilot Testing Who?  Students who are Blind or Visually Impaired  Large print forms WILL be available for the pilot  Braille forms WILL NOT be available for the pilot  Students who need accommodations  Approved Accommodations list is available in the Principal/Test Coordinator Manual  Available to all students based on individual need  Informal decision by educational team, consistent with past practice or current needs

Fall 2004 NECAP Pilot Testing What? - “We have a field trip scheduled for October 26 th. What should we do?”  You may administer two sessions of the pilot test on the same day, one in the AM and one in the PM  Make sure to give the sessions in the correct order  DO NOT administer any sessions prior to October 26th

Fall 2004 NECAP Pilot Testing What? - What should we do if students can’t finish in the stipulated time?  Testing times are estimated. Allow students who are working productively to continue up to 100% additional time.  Because testing times are estimated, don’t schedule testing right before lunch, recess, or dismissal

Fall 2004 NECAP Pilot Testing What? - What should we do if a student is absent?  Make up sessions should be scheduled only when an entire class misses a scheduled pilot testing session.  In the event of individual student absences, it is not necessary to schedule a make-up session.  DO NOT administer any sessions prior to October 26th

Fall 2004 NECAP Pilot Testing What? - What scores will we receive from the pilot tests?  No school or student scores will be generated from pilot testing  Purpose of the pilot is to gather information about the items and administration procedures  We are testing the test, not the students

The Purpose of this workshop is to ensure:  Everyone understands the administration procedures for the Pilot Tests  That NECAP is administered in a comparable way in all locations across the three states  That the information collected is of high quality  Quality control in returning materials

Important Contact Information Amanda Smith – NECAP Program Assistant Phone ext Monica Frattaroli – NECAP Program Manager Phone ext

Important Contact Information Harold Stephens – NECAP Program Director Phone ext Timothy Crockett – Assistant Vice President Phone ext –

Important Contact Information Measured Progress Service Center

Important Dates October 20: Receive/Inventory Materials October 26-28: Pilot Test Administration (Reading and Mathematics) November 1: UPS pickup of test materials for return to Measured Progress (materials must be ready at 8 AM)

Checklist for Principals and Test Coordinators (ii) Before Testing During Testing After Testing

Test Coordinator’s Responsibilities (4) Primary responsibilities are to serve as contact person with Measured Progress coordinate all test related activities prepare teachers for administration oversee the inventory, distribution, collection, and return of all test materials ensure test security and compliance with administration procedures

Test Security (5) All test items and responses to those items in the NECAP are secure material and may not be copied or duplicated in any way or retained in the school after testing is completed.

Test Security (5) Breaches in Test Security Any concern about breaches in test security must be reported immediately to the test coordinator and/or principal. The test coordinator and/or principal is responsible for immediately reporting the concern to the state director of assessment at the department of education.

Before Testing(5)

Preparation for Test Administration (5) Read the Principal/Test Coordinator and Test Administrator Manuals Student Participation and Accommodations Who Should Be Tested Determining How Students Will Participate Using Accommodations Other Accommodations Document Accommodations

Preparation for Test Administration (6) Scheduling Test Sessions Prior to Test Administration Designating Test Administrators Briefing Test Administrators

Test Materials (10) Inventory Test Materials Quantities of Test Booklets and Response Booklets Quantities of Other Materials Ordering Additional Materials Storing Test Materials

Test Materials (13) School Materials Providing Necessary Equipment and Materials Equipment and Materials Prohibited During Test Administration Summary of Test Materials

During Testing(14)

Test Administration (14) Distributing Test Materials Monitoring Test Administration

After Testing(15)

Preparation of Test Materials for Return (15) Collecting Materials after Testing Completing Student Information on Student Response Booklets Special Education Completing the Principal’s Certification of Proper Test Administration Form Return of Materials

QUESTIONS & ANSWERS

BUILDING THE TEST What’s Next?  Review data from pilot tests;  Construct forms for October 2005 testing;  Develop and revise additional items;  Develop a practice test in reading, writing, and mathematics  Develop additional support materials  Design report formats  Develop scoring rubrics  Set standards after first administration  Finalize details of Fall Testing  Make accountability decisions

Questions, Comments, Suggestions:  Tim KurtzDirector of Assessment NH Department of Education (603)  Mary Ann SniderDirector of Assessment and Accountability Rhode Island Department of Elementary and Secondary Education (401) ext  Michael HockDirector of Educational Assessment Vermont Department of Education (802)