1. New York State Education Department Understanding The Process: Science Assessments and the New York State Learning Standards

2. January 2002 NYSED New York State Learning Standards April 1994, the Board of Regents approved a plan to revise the State assessment system based on learning standards. July 1996, the Board of Regents approved 28 learning standards in seven standard areas: –Math Science and Technology, Social Studies, The Arts, English Language Arts, Languages other than English, Career Development and Occupational Studies, Family Consumer Science/Health/Physical Education

3. January 2002 NYSED New York State Learning Standards Learning standards outline what students should know, understand and be able to do in a specific subject area Learning standards contain content and performance standards –Content Standard The knowledge, skills, and understandings that individuals can habitually demonstrate over time as a consequence of instruction and experience –Performance Standards Levels of student achievement in domains of study

4. January 2002 NYSED New York State Learning Standards Learning standards consist of performance indicators at the: –Elementary ( K-4), –Intermediate (5-8), and –Commencement (9-12) levels Performance indicators are embedded in the learning standards and are aligned to Science Core Curriculum Guides and State Assessments

5. January 2002 NYSED State Assessments Provide a uniform measure of student achievement across all districts, all schools, all classrooms State tests assess the extent to which students have achieved the learning standards in a content area Are important indicators of student achievement of the learning standards Are used to understand individual student needs in conjunction with other appropriate measures Drive necessary changes in curriculum and classroom instruction

6. January 2002 NYSED Science Assessments Elementary Science Elementary Science Program Evaluation Test (ESPET) Administered at Grade 4 Intermediate Science Intermediate Level Science Administered at Grade 8 Commencement Level Regents Science Exams Living Environment Physical Setting/Earth Science Physical Setting/Chemistry Physical Setting/Physics

7. January 2002 NYSED Test Development Process in Science The test development process ensures assessments created are fair, valid and reliable measures of student performance in relation to meeting the State learning standards The process involves 19 steps and approximately two- three years to develop a State assessment

8. January 2002 NYSED Test Development Process

9. January 2002 NYSED Test Development … continued Review learning standards in subject content area Design test specifications- “test blueprint” Solicit and train Item Writers Publish prototypes of items/generic rubrics; (sample tests) Review and edit submitted items Pre-test items; scan pretests, read and score performance items Perform item analysis; review items and data

10. January 2002 NYSED Test Development … continued Field test forms; scan field tests; read and score performance items Perform item and test analysis Submit to Statewide Examination Review Committees: Sensitivity Review - ensures that all people are depicted in accord to dignity ; certified trained reviewers review or reject test items Bias Analysis - evaluates whether a test question asks the same question and at the same level of difficulty across sub- groups of test takers Determine student performance levels through Standards Setting Study - “cut scores”

11. January 2002 NYSED Test Construction New York State teachers and content consultants, in coordination with Office of State Assessment and Curriculum and Instruction, determine test specifications A “test blueprint” determines the percentage of questions weighted for each standard and key ideas

12. January 2002 NYSED Item Writing New York State Teachers & Content Specialists Are trained as item writers by New York State Education Department staff Align State learning standards contained in Science Core curriculum guides to all test items generated Write items and scoring rubrics for State tests in science

13. January 2002 NYSED Pre-Tests Prospective test items are “pre-tested” by a diverse sample of students across the State Approximately 200 students for each item are tested Results from pre-tested items are statistically analyzed to determine question “item difficulty,” and fairness

14. January 2002 NYSED Field Tests Field test items are developed from pre-test questions and administered in “short forms” to a representative sample of students across the State; (800-1000) Field tests are comparable in difficulty from different test forms based on statistical analysis and student performance

15. January 2002 NYSED Field Tests Each field test form is “equated,” meaning two or more test forms are constructed to cover the same explicit content, conform to the same statistical specifications, and are administered under identical procedures Two or more essentially parallel tests are placed on a common scale -“ equating”

16. January 2002 NYSED Field Tests to Operational Tests “State Assessments,” operational tests, are assembled from Field Test Forms Statistical analysis ensures different test forms are comparable in fairness, validity and reliability Operational tests are placed on a “scale score,” a derived score to which raw scores are converted by numerical transformation; raw scores to standard scores Full length forms are presented to the State Examinations Review Committee for sensitivity and bias review

17. January 2002 NYSED Standard Setting Process State Tests assess the extent to which students have met the learning standards in a content area Although scores for the Regents Exams are placed on a numerical scale based on field test data, there are essentially Three Performance Levels Does not meet the standards Meets the standards Meets the standards with distinction

18. January 2002 NYSED Standard Setting Performance Levels Standard Setting committee members are given definitions of student performance levels Student performance levels are applied to all State assessments that are developed including Regents tests

19. January 2002 NYSED Standard Setting…Three Performance Levels Example: Physical Setting /Earth Science Does Not Meet Learning Standards Meets Learning Standards Meets Learning Standards with Distinction – The student demonstrates, on demand, proficiency, in terms of Physical Setting /Earth Science content, concepts, science skills and basic science knowledge in any or most of the science learning standards and key ideas that are addressed for productive citizenship and has sufficient knowledge and skill for the demands of most work places or post secondary academic environments

20. January 2002 NYSED Setting the “cut score” The Board of Regents has determined “65” as passing a NY State Regents Examination and “85” as passing with distinction “Passing” =“Proficient”, the performance needed to achieve learning standards To determine the “passing score” or “65” a formal Standards Setting study is conducted based on the reasoned judgement of subject matter specialists and student performance data

21. January 2002 NYSED Scoring and Scaling Based on statistics from student pre-test and field test data, items are placed on a logarithmic scale according to item difficulty level and student ability The two points “passing/65” and “passing with distinction/85” are then algebraically mapped to scale, 0- 100 (not raw score but scale score

22. January 2002 NYSED Standards Setting Committee Committee members are: -knowledgeable in the learning standards for science -from public and nonpublic schools -are current and former classroom teachers -represent urban, suburban and rural schools -selected members from business and industry Each member makes individual judgements with respect to the item difficulty, scaling and equating of field tests, and professional expertise

23. January 2002 NYSED Standards Setting Process New York State Teachers and content experts use the “book marking” method in conjunction with professional judgement to set a “cut score” In the “bookmarking” procedure, multiple choice and constructed response items are ordered in terms of their item difficulty Test items corresponding to various points on the scale are presented as examples of test items at that difficulty level The purpose of the items is to illustrate the meaning of the difficulty scale at specific points

24. January 2002 NYSED Standard Setting Process Test items used come from an “anchor” form; a test form upon which all cut points are set and all later forms of the test will be equated Committee members apply their professional judgements to these ordered items A “cut-score”, or performance standard, is a specified point on a scale score, “65”, and is set such that scores at or above that point are acted upon differently from scores below that point

25. January 2002 NYSED Science Regents Scoring and Scaling The Conversion Chart provided for each test administration translates raw scores to scale scores (performance standards) and then maps to a 0-100 scale.

26. January 2002 NYSED Science Regents Examinations Scoring Test administration for each test form is “equated” so that the same “scale score”, represents the same level of achievement Test forms vary somewhat in the mix of easier and more difficult items, resulting in the relationship between the raw score and the scale score also varying from each test administration

27. January 2002 NYSED Science Regents Examinations Syllabus - Based Addressed a selective student population Assessments were designed from course of study Syllabi contained prescriptive content Standards -Based Universal access to all students Assessments are derived from the standards Standards drive the content of the courses designed

28. January 2002 NYSED This item’s difficulty level, based on field test data, was the easiest question on the ES June 2001 exam.

29. January 2002 NYSED Test item 9 on the LE June 2001 exam has an item difficulty level at the passing performance level, “meets the standards”, based on field test data and the standards setting process.

30. January 2002 NYSED This item’s difficulty level, based on field test data, is an example of a test item at the designated passing performance level, “meets the standards”.

31. January 2002 NYSED Test item 30 on the LE June 2001 exam has an item difficulty level at the passing performance level, “meets the standards”, based on field test data and the standards setting process.

32. January 2002 NYSED This item’s difficulty level, based on field test data, is another example of a question on the ES June 2001exam that “meets the standards”.

33. January 2002 NYSED This item’s difficulty level, based on field test data, was one of the most difficult questions on the LE June 2001 exam.

34. January 2002 NYSED This item’s difficulty level, based on field test data, was the most difficult question on the ES June 2001 exam.

35. January 2002 NYSED Science Regents Examinations Old “65”/passing was determined by a “Raw Score” A students score was based on a maximum of 100 points. Test item difficulty varied from each test form New “65”/passing is determined by a “scale score” A student score is derived by converting a raw score to a scale score based on student field test data The item difficulty values represent the same level of difficulty from each test administration

36. January 2002 NYSED Science Regents Examination Number of Students Tested - Total State

37. January 2002 NYSED Regents Science Examinations Statistics

38. January 2002 NYSED Reliability of State Assessments Reliability focuses on the consistency of test scores (performance) for a group of tests takers across measures of time Reliability is best achieved by evaluating the whole test before considering smaller portions of the test Inter- rater reliability is conducted after each test administration (Teams of teachers are provided uniform training and scoring procedures to re-score 10% of the Regents examinations audited)