1. + Monitoring Access Tools and Accommodations in a New Assessment Context Laurene Christensen, Ph.D. National Conference on Student Assessment

2. + What is NCEO? Federally-funded technical assistance center Support SEAs and consortia on issues related to inclusive assessment Students with disabilities ELLs ELLs with disabilities Website: nceo.info

3. + Changing Assessment Context

4. + From Accommodations to Accessibility Tools and Accommodations Accommodations are adjustments to the testing situation, test format, or test administration that provide equitable access during assessments for students with disabilities and students who are English learners.

5. + Smarter Balanced Framework

6. + PARCC Framework

7. + Accommodations on DLM Dynamic Alternate Assessment System Universal design Variety of response formats Compatible with variety of assistive technologies Allows for varying levels of teacher assistance

8. + Accommodations on NCSC Format of assessment (AA-AAS): 4 complexity levels; 30 items per subject; can be delivered online Most items are multiple choice Some constructed response; can use picture symbols or graphic organizers Most accommodations are embedded within items Allowed accommodations are included in test administrators’ scripts

9. + Accommodations on ELPA21 Still being decided Administrative Considerations Universal Features Designated Features Accommodations

10. + Accommodations on ASSETS ELP assessments Computer delivered Accommodations and accessibility features built-in Designed to be appropriate for a range of needs

11. + How do all of these pieces come together?

12. + First steps: The Need for Monitoring Monitoring requirements of IDEA Monitoring requirements from ESEA Expectations from peer review (uncertain right now)

13. + What is monitoring accommodations? Monitoring accommodations is a process of evaluating policies, procedures, and practices in order to ensure that a state’s assessment system is inclusive of students with disabilities and English language learners.

14. + Monitoring is about compliance The State has analyzed the use of specific accommodations for different groups of students with disabilities and has provided training to support sound decisions by IEP teams. The State routinely monitors the extent to which test accommodations are consistent with those provided during instruction. But it is about more than compliance….

15. + Monitoring is about improving outcomes Systematic attention to the provision of accommodations can ensure that students are able to show what they know and can do

16. + Monitoring Accommodations: Overview of Tool  Developed in collaboration with CCSSO  Reviewed extensively by ASES SCASS group  Based on a review of publicly available materials  5 Step process  Each step includes examples, a checklist, additional resources  Appendices include additional examples from states  Extensive glossary at the end

17. + 5 Steps to Monitoring Accommodations 1. Know the rules and regulations for accommodations monitoring 2. Document decisions about access tools and accommodations 3. Document the use of access tools and accommodations 4. Review access tool and accommodations decisions and use 5. Evaluate and report on access tools and accommodations

18. + Document Decisions about Accessibility Tools and Accommodations Keeping track of training Keeping track of decisions General ed students ELLs Students with disabilities Are access tools and accommodations used for instruction consistent with those used for assessments? Keeping track of requests for unique accommodations

19. + Document the use of accessibility tools and accommodations Documenting on the PNP/ISAAP Documenting inappropriate use of tools and accommodations on test day Documenting that students received tools and accommodations on test day

20. + Review accommodations decisions and use Direct observation on test day Online record reviews On-site visits on a day other than test day Interviews with students, teachers, and administrators about the use of accommodations

21. + Evaluate and Report on Accommodations  Analyze accommodations  Report on accommodations  Revise accommodations policies

22. + Questions to Consider How can we support educators in keeping up with the different policies of each assessment? What additional support is needed to effectively implement the PNP? With all of the data these new systems can generate, what are the questions we should ask of the data?

23. + Other Questions? Thank you! Laurene Christensen laurene@umn.edu

24. Beyond the PNP: Designing a Comprehensive State System for Systematically Monitoring Accommodations Melissa Gholson, Coordinator Office of Assessment and Accountability

25. Common Core Essential Elements Our System English Language Proficiency Standards

26. 2014 and Beyond Stakeholders meeting Reviewing current policies, processes and procedures Develop short and long term goals Supporting an ongoing research plan Focus on improving our system

27. A Systems Perspective A Systems Perspectives Research Agenda Peer Review Participation Guidelines & Policies WVEIS Data Process, Codes & Descriptions IEP, 504, SAP and LEP Plans Monitorin g and the WVS 326 Process

28. Improving Accommodations Ensure that your state has a valid method for gathering data on which students are to receive specific accommodations in assessment, and a form to document what they receive on test day. (Electronic WVS.326 Process) Document how your state analyzes its accommodations data, including timelines of analysis. (State & District Reports) Establish a specific monitoring procedure that identifies issues in the selection of accommodations for individual students or the provision of accommodations for instruction or assessment. (Participation Guidelines) Include information on any consequences that result from any irregularities in the administration of assessment accommodations. (Report analysis, investigation team and invalidation procedures) Conduct studies that examine the link between instructional accommodations and assessment accommodations (Research agenda)

29. Accommodations Research Plan 2009 Paper version of monitoring 2010 Pilot Electronic WVS 326 ACT 2011 Secondary Feasibility W2 2012 All WV-Maps ELL Research Development 2013 State & District Reporting 2014 and Beyond Focus on improving the process

30. Accommodations Research – Using last 3 years of W2 data – Will be able to look for trends – The primary purpose of this report is to describe the accommodations assigned to SWD, LEP, and 504 students for the 2011 – 2013 school years. Discussion/Interpretation: How appropriate are the accommodation assignments based upon student subgroup/performance? How do these trends vary across time? – Report may be able to - Describe: Relationship of WT2 to Accommodation Assignments Does WT2 performance predict accommodation assignments? 2010 assessment and 2011 accommodations – Do accommodation assignments influence WT2 performance? 2010 accommodations and 2011 assessment – Examine 2013-2014 online W2 data independently – After 2-3 years may be able to do DIF – Continue analyzing ELL data by ELP proficiency, achievement, by years in program, include ELL who have met exit criteria for two years.

31. Accommodation CodeAccommodation Shorthandn of Students Recommended to Receive Accommodation % of Sample Recommended to Receive Accommodation P02Have test read aloud: Math, Science, SS Tests 13,45658.8% T04Use extra time (not applicable to WESTEST 2) 6,75029.5% P26Have directions/passage and prompt read aloud -- on-line writing 6,47228.3% P15Have directions only read aloud--RLA 5,68624.8% T03Provide more breaks5,08522.2% P18Have directions rephrased by trained examiner 4,38119.1% T07Flexible scheduling4,10417.9% R02Indicate response to a scribe (selected response and on-line writing) 4992.2% R04Indicate response to a scribe (constructed response) 4451.9% P19Use a large-print edition of the test 1390.6% Most Frequent Accommodations for Students with Disabilities

32. Peer Review Requirements Providing an appropriate variety of assessment accommodations Ensuring that the use of assessment accommodations yields meaningful scores Ensuring that the use of accommodations is consistent with instructional approaches, as determined by a student’s plan Determining that assessment scores for students when administrated under accommodated conditions, allow for valid inferences about the students’ knowledge and skills Establishing clear guidelines for including all students in assessment Differentiate between accommodation policies for students with disabilities and English Language Learners.

33. Participation Guidelines The document includes references to current state policies and procedures. The document is updated and revised annually by a stakeholder group

34. Accommodations Review

35. Smarter Balanced Universal Tools Available to all Designated Supports Available for Students with LEP, IEP and 504 Plans (ISAAP) Accommodations Available for Students with IEP and 504 plans. Embedded Into the platform Breaks Calculator Digital Notepad English Dictionary English Glossary Expandable Passages Global Notes Highlighter Keyboard Navigation Mark for Review Math Tools Spell Check Strikethrough Writing Tools Zoom Color Contrast/P28 Masking /P37 Text-to-Speech /P01 Translated Test Directions/P30 Translations(Glossary)/P31 Translations (Stacked)/P32 Turn off Any Universal Tools/P33 American Sign Language/P34 Braille/P03 or P35 Closed Captioning/P36 Text-to-Speech/P13 Non- Embedded Provided locally Breaks English Dictionary Scratch Paper Thesaurus Bilingual Dictionary /R15 Color Contrast/P38 Color Overlay/P39 Magnification/P23 Read Aloud/P02 Scribe/R04 Separate Setting/T09 Translations(Glossary)/P41 Abacus/R05 Alternate Response Options/R11 Calculator/R19 Multiplication Table/R20 Print on Demand/P40 Read Aloud/P14 Scribe/R04 Speech-to-Text/R21

36. Accommodations Process Decision to provide an accommodation is documented in the student plan. The documentation is entered into the WVEIS data system Data must be verified and reviewed prior to assessments Accommodations Application provides the data on a daily delta Accommodations data is collected analyzed, reported to districts and used to inform policies, practices and provide targeted technical assistance.

37. Accommodations Electronic Application Accommodations application supports the process for identifying students who need accommodations Accommodations are updated on a daily delta and are available at the school, district and state level. The accommodations application allows users to filter and sort to by school, student ID, group, name, grade or accommodation. The application allows for printing and the creation of spreadsheets

38. Verification Process Accommodations are verified locally prior to assessment. WVEIS Accommodations App are compared to current student plans The WVS. 326 is reviewed to prepare the testing environment for the student.

39. WVEIS Data System: Accommodations Application

42. Data Sorting &Filtering

44. The WVS.326 Process: Accommodations Provision and Monitoring The WVS.326 form is provided for all state and district assessments. One form for each student receiving identified supports and accommodations. Electronic process which feeds accommodations research information to support better student outcomes. Accommodations data collected informs accommodations policies, practice and research agendas. The WVS.326 process provides feedback on accommodations provision for student plans. Documentation of the WVS. 326 form is available http://wvde.state.wv.us/oaa/videos/WV_326/Documentation%20Pro cedures.zip http://wvde.state.wv.us/oaa/videos/WV_326/Documentation%20Pro cedures.zip

45. WVS.326 The WVS.326 form includes all the accommodations All pre-slugging and data entry one side Directions are on the back Coding process includes data for accommodation refusals and is used for invalidations of over and under accommodations. Information about WVS. 326 process is included in the Participation Guidelines.

46. Let’s Talk About Codes If you mark Provided “Yes”- this means the student was given the identified accommodation. If you mark “No”- this means the student was not provided the identified accommodation. This is an accommodations breach UNLESS it is a: Code 1-the student refused; or Code 2-the accommodation is not allowed/applicable for this test If “no” is not marked with a code 1 or 2, then it constitutes an accommodations error that requires documentation. See the PG for specific details.

47. What to do about students who do not receive accommodations? All incidents for failing to provide an accommodation requires written documentation. The CTC will review all accommodation breaches with principals/building level coordinators and report any findings to the Special Education, Title III and/or 504 director(s) and to the Office of Assessment and Accountability. If a student has been denied accommodations listed within the IEP, 504 or LEP plan, there has been a breach in the integrity/accuracy of test results. Therefore, the CTC or county special education director should contact the principal who must inform the parent/guardian of the testing administration breach and provide the following options: – Test is invalidated and the student is retested using a breach form (if the testing window is still open). – Test is scored. – Test is invalidated and student receives a score of ―0. If the second or third option is selected—there must be a signed written agreement between the parent/guardian and principal/building level coordinator. Copies of the agreement must be kept on file with the CTC/county special education director, Section 504 director, or Title III director.

48. WVS 326 State and District Accommodations Report Analyzed provision data for 2012 WESTEST 2 by content area 7 reports were generated for each content area to summarize (1) provision rate, (2) refusal rate, (3) not allowable rate, and (3) number of students over- accommodated – State, district, school-level reports – Intended uses of these data – Set specific targets for provision rates – Identify common errors to inform your training – Eliminate under- and over-accommodation

49. WVS. 326 Accommodations Reporting 1. Count of Students with Accommodation This report provides the total number of students with the accommodation listed on the front of the 326 form (this number serves as the denominator for all rate calculations in reports 2 – 6). 2. Provision Rate This report provides the percentage of students with the accommodation for whom the accommodation was marked “provided.” 3. Not Allowed Rate This report provides the percentage of students with the accommodation for whom the accommodation was marked “not provided” and the “not allowed” code was provided in the reason field. 4. Refusal Rate This report provides the percentage of students with the accommodation for whom the accommodation was marked “not provided” and the “refused” code was provided in the reason field. 5. Not Provided (no reason indicated) This report provides the percentage of students with the accommodation for whom the accommodation was marked “not provided” but no reason code was provided. 6. Not Provided (blank) This report provides the percentage of students for whom no information was provided on whether or not the accommodation was provided. 7. Count of Students Over- accommodated This report provides the number of students who do not have the accommodation listed in the pre-slug file, but for whom the accommodation was marked “provided.”

50. State Reports

51. District Reports

52. Lessons Learned During the Transition Collaboration is critical. Review and update WVEIS codes and descriptions in all policy and practice documents. The codes we use for a PNP to identify specific students’ needs for example, braille, large print, etc. will correspond to the accommodations data map that must be provided to test vendors. The Participation Guidelines must be updated and cover the parameters of the designated systems such as TIDE and define the allowable universal tools designate supports and accommodations. OAA must provide training to districts, including CTC’s, Special Education Directors, 504 Coordinators, ELL Coordinators, principals/BLC, special education teachers, student assistance teams, teachers and test examiners. The LEP, 504, SAP Plan & the Online IEP system must be reviewed and incorporate available tools, designated supports and accommodations. Communicate plan changes to all impacted, including parents. Student plans must be reviewed, amendments made or meetings held in order to address the changes. The WVS.326 form must be adjusted, a new program created for pre-slugging and reading the form to provide monitoring data for accommodations

53. Melissa Gholson mgholson@k12.wv.us

54. 54 Beyond the PNP: Designing a Comprehensive State System for Systematically Monitoring Accommodations Brian M. Touchette Director Office of Assessment

55. 55 State Profile 3 counties o 19 districts o 25 charters o ~220 schools 130,000 total students (15,000 w/ IEPs) 80,000 students assessed (10,000 w/ IEPs) DE transitioned from paper based assessment to on-line assessment 2008-09 school year

56. 56 On-line IEP State mandated form Accom. Form included Form is printed for student’s audit File State can view any student’s IEP and attached forms Universal design vs. accommodations

57. 57 Accommodations Data-base On-line IEP Paper version of Accom. Form is entered Separates accoms. for each content area Often entered by office admin. Functional vs. Non-functional

58. 58 Functional vs Non-Functional Accommodations Functional Accommodations o These are provided within test delivery system to student during online assessment o These include text-to-speech, font size (can be pre-set), print-on-request, color choices (available to all) Non-Functional Accommodations o These accommodations are provided to student by test administrator o These include: human reader, external mouse (available to all), special equipment

59. 59 Universal Design vs. Accommodations Some universal design features are also included on Accommodation Form UD features still available even if IEP team did not select Way to ensure students with IEPs still received UD features not counted as accommodation in federal or state reporting

60. 60 Accommodations Data-base Accommodation Data Sent to Vendor On-line IEP Functional and Non- Functional are sent Daily upload Accoms can change during a students test (may take up to 72 hours to sync)

61. 61 Review of Assessment Accommodations in DCAS TIDE Review DCAS TIDE for accommodations o Review accommodations within TIDE o Accommodation summary displayed next to each student name

62. 62 Functional Assessment Accommodations in DCAS TIDE Review individual student accommodations o Review functional accommodations within TIDE o Select VIEW button next to a student name.

63. 63 Non - Functional Assessment Accommodations Non-Functional Accommodations o Review non-functional accommodations within TIDE o Select VIEW button next to a student name. o This will display all of the accommodations assigned to student.

64. 64 Student Accommodations Review Validate all accommodations for testing Make sure all new IEP accommodations have been modified or added 3 days before testing Please provide a hard-copy of student accommodations for Test Administrator to validate when in the test administration window Additional accommodation information can be found in the Test Administration Manual (TAM)

65. 65 Accommodations Data-base Test Session Accommodation Data Sent to Vendor On-line IEP Mandatory PD All accoms must be provided, available and accessible during test Accoms appear on Test Administrator’s screen – Must verify both functional and non-functional

66. 66 Test Administration Session – Validate Accommodations If student has accommodations, select the See/Edit details button

67. 67 Test Administration Session – Validate Accommodations Review Accommodations against Accommodations Form (paper version)

68. 68 Accommodations Data-base Accommodations Data Returned to DOE Test Session Accommodation Data Sent to Vendor On-line IEP Provides accoms at time of test Long string of information Does not show if student actually used an accom or not

69. 69 Accommodations Data-base Accommodations Data Returned to DOE Test Session Accommodation Data Sent to Vendor Monitoring On-line IEP

70. 70 Monitoring throughout the System Test Security: o From teachers, students and parents o From data forensics o From special education state complaints Primarily completed off-site Random reviews of accommodations used during testing vs what is in DE data base Do not currently compare what is on IEP paper version what is in the system (unless there is complaint)

71. 71 Accommodations Data-base Accommodations Data Returned to DOE Test Session Accommodation Data Sent to Vendor On-line IEP Future Change: New IEP forms for accoms Does student need additional practice or Preparation?

72. 72 Accommodations Data-base Accommodations Data Returned to DOE Test Session Accommodation Data Sent to Vendor On-line IEP Future Change: Approval of Read Aloud (passages) through IEP audit

73. 73 Accommodations Data-base Accommodations Data Returned to DOE Test Session Accommodation Data Sent to Vendor On-line IEP Future Change: Direct link: On-line IEP Accoms to DE Accoms Data-base

74. 74 Accommodations Data-base Accommodations Data Returned to DOE Test Session Accommodation Data Sent to Vendor On-line IEP Future Change: PD for validating accoms On-site monitoring

75. 75 Accommodations Data-base Accommodations Data Returned to DOE Test Session Accommodation Data Sent to Vendor On-line IEP Future Change: Accom data will be sent in way we can delineate accoms

76. 76 Thank you Carolyn Lazar Carolyn.lazar@doe.k12.de.us for Brian M. Touchette Brian.touchette@doe.k12.de.us

77. + Discussant Jan Sheinker, Ed. D. jansheinker@gmail.com