1. 1 2011 STAR Nuts and Bolts Riverside County Assessment Network March 4, 2011 Wes Scott Key Data Systems

2. 2 Overview What is new in 2011 Who Takes What Test To Be Gridded by All Students Hand-gridding demographic data Irregularities and Post test gridding Scorables and non-scorables packaging Special Education issues Considerations and Reminders

3. 3 What is new in 2011 CMA is now... – ELA: Grades 3 – 11 – Math: Grades 2-7, Algebra I, and Geometry – Science: Grades 5,8 and 10 Grade 4 Writing is back 2 minor editorial changes in box A3 of the answer documents

4. 4 CMA ELA grades 3-11, Math grade 3-7, Algebra I, Geometry, Science 5, 8 & 10 May test CMA in one content area and CST in another Can not test CMA and CST in same content area If CMA, can not assess with CAPA Take grade 4 or 7 CMA writing if taking CMA ELA One more test section than CST, all content areas, all grades Fewer answer options (3 in CMA versus 4 in CST) Fewer items – Grade 3-5 – 57 items, 48 count – Grade 6-8 – 63 items, 54 count – Grade 9-11 – 70 items (Grade 10 science is 66 items), 60 count Accountability – CMA grades 3-8, ELA will count in AYP – CMA grades 3-9 in 2010 Base API & 2011 Growth API – CMA Geometry, grade 8, will count in AYP, December release, not in API

5. 5 CMA (cont.) Same participation criteria as last year – CST scores in any previous year of Far Below Basic or Below Basic OR Scoring Proficient or Advanced on TWO previous CAPA administrations – Multiple measures of assessments – Determined will not attain grade level proficiency – Response to grade level instruction – IEP must state the student will take the CMA, and in which content area, before STAR testing

6. 6 Answer Document Change Box A3 added subsection identification (this is a grade 10 answer document)

7. 7 Answer Document Review Box A2 – There is no ‘All Tests’ column – In alphabetical order – A column for each possible test for this grade level (varies with grade level). This sample is grade 5.

8. 8 Answer Document Review (cont.) Box 7c on CST answer document. Used to identify which CMA tests will be taken and scored Separate answer document for CMA Grades 3, 8-11 Grades 4 -7 answer documents include CST and CMA answer sections...CST on left and CMA on right. CST and CMA version numbers at top right portion of page. Required – sets scoring key. CMA section has box around it and different icons.

9. 9 CMA Version Box around CMA answer section Grade 5 Answer doc

10. 10 Race/Ethnicity Box 8 – Hispanic/Latino – Yes/No bubbles Box 9 – Race. To be filled in regardless of box 8 answer (Federal rules) California says “do not guess” if race is blank API, AYP, If Box 8 is ‘Yes’, will be Hispanic regardless of Race bubbles If Box 8 is ‘No’ and no Race, will not be in any ethnic group. If Box 8 is ‘No’ and bubbles in two or more racial groups, ‘Two or More Races’ subgroup Note: races listed alphabetical and not by group

11. 11 Who Takes What Test All students take CST, CMA or CAPA, except… – If student already tested in a previous school – Student enrolled after the first day of testing (testing optional) – Parent Exemption If less than 85% tested in any grade level test, then the school will not get an API. Grade level tests include: – ELA grades 2-11 – Math grades 2-9 – Science in grades 5, 8 and 10 – HSS in grades 8 and 11 AYP – Grades 2-8 require 95% participation – High school without a grade 10, STAR results are used

12. 12 Who Takes What Test (cont.) Math grades 7-11 (pages 15-17, D-1 in District/Site Coordinators Manual) – Take End-Of-Course (EOC) test if in standards aligned course Grade 7 – Grade 7 Math or Algebra I Grade 8 & 9 take EOC Math or General Math (based on what Math course the student will complete this school year) Grade 10 & 11 take EOC Math unless not in a standards aligned course (e.g. Business math) If passed Algebra I and II and Geometry (new in 2010) or Integrated Math III and not taking any Math course OR taking higher-level math, take Summative Math Grade 10 & 11, take no Math test if will not complete standards aligned course and don’t qualify to take the Summative Math test If no EOC Math test taken, API score of ‘200’ (Far Below Basic) at a reduced weight (.10 versus.20)

13. 13 Who Takes What Test (cont.) Science grades 9-11 (pages 17-18, D-2) – Grade 10, all students, take Life Science test – Take EOC test if in standards aligned course Some district flexibility on this. – AP courses take corresponding CST – Higher level science, may take most closely aligned CST No EOC Science test – API score of ‘200’ (Far Below Basic) at a reduced rate (.05 versus.22) – If take Grade 10 Life Science test, no ‘200’ – Some grade 10 students take two Science tests

14. 14 Who Takes What Test (cont.) HSS Grades 9-11 – World History is an EOC test. Assess when student takes World History course May take more than once if retaking course If taking in grade 11, will take two HSS tests – Grade 11 – U.S. History (all grade 11 students)

15. 15 Who Takes What Test (cont.) Can retake EOC Math and/or Science if… – Student failed course and is retaking the course – Student is taking AP course after completing the standards based course (e.g. AP Chemistry) – Student is taking General Math in grades 8 and 9 – Student is taking the Summative Math test

16. 16 STS Grades 2-11, ELA and Math Taken in STAR testing window STS Math, grades 2-7, Algebra I and Geometry Take Algebra I/Geometry if taking equivalent CST math test Same criteria for who takes STS – English Learners in US school less than 12 months (cumulative) – EL’s receiving academic instruction in Spanish – Any Spanish speaking EL may take STS (district decision) – Non-EL’s (English Only, IFEP, RFEP) can NOT take STS

17. 17 To Be Gridded by All Students Box 1 – student name, teacher, etc. Box 7 (grade) Box 7c – CMA only Grade 7, page 2, Math test taken, version (Algebra 1). Will be scored as 7 th grade math if not bubbled Note Special stopping place depending on Math test taken for CMA Grade 4-7, page 3 – CMA version (if applicable)

18. 18 To Be Gridded by All Students Grades 9-11 – Page 2 – CST Science test taken and version – Page 3 – CST Math test taken and version (grades 8-11) – Page 2 – HSS version Grade 11, page 2, EAP-ELA (bubble releasing EAP scores) Grade 11, page 3, EAP-Math (bubble releasing EAP scores)

19. 19 Box 1 Hispanic or Latino Grade CMA – 7c SSID

20. 20 EAP (CSU and CCC) World History Version ELA Version

21. 21 EAP Science Taken and Version Math Taken And Version

22. 22 Hand-Gridding Demographic Data Why hand-grid an answer doc? – Student is new to the district since pre-id Alternative - Late pre-id up to one week before end of testing window – Student changes schools or tracks after pre-id submitted STAR web site option to change schools – only need to change CBEDS – Will result in a late label being created by ETS. Can take pre-id document to new school – online update CBEDS If change to new wave, void first pre-id and do late pre-id in new wave – Late pre-id available until week before end of testing period

23. 23 Hand-Gridding Demographic Data (cont.) – Believe pre-id data is incorrect Alternative - Online data correction available until last day of testing – Pre-id answer document becomes unusable – Pre-id for incorrect assessment (e.g. should be CMA, not CST) If should take CMA and pre-id CST, or visa versa, and grade 3 If CMA but pre-id CST, grade 4-7, use pre-id answer document If CMA but pre-id CST, grades 8-11. Hand-bubble CMA document – Only bubble boxes 1-7c, 11 and 12 – Be sure SSID and Name spelling exactly matches pre-id document

24. 24 Hand-Gridding Demographic Data (cont.) Gridding by District staff – not students All data is important – leave no blanks except those that don’t apply to student (e.g. no RFEP info if not RFEP student) Accuracy is vital as it impacts AYP and API Box 10, Parent Education – Similar Schools (Parent Ed has most impact of any component) – Socio-economically disadvantaged subgroup Box 11, student id, do NOT add zeros on the right side. Box 12, SSID, always 10 digits, never starts with zero – Used for matching Writing, CST, CMA, EAP Grid all available bubbles (e.g. 2 bubbles for day of month and all bubbles for years), ETS does not default any values

25. 25 Hand-Gridding Demographic Data (cont.) Box 14 (CBEDS). If left blank, CDE treats as YES – may be bad Box 16c (Special Ed Exit Date). Only if 16a is ‘000’. Very important for AYP Box 20 – only for students who live outside district but attend district to receive special ed services per an IEP. Must be valid county/district code Box 22 (RFEP Proficient/Advanced 3 times on CST ELA, SINCE RFEP date). If left blank (grades 5-11), CDE treats as YES – bad. Has to be blank for grades 2-4. Boxes 15 (English Proficiency), 22 (RFEP CST ELA 3 times) and 24 (Date first entered a U.S. School) used to determine EL subgroup 3% rule: Gender, Hispanic or Latino, Parent Ed, SSID, CBEDS, English Proficiency, Disability code, NSLP (boxes 6,8,10,12,14,15,16a,23) – If more then 3% blank for a school, ETS scoring is stopped – District must correct (cost is $1.32 per student corrected) – Corrections need to be made within 7 days (approximately) – Only after corrections made will processing continue.

26. 26 CBEDS English Proficiency Special Ed Info RFEP 3 Year EL Date First Enrolled in US County/District of Residence Special Ed Exit Date RFEP Date NSLP

27. 27 Most Frequent Errors Missing CBEDS (D-283/S-265) Missing or bad EL Date (487) Missing RFEP 3-Year Prof/Adv (141) Missing NSLP (924) Missing Parent Education (751) Missing English Proficiency (532) Missing Hispanic indicator (235) Missing Special Ed Exit Date ??? Bad IEP County/District of Residence (262, 145=current district)

28. 28 Hand-Gridding Demographic Data (cont.) Common questions – What is continuous enrollment? Enrolled on or before CBEDS Information day (first Wednesday in October) and has not been disenrolled since that date Requires rigorous record keeping Used to determine who is in API and Proficiency on AYP – What value should be used for ‘Date First Enrolled in a US school’ if they were born in the U.S.? When they first entered school…likely kindergarten

29. 29 Irregularities See Appendix C (page 76) for guidelines and reporting forms Irregularity – helping students or security breaches – Examiner reading the test booklet (except grade 2) – Anything left on the classroom walls that will help a student – Test booklets not being locked up – Test booklets can NOT be taken out of the room except when the teacher is returning them to the secure site – Preview/review of test booklets by anyone (exception for grade 2 teachers on the first day of testing) – Teacher interpreting items for students – Test booklets being used by more than one student

30. 30 Irregularities (cont.) Report irregularities and Inappropriate Test Preparation to the Site Coordinator who reports it to the District coordinator District coordinator fills out the form (no student or teacher names are to be mentioned) and emails or faxes to the CDE (no calling) CDE will tell District coordinator which bubble, if any, to fill in box A1

31. 31 Post Test Gridding Be sure Box 1 is complete Be sure all student gridding is complete and accurate (see earlier slides) If hand-gridded demographics, be sure it is complete (AYP and API impact and/or big dollar cost to correct later) Boxes, A2, A3 and A4 – See last page of DFA for listing of codes (content varies with grade and test)

32. 32 Post Test Gridding (Cont.) Box A2 – Testing Conditions – Valid values Pages 115-116 – Significantly impact AYP & API E,L,M,T exclude from AYP enrollment, unless... – Sample is grade 5

33. 33 A2 Accountability Impact AYPAPI EnrollTestedProficientValidScore A-Absent N N* C-Cheat N B* E-Medical EmergencyN N* F-Sick – did not complete ?N*? L-Enroll after first dayN^ N*^ M-Enroll first day then movedN^ N*^ P-Parent Exempt N N* T-Previously TestedN N X-Refused acc/modn/a Y-Replacedn/a Z-Not attempt N ? = Depends on how many items attempted and score B* = Include in Base API, exclude from Growth API N* = Excluded if no items attempted N^ = Excluded unless CBEDS=Y

34. 34 A2 – Errors L – Enrolled after first day and tested – ELA – 71, 13 had CBEDS=Y – Math – 62, 13 had CBEDS=Y T – Tested in a previous district – ELA – 26, 4 had CBEDS=Y – Math – 20, 4 had CBEDS=Y Most of L and T above were not proficient

35. 35 A2 – Errors (cont.) A – Absent, 192-ELA scores, 109-Math scores E – Medical Emergency, 5-ELA, 4-Math scores P – Parent Exempt, 41-ELA, 3-Math scores M – Enrolled first day of testing, moved before test was given, 38-ELA, 58-Math scores A, P, and M will be included in AYP/API. – E is excluded from AYP regardless of scores

36. 36 Post Test Gridding (Cont.) Box A3 – Accommodations/Modifications (page 83-87 & 143 in District Coordinators Manual) – If in IEP and used in instruction, student must be allowed to use accommodation/modification – If not in IEP, can NOT use it – ONLY mark if student was observed using it, if not used, mark box A2, row X. – Accommodations do not impact AYP or API – Modifications count as Far Below Basic for API and ‘not tested’ for AYP A4 – English Learner Test Variations – No IEP required – English Learners Only – Glossary/Word list Can not use for ELA Translations only – no definitions

37. 37 Scorables and Non-scorables Packaging Scorables (orange label) – All answer docs (includes grades 2 & 3 test booklets) – *If Grade 3 absent, pre-id CMA & CST, only return one answer document with the scorables. – SGID sheets (more on this later) – Master File Sheets – Processing Form (district only) – See pages 40-42 of the District and Test Site Coordinators Manual for a comprehensive list and the required packing order

38. 38 Scorables and Non-scorables Packaging (cont.) Non-Scorables (green label) – CAPA Examiner’s Manuals – Grade 2 DFAs – Test booklets – Voided answer docs – Braille and Large print booklets

39. 39 Scorables and Non-scorables Packaging Following packaging instructions to the ‘T’ – 1 Master File for school for each admin – SGID for each grade (one per teacher if want teacher reports) and one for CAPA – CAPA on top of lower grade (do not return stimulus cards) – Lowest grades on top – Highest grades on bottom – STS under CST

40. 40 Scorables and Non-scorables Packaging SGID – Sets school code and grade for all answer documents (that’s why one can move pre-id answer docs with student if they change schools) – *Filling in teacher name on back will save that data in the results file – School code must be ‘0000001’ for NPS – No grade for CAPA – Back Instructional Materials for grades 2-8 only – CAUTION: If SGID grade and answer doc grades are different Student counts as participating Student will receive no scores

41. 41 Scorables and Non-scorables Packaging When return Multiple-Choice and CAPA – Site: 2 working days after last test for that administration – District: 5 working days after last test for that administration – Keep administrations separate – Track all documents, from original receipt until shipping back to ETS, in order to not have to pay excessive order charges

42. 42 Special Education Issues No Below Level Testing Be sure all Special Education students are tested CAPA and CMA – Scores count the same as CST for AYP and API – Assess in STAR testing window – Assess all who qualify – Appropriate test for the student – Can only help AYP and API

43. 43 Special Education Issues (cont.) CAPA and CMA limits – applied in March 2011 AYP release – CAPA 1% rule (AYP only) Applies to non-mobile students only If number Proficient or Advanced is more than 1% tested in district, those over 1% are considered not proficient for AYP purposes *File exception request (due around July 1) if even close to 1% – CMA 2% rule (AYP only) Applies to non-mobile students only If CMA Proficient or Advanced is more then 2% tested, those over 2% are considered not proficient for AYP purposes No exceptions allowed – CAPA and CMA 3% rule (AYP only) Combined CAPA and CMA can not exceed 3% If CAPA exception granted, ‘combined’ limit is 2% plus CAPA exception amount.

44. 44 Special Education Issues (cont.) CMA – Grade 4-7 answer document, cover up or fold under CST answer page when testing CMA – Test separately from CST because CMA DFA and number of sections is different than CST – Coordinate schedule with CST since some students take CMA and CST (e.g. Grade 8 CMA students will take some CSTs (at least HSS))

45. 45 Special Education Issues (cont.) Accommodations and Modifications – See Pages 83-87,143 for complete list Separate list for CST (page 83-87) and CMA (page 143) – IEP can not exempt a student from testing – Accommodations Result in a valid test Must match instructional accommodations – Modifications change test interpretation Actual score on individual student reports Far Below Basic for API Considered not tested for AYP

46. 46 Special Education Issues (cont.) CMA Accommodations that are CST Modifications – ELA American Sign Language/Manually Coded English to present questions ELA Test questions/answer options read aloud – Math Use of Calculator on grade 5 math test Math Manipulatives on Math or Science tests If modification required, must take CST (e.g. use dictionary)

47. 47 Testing Considerations Test early in the day Test with normal teacher if possible (especially for Elementary) Consider not testing on Monday and Friday Allow plenty of time for make-ups Tests are untimed (within reason) – set up room for those needing additional time Grades 2 and 3, limit to one section per day No breaks within a section except supervised bathroom breaks Multiple sections must be administered sequentially Multiple sections in a day require a 10 minute break between sections If complete a section, after starting the next section, can not go back and review answers in first section Do not administer CMA and CST in the same room at the same time

48. 48 Additional Reminders Distribute non-secure DFAs (grades 3-11) before first day of testing Grade 2, secure version, DFA (includes items) can NOT be distributed before first day. Non-secure version is available on STAR web site. *Grade 2, all test booklets need to be the same version in a classroom Distribute test materials first day of testing…not earlier Have student write their name on the answer document and all test booklets Be sure students only get their answer doc and test booklets Students can write in test booklets – Grade 2 & 3. Erase extraneous marks. Recommend no use of Highlighters. If too much bleed-through, transcribe to a new answer booklet and ‘void’ the original answer booklet Excessive erasures (all grade levels) raises a red flag for CDE Immediately report possible irregularities Administrative errors (page 78 & Form A-5 page 80) need to be documented but are not reported to the CDE

49. 49 Additional Reminders (cont.) Box A3. Note that each DFA lists only accommodations/modifications applicable to that grade level/test. Writing has additional codes. *Unlisted Accommodations/Modifications must be cleared through the CDE prior to testing…do it 30 days before Grade 2, teacher read items…do not mark as ‘read aloud’ modification when DFA says to read the item aloud. Directions for Administration (DFA) – Must read ‘Say’ boxes exactly as written – ‘T’ means can translate ‘Say’ box *No modifications allowed for CMA. If modification required, must take CST Before testing, give teachers list of any demographic information needing to be gridded. Request ETS to send data as ASCII text, not XML There is a late processing fee of $385 if ‘extra’ answer docs sent in late and ETS has already processed the district data

50. 50 Additional Reminders (cont.) Miscellaneous – CMA – Braille for grades 3-8 except for Algebra I – Call ETS if running short on materials. They can deliver quickly using Supplemental Order process (need to call ETS) – Be aware of excessive order charge (if use less than 90% of materials ordered...ETS overage is excluded from calculation). – No Teacher Reports – API includes CAPA science (grades 5,8 and 10)

51. 51 Final Thoughts Be familiar with the following. They are all available on the STAR web site. – Directions for Administration (all grades and test types) – District and Test Site Coordinator Manual – Sample answer documents Maintain strict security Accuracy of demographic and other gridded data on the answer document is vital

52. 52 Contractor Contact Information STAR (ETS) – TAC: 1-800-955-2594 – FAX: 1-800-541-8455 or 1-925-808-2118 – STAR@ets.org STAR@ets.org – www.startest.org www.startest.org CDE STAR Office – Phone: 1-916-445-8765 – STAR@cde.ca.gov STAR@cde.ca.gov – www.cde.ca.gov www.cde.ca.gov

53. 53 QUESTIONS? Wes@KeyDataSys.com 951-245-0828 x 212