1. 2015 TELPAS HOLISTIC TRAINING FOR CTC’ S Assessment, Research & Evaluation Department January 26, 2016

2. D ISCLAIMER This training does NOT take the place of reading the appropriate manuals. 2

3. 2016 TELPAS Changes ■Key changes for this year include: –New online delivery system –Calibration must be completed in a monitored setting –Two calibration sets only –Supplemental support training is no longer required 3

4. T EST S ECURITY AND C ONFIDENTIALITY R EQUIREMENTS 2015 TELPAS Rater Manual – pg. 3 4

5. Test Security ■Test security of the TELPAS program involves protecting the contents of all online assessments and student performance documentation. This requires compliance with, but is not limited to, the following guidelines –Personnel meet the requirements, have been trained and have signed the appropriate oath to have access to the assessment materials and information –Campuses implement the controls necessary to ensure proper storage of secure materials throughout the test administration –Campuses place confidential documentation (i.e. - rating rosters) in a limited- access locked storage –Any scratch paper used during the assessment must be collected and returned to Assessment immediately after the test administration 5

6. Confidentiality Requirements ■Testing personnel undergo training and sign the oath affirming they understand their obligations concerning the security and confidentiality of the TELPAS program ■ Testing personnel administer the tests in strict accordance with the instructions ■Testing personnel do not view, reveal or discuss the contents of an assessment before, during, or after a test administration 6

7. Confidentiality Requirements ■No person may DUPLICATE/PRINT/RECORD/WRITE NOTES/CAPTURE any portion of an assessment (electronically or by any other means) ■ Only students may respond to test questions ■No person may receive or provide answers to student profiles during TELPAS calibration activities 7

8. Penalties for Prohibited Conduct Any person who violates, assists in the violation of, or solicits another to violate or assist in the violation of test security or confidentiality, as well as any person who fails to report such violation is subject to the following penalties Placement of restrictions on renewal or holding of educator certificate Issuance of an inscribed or non-inscribed reprimand Suspension of a Texas educator certificate for a set term Revocation or cancelation of an educator certificate without opportunity for reapplication for a set term or permanently 8

9. Penalties for Prohibited Conduct (continued) Release or disclosure of confidential test content could result in criminal prosecution under TEC §39.0303, Section 52.352 of the Texas Government Code, and Section 37.10 of the Texas Penal Code. Further, 19 TAC §249.15 stipulates that the State Board for Educator Certification may take any of the above actions based on satisfactory evidence that an educator has failed to cooperate with TEA in an investigation. Additionally, irregularities resulting in a breach of test security or confidentiality may result in the invalidation of students’ assessments. 9

10. Security Oaths and Confidentiality Statements All personnel must meet the eligibility requirements and sign a security oath Any person who has more than one testing role must receive appropriate training and sign a security oath for EACH role Each oath must be read and completed AFTER the training and BEFORE handling or viewing any secure materials or confidential information Oaths must be kept on file for at least five years 10

11. Testing Irregularities Serious These constitute severe violations of test security and/or confidentiality and can result in the individual(s) responsible being referred to the TEA Educator Standards and Certification Legal Division for consideration of disciplinary action. Examples –Using answer sheets or notes that contain answers to the TELPAS calibration activities –Receiving or providing answers to student profiles during TELPAS calibration activities –Falsifying TELPAS ratings or writing samples –Fraudulently exempting a student from the administration of TELPAS –Assisting students directly or indirectly with responses to test questions 11

12. Testing Irregularities NOTE!!!! ■It is a violation of state assessment procedures and a serious testing irregularity to record, discuss, or share answers to the rating practice and calibration activities. –NEW: After completion of calibration activities, raters must turn-in all notes taken during the calibration to the proctor.

13. Testing Irregularities Procedural Less severe and more common irregularities that are typically the result of minor deviations in testing procedures. Examples: –Eligibility error –IEP implementation issues –Improper accounting of secure materials –Monitoring error –Other procedural error 13

14. Testing Irregularities Eligibility error Student not rated in one or more domains Student not administered the reading test Ineligible student was assessed Procedural Irregularities (examples) IEP Implementation error SPED student was/was nor provided an accommodation

15. Testing Irregularities Improper Account of Secure Materials error Teacher misplaced a writing collection Monitoring error TA left room unattended when online tests were open Student was not prevented from using cell phone Procedural Irregularities (examples) Procedural error TA who wasn’t trained rated students A writing collection was not submitted according to the assembly criteria TA failed to provide student the correct authorization for online testing

16. Reporting Test Irregularities Each person is responsible for reporting ANY violation or suspected violation Campus staff should notify their CTC or DTC if they witness an irregularity or suspect one has occurred DTC will contact TEA to report ANY incidents as soon as DTC is made aware of the situation Contact DTC if unsure about whether an irregularity has occurred 16

17. CAMPUS TRAINING REQUIREMENTS 17

18. Campus Training Requirements You are expected to:  Train staff on holistic rating training requirements  Train staff on writing collection procedures and timelines  Ensure raters understand the holistic rating process  Ensure raters are informed about the date, time, location of monitored sessions for calibration  Understand the consequences of cheating during calibration activities Responsibilities of TELPAS Coordinators 18 DCCM, T-16

19. You are expected to:  Schedule monitored calibration sessions  Establish procedures to ensure validity and reliability  Maintain documentation of procedures followed  Implement rating support requirements for individual raters 19 Campus Training Requirements Responsibilities of TELPAS Coordinators DCCM, T-17

20. Prepare For Training ■Coordinators ensure that: –All ELL’s are identified and assessed –TELPAS Raters & Verifiers are identified and properly trained –Procedures are established to collect & verify the years in the U.S. –Ensure that testing infrastructure is in place –Ensure that assessment information is entered and verified –Materials are collected and filed as appropriate –Ensure security oaths are signed and returned after the administration

21. Online Training Components Online BASIC TRAINING Course ■Required for all raters in Brownsville ISD. ■Gives raters practice rating students in each language domains. ( approximately 4-5 hours) 21

22. Online Training Components Online CALIBRATION Course ■Required for ALL raters. ■Each calibration set contains 10-12 students to rate and all holistically assessed language domains are represented in each set. (approximately 1- 2 hours) –Raters taking the grades Kinder-1 st course – will rate listening, speaking, reading and writing –Raters taking the Grades 2 nd – 12 th course – will rate in listening, speaking and writing 22

23. 23 When and where do raters complete calibration? ■All raters must complete calibration in a monitored session that has been established by the campus. ■Monitored calibration sessions may take place during the school day, after school, or as part of professional development. It is a violation of state assessment procedures to record, discuss, or share answers to the rating practice and calibration activities. Prior to creating a TELPAS Online Training Center account, raters are required to read an online statement and affirm that they will complete the rating activities independently.

24. Monitored Calibration Sessions ■All raters must receive administration procedures training prior to attending a calibration session and must have a signed oath on file. A roster of participants in each session will be maintained. ■Raters will be given a passcode by the session proctor in order to access calibration. ■Raters may use their TELPAS Rater Manual or a copy of the PLDs and scratch paper to make notes during calibration. All notes must be turned in to the proctor at the end of the session. ■Raters will need to complete calibration activities individually within the monitored group session. Therefore, the use of headphones is required. 24

25. ■Raters must begin with calibration set 1. If successful, a certificate will be available in the scoring summary tab of the TELPAS Online Training Center. The certificate should be printed and turned in to the proctor. ■If raters are unsuccessful on set 1, they will be notified by TELPAS coordinator when monitored session for set 2 will be held. ■After completing a calibration set, raters immediately see results and annotations explaining the correct ratings. ■Raters will need 1-2 hours to complete a calibration set. Raters must finish a calibration set once it is started. They will not be able to save their progress and return later to complete the set. Monitored Calibration Sessions 25

26. Recap of Calibration Process  Calibration is conducted in monitored sessions.  A passcode is required to access calibration.  There are 2 calibration sets.  Raters who calibrate on set 1 are done.  Raters who don’t calibrate on set 1 must complete set 2.  Certificates must be printed and turned in to the proctor. 22

27. Other Things to Know About Calibration 27  Raters affirm online that they will keep the contents of the calibration sets secure and confidential.  Calibration activities are taken from a bank and randomized. Trainees will rate different sets of students.  Raters can work at their own pace, go back and review students, and change ratings as they work.

28. HOLISTIC RATING TRAINING REQUIREMENTS

29. Rater Credentials & Responsibilities RATER: Teacher designated to be the official rater of an ELL’s language proficiency Each teacher selected to rate must: –have student in class at the time of the spring assessment –be knowledgeable about the student’s language ability (instructional and informal) –have a valid Texas teaching certificate –be appropriately trained –rate the student in all eligible domains 29 Districts must implement validity and reliability checks during the testing window ( 2 nd rater ).

30. Returning Raters Online Basic Training Course Online Calibration (Sets 1 and 2*) Spring TELPAS Administration Procedures Training As a key part of this training, information from the TELPAS Rater Manual is reviewed to prepare raters to proceed with online holistic rating training Online Calibration (Sets 1 and 2*) New Raters *Set 2 required only if not successful on Set 1 Holistic Rating Process 30 Online Basic Training Course It is a violation of state assessment procedures to record, discuss, or share answers to the rating practice and calibration activities. Prior to creating a TELPAS Online Training Center account, raters are required to read an online statement and affirm that they will complete the rating activities independently.

31. Grade Cluster Organization 31 Online Training Process Basic Training Course Kinder - 1 Basic Training Course 2 – 12 Calibration Clusters Kinder - 1 Calibration Clusters Grade 2 Grades 3 – 5 Grades 6 – 8 Grades 9 – 12

32. Why is calibration necessary? ■Calibration helps ensure that raters have adequate training, including ample practice and feedback, before they assess their students in the spring. ■Calibration ensures that raters clear their heads and consider only the elements of student performance included in the PLDs. ■Calibration supports assessment validity and reliability and is an important part of holistically scored assessment processes. 32

33. Will raters be able to refer to any resources during calibration activities? ■Yes, raters should use their rating rubrics (PLDs) and TELPAS Rater Manual, as needed. 33 Any notes taken during calibration should be done on the scratch paper provided and not in the TELPAS manual. At the end of each calibration session, all notes taken by raters must be turned into the proctor and destroyed.

34. Each set contains 10 students to be rated. You must rate at least 70 percent of the students correctly within a set to demonstrate sufficient calibration. Each set contains 12 students to be rated. You must rate at least 67 percent of the students correctly within a set to demonstrate sufficient calibration. Kinder and 1 st 2 nd – 12 th grade 34 How many students must be rated successfully?

35. What happens if a rater is unsuccessful in calibrating? Two outcomes are possible for individuals who complete the calibration activities but are not successful: –The district may choose not to assign the individual to be a TELPAS rater. –If the individual is needed to serve as a rater, the district must implement rater support procedures to ensure that the rater’s students are evaluated consistent with the rating rubrics. Individuals are not authorized to serve as raters unless they complete all required training components. 35

36. Course and Calibration Certificates Basic Training ■After completing all modules in the online basic training course, your TELPAS coordinator will be able to provide you with a certificate of completion. Calibration ■After you have successfully calibrated, go to the scoring summary tab of the TELPAS Online Training Center to access a certificate. Print a copy and turn it in to the session proctor. 36

37. WRITING COLLECTION OVERVIEW 37

38. February 15, 2016  Writing assigned on or after February 15, 2016 may be considered.  Writing samples may continue to be gathered until the date designated by the district as the deadline for completing the collections in order to submit the ratings. 38

39. Samples Required  At least 5 total samples are required in each collection.  In each collection there must be –at least 1 writing sample that elicits the use of past tense –at least 2 writing samples from math, science, or social studies 39

40. Some Eligible Types of Writing  Descriptive writing on a familiar topic  Writing about a familiar process  Writing that elicits the use of past tense  Personal narratives and reflective pieces  Expository and other extended writing from language arts classes  Expository or procedural writing from science, math, and social studies classes 40

41. Papers Not to Include  Papers containing copied language  Papers in which student relies heavily on resources (dictionary, thesaurus, etc.)  Papers showing teacher comments and corrections  Worksheets and question-answer assignments  Papers that have been polished with help from peers or teachers  Papers written primarily in student’s native language  Papers that are brief, incomplete, or rushed 41

42. Building Collections  Strive to gather more than 5 writing samples for each student.  Choose at least 5 samples that meet the criteria and do the best job of portraying the student’s current proficiency level.  If a student is near the border between two proficiency levels, consider including samples written in the latter part of the TELPAS assessment window. The more recent the writing samples, the more accurately they will reflect the proficiency level of these students. 42

43. Building Collections  Collections should contain some papers in which students showcase English they know and feel comfortable using. Comfort zone writing is especially important for students at lower proficiency levels.  Collections should also include papers in which students are stretched and pushed beyond their comfort zone so the collection shows that a student has not yet reached the next level (the student is beginning but not yet intermediate, intermediate but not yet advanced, advanced but not yet advanced high). 43

44. Building Collections The papers you assemble need to help you determine and justify your ratings. They must give you evidence to say, “I know the student is at least at X proficiency level because of these characteristics in his or her writing. I know the student is not yet at the next proficiency level because of these other characteristics in his or her writing.” Reminder The characteristics you consider must come from the PLDs. 44

45. Beginning Level  The ELPS require ELLs of all proficiency levels to learn to write in English.  It is not acceptable for writing collections of students at the beginning level to include samples written primarily in the native language. Such samples do not provide evidence of English writing proficiency.  Writing tasks of these students should be adapted to their needs. Their writing in English will likely be formulaic or memorized, include recently practiced vocabulary, lack detail, etc. 45

46. Summarizing Tips ■Students who are capable of expressing themselves in English in a detailed, extended way should do so. Do not include brief responses from students who know enough English to respond to writing tasks in extended ways. ■In other words, consider students’ English-language proficiency levels in determining whether papers are too brief to be included. 46

47. Summarizing Tips ■Build writing collections that have a balance of writing from language arts and other core content areas. ■Collections should show what the student knows and can do as well as what the student struggles with in second language acquisition. ■Remember, build the collections to portray the student’s overall ability to communicate in writing in English. 47

48. WHAT’S NEXT?

49. Recommended for all students taking TELPAS Several different test item formats Since no sample items with test, practice with item formats and online interface is useful As in the past, tutorials are separate from tests Schedule TELPAS Tutorials

50. Plan the holistic process Where do I start? Schedule your trainings With your principal, determine how your campus will conduct basic training and calibration activities Verify trainings have been completed (obtain all documentation) Download rating rosters, coversheets, etc from ARE website Be sure to collect and have the correct Years in US schools of all students (ELLs)

51. Next Training  TELPAS Online Reading Test Training  Friday, February 19, 2016  2:00 – 5:00  CAB Cafeteria 51

52. QUESTIONS AND ANSWERS…