1. 1 Topical Call Series: Improving Data Quality and Use CSPR Data Collection Tuesday, September 15, 2015

2. 2 Who Is on the Call? What is your role? Title I, Part D State Coordinator EDFacts or CSPR Coordinator at SEA Other SEA staff State-run facility staff (Subpart 1) Locally run facility staff (Subpart 2)

3. 3 Agenda Overview of CSPR Data Collection Overview of CSPR Resources and Tools CSPR Collection: Table-by-Table –Review data indicators –Data quality checks State Data Collection Tools and CSPR Timelines in States NDTAC is funded through a contract with U.S. Department of Education. The content of these slides and presentation does not necessarily reflect the views or policies of the U.S. Department of Education, nor does it imply endorsement by the U.S. Government.

4. 4 Overview of CSPR Data Collection

5. 5 Statutory Requirements Each State agency and local educational agency shall— (1) submit evaluation results to the State educational agency and the Secretary; and (2) use the results of evaluations under this section to plan and improve subsequent programs for participating children and youth. State and local agencies receiving Title I, Part D funds must evaluate their programs’ impact on the ability of students: (1) to maintain and improve educational achievement; (2) to accrue school credits that meet State requirements for grade promotion and secondary school graduation; (3) to make the transition to a regular program or other education program operated by a local educational agency; (4) to complete secondary school (or secondary school equivalency requirements) and obtain employment after leaving the correctional facility or institution for neglected or delinquent children and youth; and (5) as appropriate, to participate in postsecondary education and job training programs. Source: Title I, Part D, Statute, Subpart 3

6. 6 Federal Collection Requirements All Title I, Part D-funded programs must collect data to report to the U.S. Department of Education (ED). Program categories: At-Risk programs— Subpart 2 only Other Programs Neglected Programs Juvenile Detention Juvenile Corrections Adult Corrections— Subpart 1 only

7. 7 Federal Reporting Systems: CSPR and EDFacts CSPR is a data collection instrument administered annually by ED’s Office of Elementary and Secondary Education (OESE). EDFacts is an ED initiative to collect, analyze, report on, and promote the use of high-quality performance data. Most of the Title I, Part D data are now reported through EDFacts. Eventually, all data will be reported through the EDFacts’ online EDEN Submission System (ESS).

8. 8 Relation Between Federal Reporting Systems Category/Area CSPR Tables EDFacts Specification Files Subpart 1Subpart 2 Facility Counts/Programming 2.4.1.1 & 2.4.1.1.1 2.4.2.1 & 2.4.2.1.1 Not included Student Count/Demographics 2.4.1.22.4.2.2 S1 = C119 (Data Group 656) S2 = C127 (Data Group 657) Transition Services 2.4.1.3.12.4.2.3.1 S1 & S2 = C182 (Data Groups 786, 787) Academic/Vocational Outcomes 2.4.1.3.22.4.2.3.2 In Program = C180 (Data Groups 782, 783) Exited Program = C181 (Data Groups 784, 785) Pre- and Posttesting in Reading and Math 2.4.1.6.1 & 2.4.1.6.2 2.4.2.6.1 & 2.4.2.6.2 S1 = C113 (Data Group 628) S2 = C125 (Data Group 629)

9. 9 Why Is Data Quality Important? Trusting your data is important for informing: funding and other decisionmaking; technical assistance (TA) needs; subgrantee monitoring; and student programming.

10. 10 Role of the Part D Coordinator in Improving Data Quality Ultimately, coordinators cannot “make” the data be of high quality, but they can implement systems that make it more likely by: understanding the collection process; providing TA in advance; developing relationships; developing multilevel verification processes; tracking problems over time; using the data; and linking decisions (funding, hiring, etc.) to data evidence.

11. 11 Overview of CSPR Resources and Tools

12. 12 EDFacts Initiative Web Page http://www2.ed.gov/about/inits/ed/edfacts/index.html Overview of EDFacts EDFacts File Specifications CSPR Documentation (a.k.a. Data Reporting Forms)

13. 13 Partner Support Center Provides assistance to EDFacts customers who submit data to the Education Data Exchange Network (EDEN) or use its analytical and reporting tools. Contact Information: Toll Free: 877-457-3336 (877-HLP-EDEN) Fax: 888-329-3336 (888-FAX-EDEN) TTY: 888-403-3336 (888-403-EDEN) E-mail: EDEN_SS@ed.govEDEN_SS@ed.gov

14. 14 CSPR Resources: CSPR Data Reporting Forms for School Year 2014–15 Official Part II reporting forms are on the ED Web site. Part D (Section 2.4) can be found on page 35. http://www2.ed.gov/admins/lead/account /consolidated/csprpart21415.doc

15. 15 CSPR Resources: The Instructional Guide to Reporting Title I, Part D Data in the CSPR for SY 2014–15 Also known as “The CSPR Guide” The “encyclopedia” of the CSPR Data Collection http://www.neglected-delinquent. org/sites/default/files/ NDTAC_CSPR_guide_201314.pdf Topical call on how to utilize “The CSPR Guide”: http://www.ndcommunities.org/ calls/ndtac-resources-meet- technical-assistance-needs-call- 1

16. 16 CSPR Resources: TIPD Data Collection List for SY 2014–15 Lists data indicators needed for CSPR and EDFacts data collection Briefly defines each item It’s a good place for your “shopping list” of data items. http://www.neglected-delinquent.org/ sites/ default/files/CSPR_collection- list.doc

17. 17 CSPR Resources: TIPD Data Collection and Submission Timeline for SY 2014–15 This will help you see the bigger picture during planning and explain data collection process to stakeholders. http://www.neglected-delinquent.org/sites/default/files/TIPD_DataCollectTimeline.pdf

18. 18 CSPR Resources: Checklist for Performing Data Quality Reviews of TIPD Data Can be used for in-depth data reviews Checklists for: –programs and facilities; –students served; –transition services; –academic and vocational outcomes; and –academic performance in reading and math. Can be used by anyone responsible for collecting, entering, or reviewing the data (at school, LEA, SA, or SEA levels) http://www.neglected-delinquent.org/sites/default/files/NDTAC_TIPD_DataQualityChecklist.pdf

19. 19 Reporting Tool: Creating a Weighted Average Length of Stay Instructions with examples that guide calculations for reporting average length of stay Can be used by anyone responsible for collecting and entering data (at LEA, SA, or SEA level) http://www.neglected-delinquent.org/ sites/default/files/NDTAC_Reporting Tool_AverageLengthofStay.doc

20. 20 Reporting Tool: Collecting and Reporting Racial/Ethnic Data in Seven Categories Describes how data should be collected (two-part question) and reported (seven categories), per ED requirements Can be used by anyone responsible for collecting and reporting data (at school, LEA, SA, or SEA level) http://www.neglected- delinquent.org/administering-title-i- part-d/reporting-and-evaluation

21. 21 Reporting Tool: Reporting Complete Pre- and Posttest Results for Reading/Mathematics Provides sample calculations to help avoid common data collection errors Can be used by anyone responsible for collecting, reporting, or reviewing data (at school, LEA, SA, or SEA level) to identify inconsistencies in performance data http://www.neglected- delinquent.org/sites/default/files/ ReportingTool_PrePostResults.pdf

22. 22 CSPR Data Collection: Table-by-Table

23. 23 2.4.1.1/2.4.2.1 Programs and Facilities Program/Facility Type # Programs/ Facilities Average Length of Stay in Days Neglected Programs Juvenile Detention Juvenile Corrections Adult Corrections/At-Risk Other Total (Auto calculated)//////////////////////////////

24. 24 2.4.1.1.1/2.4.2.1.1: Programs and Facilities That Reported Program/Facility Type# Reporting Data Neglected Programs Juvenile Detention Juvenile Corrections Adult Corrections/At-Risk Other Total (Auto calculated)

25. 25 Examples From Colorado: What To Do When a Facility Closes or Refuses Future Funds Colorado had an S1 facility submit an annual count in SY 2013–14, but it chose to refuse funds for SY 2014–15. –SEA had the facility submit a letter saying that they were turning down the funds for SY 2014–15 and that they would still submit data for the youth they served in SY 2013–14. Colorado had an S2 neglected facility close in January 2014 (mid SY 2013–14). –Going through the district, the SEA obtained the SY 2013–14 data from the facility that closed.

26. 26 2.4.1.1/2.4.2.1 and 2.4.1.1.1/2.4.2.1.1 Data Quality Checks 2.4.1.1.1Programs and Facilities That Reported 2.4.1.1 Programs and Facilities Program/Facility Type # of Programs/ Facilities Neglected Programs2 Juvenile Detention3 Juvenile Corrections2 Adult Corrections4 Other2 Total13 Program/Facility Type # Reporting Data Neglected Programs2 Juvenile Detention3 Juvenile Corrections2 Adult Corrections4 Other1 Total12

27. 27 2.4.1.1/2.4.2.1 and 2.4.1.1.1/2.4.2.1.1 Data Quality Checks 2.4.1.1.1Programs and Facilities That Reported 2.4.1.1 Programs and Facilities Program/Facility Type # of Programs/ Facilities Neglected Programs2 Juvenile Detention3 Juvenile Corrections2 Adult Corrections4 Other2 Total13 Program/Facility Type # Reporting Data Neglected Programs2 Juvenile Detention3 Juvenile Corrections2 Adult Corrections4 Other1 Total12

28. 28 2.4.1.2/2.4.2.2 Students Served: Unduplicated Count and Long-Term Students “Unduplicated” means that the student is counted only once, even if admitted to a facility multiple times within the year. “Long-term” refers to students with a length of stay of 90 consecutive calendar days or more within the reporting year. Do not add together separate enrollments. The number of long-term students reported should not exceed the unduplicated count for each program. # of Students Served Neglected Programs Juvenile Detention Juvenile Corrections Adult Corrections/ At-Risk Other Programs Total Unduplicated Students Served Long-Term Students Served

29. 29 2.4.1.2/2.4.2.2 Student Subgroups Student Subgroups Neglected Programs Juvenile Detention Juvenile Corrections Adult Corrections/ At-Risk Other Programs Students With Disabilities (Individuals with Disabilities Education Act of 1990 (IDEA)) Limited English Proficiency (LEP) Students

30. 30 2.4.1.2/2.4.2.2 Students Served: Race/Ethnicity and Sex Race/Ethnicity Neglected Programs Juvenile Detention Juvenile Corrections Adult Corrections/ At-Risk Other Programs American Indian or Alaska Native Asian Black or African American Hispanic or Latino Native Hawaiian or Other Pacific Islander White Two or More Races Total (Auto calc) Sex Male Female Total (Auto calc)

31. 31 2.4.1.2/2.4.2.2 Students Served: Age Age in Years Neglected ProgramsJuvenile Detention Juvenile Corrections Adult Corrections/ At-Risk Other Programs 3 through 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 Total(Auto calc)

32. 32 2.4.1.2/2.4.2.2: Race/Ethnicity Polling Question: You have a facility that served 100 students, but 5 students did not self identify their race/ethnicity. What do you do? Have the facility submit what they think the race/ethnicity is when a respondent refuses to self identify their race/ethnicity Submit the data for the 95 students without a comment Submit the data and provide a comment about the missing data for five unidentified students Include the five students in a randomly chosen race/ethnicity category

33. 33 Example From Alaska: How To Submit Data on Students Who Age Out of a Program Alaska had a few students who aged out of the program (i.e., turned 22 during the school year). Initially, Alaska submitted the age data without these students and added a comment. NDTAC provided guidance to include these students as under age 21, because you are to report the age the student was when he/she left the facility/program (i.e., the student was no longer served by Part D after turning 22). Alaska has since updated its data collection tool to reflect the guidance they received.

34. 34 2.4.1.2/2.4.2.2 Data Checks # of Students Served Neglected Programs Juvenile Detention Total Unduplicated Students Served 276556 Race/Ethnicity Neglected Programs Juvenile Detention American Indian or Alaska Native 1025 Asian2025 Black or African American 90150 Hispanic or Latino 50100 Native Hawaiian or Other Pacific Islander 20100 White90150 Two or More Races 86 Total278556 Sex Neglected Programs Juvenile Detention Male200354 Female76200 Total276554

35. 35 2.4.1.2/2.4.2.2 Data Checks # of Students Served Neglected Programs Juvenile Detention Total Unduplicated Students Served 276556 Race/Ethnicity Neglected Programs Juvenile Detention American Indian or Alaska Native 1025 Asian2025 Black or African American 90150 Hispanic or Latino 50100 Native Hawaiian or Other Pacific Islander 20100 White90150 Two or More Races 86 Total278556 Sex Neglected Programs Juvenile Detention Male200354 Female76200 Total276554

36. 36 2.4.1.3.1/2.4.2.3.1 Transition Services In the first row of the table below, indicate whether programs/facilities receiving Subpart 1 funds within the State are legally permitted to track student outcomes after leaving the program or facility by entering Yes or No. In the second row, provide the unduplicated count of students receiving transition services that specifically target planning for further schooling and/or employment. If not, provide more information in the comment field. FAQ on facilities collecting data on student outcomes after exit: If only some, but not all, facilities in the State can collect data on student outcomes after exit, enter “yes” for the first question and provide a comment indicating why some facilities are unable to collect these data. Neglected Programs Juvenile Detention Juvenile Corrections Adult Corrections/ At-Risk Other Programs Are facilities in your State permitted to collect data on student outcomes after exit? (Yes or No) Number of students receiving transition services that address further schooling and/or employment

37. 37 2.4.1.3.1/2.4.2.3.1 Transition Services Are facilities legally permitted to collect data after exit? What to submit in table 2.4.1.3.1/2.4.2.3.1 Yes, all facilities are permitted to collect data after exit. Yes Some, but not all of the facilities in my State are permitted to collect data after exit. Yes Also provide a comment indicating why some facilities are unable to collect these data. No, none of the facilities are permitted to collect data after exit. No

38. 38 2.4.1.3.1/2.4.2.3.1 Transition Services If facilities are legally permitted to collect student outcome data after exit (as indicated by a “yes” in table 2.4.1.3.1/2.4.2.3.1), but lack the resources or capacity to collect the data (as indicated by zeros/blanks in table 2.4.1.3.2/2.4.2.3.2), please explain via comment. The leading indicators for Part D include two transition- related indicators: –Percentage of students who enrolled in a local district school after exit from a Part D program. –Percentage of students served who earned high school course credits up to 90 days after exit.

39. 39 2.4.1.3.1/2.4.2.3.1 Data Checks # of Students Served Neglected Programs Juvenile Detention Total Unduplicated Students Served 276556 Neglected Programs Juvenile Detention Are facilities in your State permitted to collect data on student outcomes after exit? (Yes or No) Yes0 Number of students receiving transition services that address further schooling and/or employment. 279556

40. 40 2.4.1.3.1/2.4.2.3.1 Data Checks # of Students Served Neglected Programs Juvenile Detention Total Unduplicated Students Served 276556 Neglected Programs Juvenile Detention Are facilities in your State permitted to collect data on student outcomes after exit? (Yes or No) Yes0 Number of students receiving transition services that address further schooling and/or employment. 279556

41. 41 2.4.1.3.2/2.4.2.3.2 Academic and Vocational Outcomes (1) The table of academic and vocational outcomes is reorganized into three smaller tables to group outcomes by (1) the setting in which the outcomes are achieved (in facility vs. out of facility) and (2) how many times students can achieve them. The instructions for the tables have been altered to reflect the new groupings: The first table includes outcomes a student can achieve only after exit. In this table, provide the unduplicated number of students who enrolled, or planned to enroll, in their local district school within 90 calendar days after exiting. A student may be reported only once, per program type. Outcomes (once per student, only after exit) Neglected Programs Juvenile Detention Juvenile Corrections Adult Corrections/ At-Risk Other Programs # of Students Who 90 days after exit Enrolled in their local district school

42. 42 2.4.1.3.2/2.4.2.3.2 Academic and Vocational Outcomes (2) The second table includes outcomes a student can achieve only one time. In this table, provide the unduplicated number of students who attained the listed outcomes either while enrolled in the LEA program/facility column (“in fac.”) or in the “90 days after exit” column. A student may be reported only once across the two time periods, per program type. Outcomes (once per student) Neglected Programs Juvenile Detention Juvenile Corrections Adult Corrections/ At-Risk Other Programs # of Students Who In fac. 90 days after exit In fac. 90 days after exit In fac. 90 days after exit In fac. 90 days after exit In fac. 90 days after exit Earned a GED Obtained high school diploma

43. 43 2.4.1.3.2/2.4.2.3.2 Academic and Vocational Outcomes (3) The third table includes outcomes that a student may achieve more than once. In the “In fac.” column, provide the unduplicated number of students who attained academic and vocational outcomes while enrolled in the LEA program/facility. In the “90 days after exit” column, provide the unduplicated number of students who attained academic and vocational outcomes within 90 calendar days after exiting. If a student attained an outcome once in the program/facility and once during the 90-day transition period, that student may be reported once in each column. Outcomes (once per student per time period) Neglected Programs Juvenile Detention Juvenile Corrections Adult Corrections/ At-Risk Other Programs # of Students Who In fac. 90 days after exit In fac. 90 days after exit In fac. 90 days after exit In fac. 90 days after exit In fac. 90 days after exit Earned high school course credits Enrolled in a GED program Accepted and/or enrolled into postsecondary education Enrolled in job-training courses/programs

44. 44 2.4.1.3.2/2.4.2.3.2 Data Checks # of Students Served Neglected Programs Juvenile Detention Total Unduplicated Students Served 276556 Outcomes (once per student) Neglected Programs Juvenile Detention # of Students Who In facility 90 days after exit In facility 90 days after exit Earned a GED782001682 Obtained high school diploma 14781332

45. 45 2.4.1.3.2/2.4.2.3.2 Data Checks # of Students Served Neglected Programs Juvenile Detention Total Unduplicated Students Served 276556 Outcomes (once per student) Neglected Programs Juvenile Detention # of Students Who In facility 90 days after exit In facility 90 days after exit Earned a GED782001682 Obtained high school diploma 14781332

46. 46 2.4.1.6.1/2.4.2.6.1 and 2.4.1.6.2/2.4.2.6.2 Academic Performance in Reading and Mathematics Performance Data (Based on most recent pre- /posttest data) Neglected Programs Juvenile Detention Juvenile Corrections Adult Corrections/ At-RiskOther Programs Long-term students with negative grade-level change from the pre- to posttest exams (optional for At-Risk) Long-term students with no change in grade level from the pre- to posttest exams (optional for At-Risk) Long-term students with improvement up to one full grade level from the pre- to posttest exams (optional for At-Risk) Long-term students with improvement of more than one full grade level from the pre- to posttest exams (optional for At-Risk)

47. 47 Example From California: Where are we and what do we want to accomplish? Purpose of Title I, Part D Funds To provide opportunity for youth to meet the same academic content standards as other students To operate high quality programs to prepare youth for secondary school completion, training, employment, or further education To provide activities to facilitate the transition of youth from correctional facilities to further education or employment To operate programs in local schools for youth returning from correctional facilities; and programs which may serve at-risk youth

48. 48 Example From California: How are we doing? Educational programs of the correctional facility help students meet state academic content standards. At least once every three years, the agency evaluates the Neglected or Delinquent Program, disaggregating data by gender, race, ethnicity, and age, to determine its effectiveness with students’: Ability to maintain and improve educational achievement. Completion of secondary school requirements and ability to obtain employment. Accrual of credits toward promotion and graduation. Transition to a regular program or other education program. Participation in postsecondary education and job training programs as appropriate. The agency uses multiple and appropriate measures of student progress in evaluating Neglected or Delinquent Programs. The agency uses evaluation results to improve programs.

49. 49 2.4.1.6.1/2.4.2.6.1 and 2.4.1.6.2/2.4.2.6.2 Data Checks # of Students ServedNeglected ProgramsJuvenile Detention Long-Term Students Served84 Performance Data (Based on Most Recent Pre-/Posttest Data) Neglected Programs Juvenile Detention Long-term students with negative grade-level change from the pre- to posttest exams 11 Long-term students with no change in grade level from the pre- to posttest exams 11 Long-term students with improvement up to one full grade level from the pre- to posttest exams 32 Long-term students with improvement of more than one full grade level from the pre- to posttest exams 32 Total86

50. 50 2.4.1.6.1/2.4.2.6.1 and 2.4.1.6.2/2.4.2.6.2 Data Checks # of Students ServedNeglected ProgramsJuvenile Detention Long-Term Students Served84 Performance Data (Based on Most Recent Pre-/Posttest Data) Neglected Programs Juvenile Detention Long-term students with negative grade-level change from the pre- to posttest exams 11 Long-term students with no change in grade level from the pre- to posttest exams 11 Long-term students with improvement up to one full grade level from the pre- to posttest exams 32 Long-term students with improvement of more than one full grade level from the pre- to posttest exams 32 Total86

51. 51 State Data Collection Tools and CSPR Timelines in States

52. 52 State Data Collection Tools Polling Question: How does your SEA collect data from its subgrantees? Word document Excel spreadsheet (no rollup feature) Excel spreadsheets with rollup feature Survey tool (Survey Monkey, SurveyGizmo, etc.) Customized online tool Other

53. 53 State Data Collection Tools Polling Question: How does your SEA conduct data quality checks? The Title I, Part D Coordinator conducts data quality checks “by hand” (e.g., eyeballs it, does rough calculations, recreates Excel equations each year). Data quality checks are programmed to check data once they are compiled across state. Data quality checks are programmed to pop up when each subgrantee submits data. Other. I’m not sure how the data quality checks are done.

55. 55 State CSPR Data Collection Timeline Polling Question: When does the SEA collect data from the subgrantees?