1. ONSITE TRAC TRAINING May 16, 2014

2. Joanne Mahony mahonyj@michigan.gov (517) 335-0405

3. TRAC OVERVIEW 3 T echnical R eview, A ssistance, and C ompliance The purpose of TRAC is: To provide technical assistance, and Monitor for compliance with State and Federal laws

4. Each Region Every Five Years Added Cost CTE Perkins CTE Instructional Programs 4 OVERVIEW OF TRAC PROCESS

5. MAJOR STEPS Pre-visit preparation and desk review Onsite visit Report to region Compliance plan submitted by region Compliance plan and evidence approved by OCTE 5

6. REVIEW CATEGORIES Data Finance Grants CTE Instructional Programs NOTE: The Office of Professional Preparation Services (OPPS) will join OCTE on TRAC reviews to complete OPPS review of Teacher Certification. More information will be given during OPPS Presentation this afternoon. 6

7. WARNING: Submission of documents that are subject to confidentiality and privacy laws should NOT be submitted electronically. These documents will be reviewed onsite. 7

8. QUESTIONS?

9. Elizabeth Collins collinse2@michigan.gov (517) 335-3149

10. TRAC TIMELINE (OCTE) What’s happened: Region selection Onsite visit notification TRAC training What’s ahead: Grant activity selections Fiscal agency selections CTE instructional program selections 10

11. TRAC TIMELINE (Region) Document Submission for Desk Review: CIP Self-Review documentation for selected programs is submitted through Navigator Data, Finance, and Grant documents submitted through GEMS (checklists) All documents are due September 8, 2014 Agenda (due three weeks prior to onsite) 11

12. TRAC TIMELINE Day 1 - Program reviews begin Day 2 - Program reviews continue Introduction meeting Data/Finance/Grant reviews begin Day 3 – Program reviews continue Interviews Wrap Up meeting 12

13. TRAC TIMELINE Desk Review and Onsite Visit OCTE Report Of Findings in GEMS Not In Compliance - Action Required Recommendations (comments, technical assistance, observations) Commendations (exemplary program, best practices) 13

14. TRAC TIMELINE Appeals Process Compliance Plan (created by Region) Compliance Plan and Evidence (Approved by OCTE) Final Letter 14

15. IMPORTANT CHANGES Expanded State Desk Review Documents Required No Compliance Plan Workshop No Priority (I/II) Findings Implementation of GEMS Navigator – CIP Programs 15

16. QUESTIONS?

17. Joanne Mahony mahonyJ@michigan.gov (517) 335-0405

18. GRANTS ALL desk review documents for Grant review must be uploaded to GEMS by September 8, 2014. Personally identifiable information must be redacted from documents before upload or documents must be provided onsite. 18

19. GRANTS G01 – 20% of CPI Indicator Activities from 2013-14 application Factors considered in selection process: CPI Deficiencies Regional Improvement Plan Activities Amount of grant funds budgeted Relevancy of activity 19

20. GRANT G01 DOCUMENTATION Evidence as identified and approved in the application must be provided Evidence must reflect the expected outcomes as identified in the application Evidence must be consistent with the scope of the activity and outcome 20

21. GO1 – Evidence Examples G01 evidence may include: Meeting minutes and sign-in sheets Pre- and post-test scores Revised curriculum Planning documents Documentation of services provided to students (payroll records and work logs) Dated vendor payments Other 21

22. EXAMPLE EXPECTED OUTCOME CPI: 1S1 Academic Attainment/Reading Activity: Curriculum Integration Expected Outcome: 60% of the region's CTE teachers will participate in professional development related to the integration of reading concepts and will integrate contextual reading activities into curriculum to support student growth in reading. 22

23. EXAMPLE 23

24. G01 - Evidences Provide Regional/CEPD/District documents (as appropriate) listed under DESK REVIEW for review items (participation may have been at the regional, CEPD or district level). No personally identifiable information 24

25. 25 Evidences Supporting: Regional Long Range Plan Eligibility for Perkins Funding (G02 – G06)

26. G02 - Equity and Access Documents that demonstrate: Students have equal access to CTE programs All students are included in outreach activities Compliance with federal law and regulations (includes eligible charter schools and home-schooled students) 26

27. G02 - Evidences Evidences may include: Counseling, advising and admission policies and procedures demonstrating equal access Outreach materials which may include brochures, videos, letters or flyers (including ELL languages as applicable) 27

28. G03 – Regional Planning/Stakeholder Documents that demonstrate: Grant priorities Joint planning and coordination of CTE programs WDB/EAG was provided the opportunity to review and provide input 28

29. G04 – Special Population Services Documents that demonstrate: Region/CEPD/District plan is in place to identify and provide services to SPOPS (LRP) Region/CEPDs/Districts have assessed and met the needs of SPOPS Region/CEPDs/Districts have conducted outreach targeting SPOPS 29

30. G05 – Professional Development Documents demonstrating Professional Development: Secondary teachers and personnel Administrators Career Guidance and Academic Counselors 30

31. G06 – Nonpublic and Charter Schools Documents that demonstrate compliance with federal laws and regulations regarding participation by students, teachers and administrators in: Nonprofit, nonpublic schools and home- schooled students Charter schools 31

32. G07 – Previous Onsite Only provide documents if requested after the desk review is completed. 32

33. David MacQuarrie macquarried@michigan.gov (517) 241-6202 Krishnan Sudharsan sudharsank@michigan.gov (517) 241-7652 Valerie Felder felderv@michigan.gov (517) 335-1066

34. 34 State Fund Reporting: 61a1 Course Enrollment Federal Fund Reporting: CTEIS Data Report Evidence District Evidence Accuracy between data evidences for Core Performance Indicators Report Use for Program Improvement: Share CPI Data Reports Use CPI Data for Program Improvement Policies and Activities that promote quality programs Focus of Data Review State Funding & Reporting Data Use for Program Improvement CTE Program Federal Funding and Reporting

35. DATA - Common Findings Course section minutes or weeks reported in CTEIS and shown on X0107 report, do not match district documentation: Master schedule Bell schedule District calendar Failure to provide bell schedule, master schedule or district calendar for ALL PSNs specified by OCTE for review 35

36. DATA - Common Findings Course section titles often don’t match those in CTEIS and shown on the X0107 report. Could result in findings of noncompliance If master schedule course names do not match course section names in CTEIS, please provide a crosswalk showing both course names Enrollments reported to OCTE don’t match teacher class/course records Please be sure to provide documents for the year under review (usually prior school year) 36

37. DATA - Common Findings Documentation not provided to demonstrate that: CPI reports were shared with instructors CPI reports were used for program improvement Follow Up Survey results, including related placement data shared with instructors (Examples of adequate evidence include meeting minutes or memos) 37

38. DATA - Common Findings Data processes and procedures documentation such as data security procedures, not provided 38

39. DATA - Common Findings Evidence not provided for all CEPDs or districts in Region. PSN-specific documentation is required for data reported at the course-section level (and is only required for the specified PSNs): Examples: Class lists Instructional Design Minutes Weeks 39

40. DATA - Common Findings Evidence not provided for all CEPDs or districts in Region (continued): Documentation demonstrating effort across the entire region is required for non-PSN-specific data activities: Examples: Sharing CPI results, Sharing follow-up results, Data security procedures Documentation must be provided demonstrating that activity was carried out at the regional level or, if carried out at CEPD or district level, a representative sample of documentation must be provided for every CEPD. 40

41. TIPS FOR REVIEW ITEMS D01-Perkins quality data: Provide teacher class lists for specified PSNs on-site (do NOT upload Personally-Identifiable Information (PII) to GEMS Provide PII onsite Examples: documents with student names, UICs, information on individual students, IEP (even without name) 41

42. TIPS FOR REVIEW ITEMS D02-Accurate CPI data: Teacher class lists for specified PSNs (see D01). Provide onsite only. Do not upload to GEMS. List of Students showing Special Populations category From DISTRICT records (not exported from CTEIS). Provide onsite only. Do not upload to GEMS. 42

43. TIPS FOR REVIEW ITEMS D02-Accurate CPI data: Teacher instructional design document for every PSN identified by OCTE (not printed from CTEIS) Articulation agreements for PSNs identified by OCTE Regional list of PSNs that have articulation agreements 43

44. TIPS FOR REVIEW ITEMS See review checklist for recommended evidence. Findings of noncompliance are common in: D03-CPI Reports/CPI Activities D04-Added Cost Quality Data Tip: provide counts of days of instruction on calendars and minutes of instruction on schedules. 44

45. TIPS FOR REVIEW ITEMS See review checklist for recommended evidence. Findings of noncompliance are common: D05-Placement data (follow-up survey) Tip: Remember to provide evidence for: Total placement Related placement CPIs used for program improvement CPIs shared with staff 45

46. TIPS FOR REVIEW ITEMS D06-CPI Report availability Tip: Provide web links to reports for all districts in the region D07-Data security Tip: Provide district-level policy documents for all districts in region. Tip: Agendas, meeting minutes or memos can be used to demonstrate that policies and procedures were shared with staff. 46

47. TIPS FOR REVIEW ITEMS D08-Previous monitoring review Only provide evidence if requested from OCTE after desk review Tip: Review prior TRAC Report to avoid repeat findings. 47

48. Naming Evidence Files In GEMS Label each file uploaded to GEMS with: Review item code (example: “D02”) PSN (for PSN-specific evidence) -or- District (for District-level evidence) -or- CEPD (for CEPD-level evidence) Description: (example: “InstructDesign”) EXAMPLE FILE NAME: Example: ReviewItemCode_PSN_Description PSN example: D02_12509_InstructDesign Building example: D04_Randolph_MasterSchedule District example: D04_DetroitPS_DistrictCalendar CEPD example: D03_CEPD41_AgendaWhereCPIsShared 48

49. For Evidence Provided Onsite Don’t forget to use: A sticky note on a page Highlight a sentence, paragraph, etc. Draw an arrow to the sentence, picture, etc. Provide crosswalks between district documents and or with CTEIS reports, if different 49

50. QUESTIONS?

51. BREAK TIME! Please reconvene in 15 minutes

52. Christine Black blackc11@Michigan.gov (517) 241-0260

53. FINANCE AGENCY REVIEWS 2013-2014 CTE Perkins fiscal agency and 1-3 additional participating agencies 53

54. F01 – Level of Effort: Supplement Not Supplant Documentation demonstrating that federal funds were used to supplement, not supplant, nonfederal funds expended to carry out career and technical education activities. 54

55. F01 - Evidences Desk Review documents: Board policy prohibiting supplanting Onsite documents: Interview with business official Interview with CTE administrator OCTE documentation 55

56. F02 - Procurement Documentation that region maintains a written code of standards of conduct governing the performance of its employees engaged in the award or administration of contracts; and maintains records sufficient to detail the history of a procurement transaction; and procurement procedures provide for full and open competition to the extent practicable, and no contract is made with a vendor that has been suspended or debarred. 56

57. F02 - Evidences Desk Review documents: Procurement/purchasing policies and procedure Board-approved Code of Conduct or Conflict of Interest policy for employees and officers involved in procurement Onsite documents: Evidence that the System for Award Management was checked ( www.sam.gov )www.sam.gov Employee-signed copy of Code of Conduct or Conflict of Interest (required if Board-approved Code of Conduct or Conflict of Interest policy is not evident) Interview with business official Interview with CTE administrator OCTE documentation 57

58. F03 – Expenditure Review Documentation demonstrating fiscal control and fund accounting policies and procedures ensure proper expenditure of and accounting for federal and nonfederal funds expended for career and technical education. 58

59. F03 – Evidences Desk Review documents: General ledger summary and transaction details (or equivalent) for federal CTE grant expenditures General ledger summary and transaction details (or equivalent) for nonfederal expenditures used to support required CTE Perkins grant activities Budget policies and procedures Accounting policies and procedures Process, procedures and forms used to monitor reimbursements to participating agencies (fiscal agency only) Chart of accounts (or equivalent) that defines accounting codes used in general ledger 59

60. F03 – Evidences Continued Onsite documentation: Supporting documentation for selected sample. Signed Cooperative Agreements for Designation of Fiscal Agency for ALL participating agencies generating CTE Perkins funds (fiscal agency only) Interview with business official Interview with CTE administrator or CTE Perkins federal grant coordinator OCTE documentation 60

61. F04 – Time and Effort Reporting Documentation demonstrating employees whose salaries and benefits are paid in whole or in part with federal funds maintain time distribution and personnel activity reports (PARS) in accordance with OMB Circular A-87. 61

62. F04 – Evidences Desk Review documents: List of employees and titles whose salary and benefits were paid in whole or in part with CTE Perkins funds. Payroll ledger Policy and procedures complying with federal time and effort reporting requirements 62

63. F04 – Evidences Continued Onsite documentation: Reconciliation of actual payroll costs to budgeted amounts Semi-annual certifications (100%) Personnel Activity Reports for split-funded personnel Interview with business official Interview with CTE administrator or CTE Perkins federal grant coordinator OCTE Documentation 63

64. PAR vs. Semi-Annual Certification There are only four (4) possible options to consider when determining whether to complete a semi-annual certification or personnel activity report. All possible circumstances will apply to one of the four options. 64

65. Timekeeping Decision Making Rubric 65

66. F05 – Property Management Documentation demonstrating prior written approval is required for purchasing equipment for state-approved career and technical education programs and that recipient's inventory management system meets the requirements in EDGAR. ONLY applicable if equipment was purchased with 2013-2014 Perkins funds. Equipment is defined as tangible, nonexpendable property having a useful life or more than one year and an acquisition cost of $5,000 or more per unit. 66

67. F05 - Evidences Desk Review documentation: Equipment inventory Board policy for capital outlay purchases (if not submitted for F02) Onsite documentation: Accounting records for instructional equipment purchases Interview with business official Interview with CTE administrator or CTE Perkins federal grant coordinator OCTE documentation 67

68. F06 - Previous Onsite Monitoring Visit Provide evidence for F06 only if requested by OCTE after the desk review. 68

69. FINANCE INTERVIEWS Interviews as needed with: Business office representatives CTE directors or administrators 69

70. FINANCE CHECKLIST Provide DISTRICT documents listed under DESK REVIEW for all review items Due to OCTE on or before September 8, 2014 ONSITE documents should be provided at the site. 70

71. QUESTIONS?

72. DESK REVIEW DOCUMENTS Before Onsite Visit Collect and organize materials electronically Virtual folders to prepare for GEMS Label with review item number (D01, F01, G01, etc.) and Document Name including CEPD, PSN, Building as necessary Mark pages/highlight sections (if possible) Provide all requested documentation Submit Desk Review documentation in GEMS on or before September 8, 2014. 72

73. Example of Virtual Folders 73

74. GEMS Access Each Region Liaison will need access to GEMS. This access is granted by your region’s GEMS Sub-recipient administrators. Information about this was sent to each Region Liaison via email on May 13, 2014. 74

75. GEMS Demonstration 75

76. ONSITE DOCUMENTS Only one copy of each item Documents should be placed in the document review room at the homebase location for day two of the agenda Materials remain with region 76

77. FINAL REMINDERS Documentation should represent the whole Region - not just one CEPD or district unless otherwise specified Don’t treat each CEPD separately; put all evidence for each review item together Have someone available on day two and three of the onsite visit who knows the documents Watch for program and grant selection notification via email on June 2, 2014. 77

78. Onsite Visits Dates Please see the TRAC Schedule in your folder for your region’s scheduled visit date. 78

79. OCTE TRAC COORDINATOR Elizabeth Collins (517) 335-3149 collinse2@michigan.gov 79

80. QUESTIONS?

81. Debbie Ryan Office of Professional Preparation Services ryand@michigan.gov 517-335-0585

82. Norma Tims timsn@michigan.gov 517-241-2091

83. Program Selection Criteria Added Cost funding amounts Enrollment and completion data Geographical location New program established within last five years New teacher for program OCTE program file information Program and building level core performance indicator data including special populations Regional CIP Summary information and CIP Improvement Plans 83

84. Program Selection Notification OCTE notifies Regional TRAC Liaison Regional TRAC Liaison notifies CEPD administrators CEPD administrator notifies building administrator, teacher(s), and OCTE consultant Teacher uploads all information into MCCTE Navigator by September 8, 2014 OCTE consultant contacts teacher(s) to finalize visit details and copies the CEPD administrator, building administrator, Regional TRAC Liaison, and OCTE TRAC Coordinator 84

85. Program Selection Follow-Up 2014-15 Master schedule highlighting program teacher(s) schedule Program Location - Building address Exceptions - Notation of teacher on leave, retired; program relocated, closed, etc. 85

86. CIP Self-Review Complete and upload in Navigator by PSN (http://ctenavigator.org)http://ctenavigator.org Single review for each PSN-all teachers Evidence for 2013-2014 school year Must be complete by September 8, 2014 86

87. CO1 – Program Teacher Certification Enter legal name(s) exactly as printed on teaching credential(s) Upload: Copy of state professional license, if applicable Copy of industry certification, if applicable Copy of Community College Instructor letter, if applicable 87

88. CO2 - Program Teacher PD To stay current with all aspects of program-related industry That promotes the integration of coherent and rigorous content aligned with challenging academic standards and relevant CTE. Effective integration and use of challenging academic in CTE provided jointly with academic teachers Effective research-based teaching skills Effective practices to improve parental and community involvement Effective use of research and data to improve instruction Provide relevant experience in the related business or industry (internship) Training in the effective use and application of technology to improve instruction. 88

89. Upload Professional Development Records To show: Date Sponsor(s) Topic(s) or focus of Professional Development 89

90. CO6 - Program Advisory Committee Upload: Advisory Committee roster-appropriate businesses, parent, and postsecondary Documentation of two meetings for year. Evidence of Advisory Committee assessment of and input on facilities, equipment, materials, and resources that reflect current industry standards and technology. 90

91. C10 – Implementation of Program Standards/Course Content Upload: Program Gap Analysis Syllabi for all courses within the program Program Instructional Design Form (Segments) Record of safety training (CIP specific) Documentation of work-based learning opportunities for students (CIP specific) Documentation of student leadership opportunities 91

92. Upload: Program Facilities (evidence of accessibility) Bias review results Recruitment materials/brochures Program applications/prerequisites Enrollment demographics (over/under representation of sub groups) 92 C13 – Strategies to Eliminate Barriers to Program Access

93. Upload one or more of the following: Program of Study (CIP specific) Articulation Agreement Concurrent/Dual enrollment agreement, if applicable Direct Credit agreement, if applicable 93 C16 – Secondary-Postsecondary Connections

94. QUESTIONS?

95. ONSITE TRAC TRAINING May 16, 2014