1. Postsecondary Accountability 2017 CTE Fall Webinar Series September 7, 2017
Denise
Your phone line is muted. If you have a question, type it in the chat box and we will respond to your questions throughout the webinar.
Again, your phone line is muted, type all questions in the chat section and refer to the PowerPoint side number when possible.

2. Today’s Presenters
Minnesota State Katie Vaccari Senior Research Associate Jeralyn Jargo System Director of Career Technical Education Debra Hsu Associate Director of Career Technical Education
Denise
Slide 2

3. Chat, Comment, React Your Phone Line Is Muted
Use Chat to send questions/comments
Look for links, info in Chat box
Send notifications to Presenters
Denise
Your phone lines are muted to eliminate feedback and background noise.
Use the chat box to ask your questions and comments throughout the webinar. Also look in the chat box for links and resources related to this topic.
Slide 3

4. Agenda
We will review:
Perkins postsecondary data structure and reporting status/accountability indicator definitions.
Target negotiation process.
Data sources for determining your accountability indicator performance.
How to access Perkins data in EPM 11.
Slide 4

5. Postsecondary Data Structure
Minnesota is one of a few states to use a cohort model. The use of cohorts aligns with other system initiatives and reporting.
Katie: Perkins postsecondary accountability reporting is based on a cohort model (i.e., following an entering cohort of students for a three-year period).
Minnesota is one of only a few states that does their federal Perkins postsecondary accountability reporting based on a cohort model. This model aligns with other types of reporting done on student success within the system.
The FY cohort includes students entering in FY 2015 and tracked for three years: 2015, 2016 and 2017.
Slide 5

6. Postsecondary Definitions
Completer
A student who received a CTE award (AAS, AS, Certificate or Diploma) in a Perkins eligible program.
Concentrator
A student with a Perkins eligible major who declared a CTE award (AAS, AS, Cert, Dip) and earned 12+ college level credits OR who received an award in a short term Perkins eligible program (less than 12 credits).
Katie: These “working” definitions of Perkins status and the Perkins core accountability indicators are important for understanding your reporting data and what makes up each core indicator.
Participant
A student taking a CTE course OR who has a Perkins eligible major.
Slide 6

7. Reporting Status —Completer —Concentrator —Participant
Status used for reporting is typically the highest recorded Perkins status during the cohort period.
If a Concentrator changes to non-Perkins major, that student will be counted as a Participant, not Concentrator.
Katie
Slide 7

8. 1P1 Technical Skill Attainment
Number of CTE concentrators who passed technical skill assessments
Numerator
Number of CTE concentrators who took technical skill assessments
Denominator
Katie: Next, we’ll move on from the Perkins reporting status definitions and review the definitions of the six postsecondary Perkins core indicators. These are the six indicators for which the state and each consortium are held responsible for setting and meeting performance targets.
1P1-includes some licensure exam data, usually at the aggregate level (nursing, law enforcement, EMT/paramedic and radiography) and data from a few third-party testing organizations (NOCTI, ASE – automotive, and NBCOT – occupational therapy).
Additional assessment results are incorporated into the technical skill assessment indicator (1P1) each year as data become available and data sharing agreements are established.
The reporting timeframe for 1P1 is different from the other core indicators. 1P1 includes a mixture of licensure exam data from the most recent calendar year available (at the time we assemble data for reporting) and data from NOCTI (or other third party assessments) from testers during the most recently completed fiscal (reporting) year.
Slide 8

9. 2P1 Credential, Certificate or Degree
Number of CTE concentrators in a given entry cohort who, anytime in the cohort time frame, received a CTE certificate, diploma, AAS or AS
Numerator
Katie
Number of CTE concentrators in the cohort
Denominator
December 2017 CAR will be based on the FY cohort.
Slide 9

10. 3P1 Student Retention or Transfer
Number of CTE concentrators in a given entry cohort who, at the end of the last year of the cohort time frame, were still intending to complete their program or had transferred to a 2-year or 4-year institution
Numerator
Katie
Number of CTE concentrators in the cohort
Denominator
December 2017 CAR will be based on the FY cohort. Retention and transfer data are frozen as of 8/31.
Slide 10

11. 4P1 Student Placement
Number of CTE completers in a given entry cohort who were placed or retained in employment in the 2nd quarter following the program year in which they left postsecondary education
Numerator
Katie
Denominator
Number of CTE completers
December 2017 CAR will be based on the FY cohort. There is a one year lag from other indicators.
Slide 11

12. 5P1 Nontraditional Participation
In a given entry cohort, the number of CTE participants during the cohort tracking period who were enrolled in a CTE program classified as nontraditional for their gender
Numerator
In a given entry cohort, the number of CTE participants during the cohort tracking period who were enrolled in a CTE program classified as nontraditional
Katie
Denominator
December 2017 CAR will be based on the FY cohort.
Slide 12

13. 5P2 Nontraditional Completion
In a given entry cohort, the number of CTE completers in a CTE program classified as nontraditional for their gender
Numerator
In a given entry cohort, the number of CTE completers in CTE programs classified as nontraditional
Denominator
Katie
Example:
# of Male Nursing Graduates + # of Female Welding Graduates
# Total Nursing Graduates + # Total Welding Graduates
December 2017 CAR will be based on the FY cohort.
Slide 13

14. Accountability Indicators: Data Sources
Postsecondary
Accountability Indicators:
Data Sources
Indicator
Data Source
1P1 – Technical Skill Attainment
Licensure data for Radiography, Law Enforcement, Nursing; added EMT/Paramedic licensure data beginning in 2016 (aggregate data only)
Other TSA data also included from NOCTI; ASE (Automotive) and NBCOT (occupational therapy) beginning in 2016
2P1 – Completion
3P1 – Retention and Transfer
5P1 – Nontraditional Participation
5P2 – Nontraditional Completion
ISRS (Integrated Statewide Record System) enrollment and completion data on CTE students
National Student Clearinghouse data on transfers to other colleges or universities
4P1 – Placement
Minnesota Dept. of Employment & Economic Development (DEED) and Graduate Follow-up data on employment following program completion
Katie: The data used to complete the postsecondary core indicator reporting comes from a number of sources.
Knowing the data sources is important to understanding how the core indicators are measured and understanding the types of changes and issues that may impact consortium performance on those measures.
You should have a basic understanding of the data sources for the core accountability indicators so you are aware of key impact points on the data/on the indicators.
Slide 14

15. Program Inventory Tool
Use the Program Inventory tool to produce a PDF of your institution’s current academic programs.
Slide 15

16. Program Inventory
Slide 16

17. State and Local Negotiations
Federal/State Negotiations
State/Local Consortia Negotiations
Evaluation of State Performance
Improvement Plans
Katie: The process of negotiating performance levels for core indicators and evaluating annual performance against those targets is similar between the state and local consortium level.
This process of establishing agreed upon levels of performance for all of the core indicators (secondary and postsecondary) is required at both the state and local level by the Perkins IV act. The process followed at both the state and local levels is quite similar. Both state and local negotiated levels of performance need to show continued progress toward improving the performance of CTE students. In addition, local performance targets are negotiated with the consortium (not with individual members of the consortium), so if you are a consortium with multiple colleges, you negotiate a single “consortium” target for each indicator (not a target for each college). That is a bigger issue for the secondary side where several schools/school districts are members of a single consortium.
Evaluation of Local Performance Improvement Plans
Slide 17

18. State Negotiated Postsecondary Targets
Katie: This slide shows what the state level negotiated targets on the six postsecondary core indicators were/are for a five-year timeframe along with actual state level performance on these indicators for We will begin preparing data for reporting on 2017 performance in next couple months.
Slide 18

19. State/Local Negotiation Process for FY 2018 Targets
Target Time Frame
Accountability Webinars: Postsecondary
September 7, 2017
Development of State Proposed Targets
November/December 2017
Distribute Proposed Targets to Consortia
December 2017
Consortia Responses Due
December 2017/January 2018
Initial State Response
January 2018
Local Negotiations Completed
January/February 2018
Evaluation of 2018 Performance
January 2019
Katie: Here are some of the dates that are important to keep in mind for the negotiation process.
Slide 19

20. Negotiating Your Targets
Review the proposed targets
Examine your Consortia & District level data
Look at trends/preliminary data if available
Discuss the targets with your partners/constituents
Have there been any changes in policy or practice that may have affected performance outcomes in the coming year?
Understand the context and rationale for the state proposed targets
Expectation of continuous improvement at the state and local level
Need for the state to be able to meet state targets negotiated with OCTAE
So, how does the target negotiation process work? What do you need to do?
Slide 20

21. Negotiating Your Targets
If you DO accept the state proposals:
Return the FY 2018 Negotiated Performance form
Include BOTH secondary AND postsecondary signatures
Determine if you will accept the proposed targets, or, propose an alternative target
If you DO NOT accept the state proposals:
Notify the state
Submit alternative target(s) along with the rationale for the proposed alternative(s)
Katie
Slide 21

22. Negotiating Your Targets
If submitting a counter proposal, the proposal must include:
The state proposal and your alternative proposed target
Trends in past performance
Rationale supporting the counter proposal, including any additional relevant data
The rationale must document the context of what is driving observed trends and factors that may impact future performance.
New or closed programs
TSA’s added, removed, or changed
Shifts in enrollment and the impact on programs
Changes in policies or practices that may impact program performance or enrollment
Katie
Keep in mind that in reviewing counter proposals the state must consider the ability of the state to meet their negotiated target (i.e., the ability of the other consortia to make up the difference).
Slide 22

23. Evaluation of Annual Local Performance
No action
At or above target
Negotiated Level
Report Required
Between 90% and 99%
Katie
90% of Negotiated Level
Improvement Plan
Below 90% of target
Slide 23

24. Accountability Indicators: How/Where Do I Get My Data?
Postsecondary
Accountability Indicators:
How/Where Do I Get My Data?
The System Office distributes four reports to Consortia Coordinators early in the calendar year (for the most recent reporting year):
Annual Core Indicator Report
Three-Year Trend Performance Report
1P1 (Technical Skill Attainment) Report
4P1 (Student Placement) Report
More detailed data for indicators 2P1, 3P1, 5P1 & 5P2 are available in EPM 11, from aggregate data all the way down to individual person level.
Katie: Data that are not available beyond an aggregate level and/or that are reported on a different timeframe from the majority of the postsecondary indicators (i.e., 1P1 and 4P1) are distributed to consortia as a PDF report once/year. Data for the other indicators are available in EPM 11, which allows consortium staff to “dig in” to that data and analyze it down to the student level (although it is also available rolled up at the aggregate level within EPM 11 as well). Official annual and trend core indicator performance reports are also distributed to consortia coordinators for ALL core indicators (secondary and postsecondary) as a PDF report once/year.
Slide 24

25. Annual Core Indicator Report
Katie
Your Annual Core Indicator Report, which specifies your consortium’s actual performance level as compared to your consortium’s negotiated target for each core indicator, is typically distributed in January/February.
Slide 25

26. Three-Year Trend Performance Report
Katie
A Trend Report showing performance status for the most recent three years along with an indication of whether technical assistance is needed for an indicator, is also distributed in January/February with the Performance Report.
Slide 26

27. Technical Skill Assessment Data (1P1)
Katie
The 1P1: Technical Skill Attainment report is prepared and distributed in January for the most recent reporting year. The report displays the consortium overall 1P1 performance rate as well as performance by specific assessments included in the measure
Slide 27

28. Student Placement Data (4P1)
Katie
The 4P1: Student Placement report is prepared and distributed in January for the most recent reporting year. The report displays the consortium overall 4P1 performance rate and performance by each of the special population and demographic groups.
Slide 28

29. EPM 11 Login
Data for Core Accountability Indicators 2P1, 3P1, 5P1 and 5P2 are available in EPM 11.
Katie
Slide 29

30. Student Carl Perkins/ Perkins IV Cohort Data Folder
Katie
Slide 30

31. Process Data for Your Institution
Katie
Slide 31

32. Select Your Institution RC_ID
Katie
Slide 32

33. CAR Report Info (Summary Page)
Katie
Slide 33

34. Results
Katie
Slide 34

35. Indicator Reports (Summary)
Katie
Slide 35

36. Pivot Tables by Indicator
Katie
Slide 36

37. EPM 11 – Accessing the Data
“Accessing Perkins IV Postsecondary Data Using EPM 11” document (scroll down to “Fall Webinar Series” section – Postsecondary Accountability):
Document also located in the main “Student Carl Perkins” folder in EPM 11.
Katie: “Accessing Perkins IV Postsecondary Data Using EPM 11” outlines the basic steps of accessing your Perkins data. A copy is located with this presentation on the CTE Professional Development website. There is also a copy located in the main “Student Carl Perkins” folder in EPM 11, so if you can get logged in and get to that folder, this document provides an overview for accessing the Perkins data available in EPM 11.
Slide 37

38. EPM 11/ITS/Other Links EPM 11 Login Page
Operational Data Security Request Form (add access for the Carl Perkins Data module)
EPM 11 Training (General)
ITS Service Desk
Reporting and Data Services SharePoint Site (Must log in with Star ID)
Program Inventory (official)
(Click “Program Inventory” link, which will open EPM 11)
Katie: Here are also some other sites that might be useful to you.
Slide 38

39. CTE Professional Development Resources
professionaldevelopment/monthly-webinars.html
Past Webinar Recordings and PowerPoints
Calendar of Monthly Webinars (Thursdays, 9am & 10:30am)
Sept 14, 10:30am: Preparing for your Annual Performance Review (APR)
Sept 21: Postsecondary Fiscal Management
Denise
Slide 39

40. Perkins Accountability ~ Postsecondary
Katie Vaccari, (651) Jeralyn Jargo, (651) Debra Hsu, (651)
Denise
Enter your questions in the “Chat” field OR request to unmute your phone line and ask your questions.
Slide 40