Postsecondary Accountability 2017 CTE Fall Webinar Series September 7, 2017 Denise Your phone line is muted. If you have a question, type it in the chat box and we will respond to your questions throughout the webinar. Again, your phone line is muted, type all questions in the chat section and refer to the PowerPoint side number when possible. http://www.minnstate.edu/system/cte/professionaldevelopment/index.html

Today’s Presenters Minnesota State Katie Vaccari Senior Research Associate Jeralyn Jargo System Director of Career Technical Education Debra Hsu Associate Director of Career Technical Education Denise Slide 2

Chat, Comment, React Your Phone Line Is Muted Use Chat to send questions/comments Look for links, info in Chat box Send notifications to Presenters Denise Your phone lines are muted to eliminate feedback and background noise. Use the chat box to ask your questions and comments throughout the webinar. Also look in the chat box for links and resources related to this topic. Slide 3

Agenda We will review: Perkins postsecondary data structure and reporting status/accountability indicator definitions. Target negotiation process. Data sources for determining your accountability indicator performance. How to access Perkins data in EPM 11. Slide 4

Postsecondary Data Structure Minnesota is one of a few states to use a cohort model. The use of cohorts aligns with other system initiatives and reporting. Katie: Perkins postsecondary accountability reporting is based on a cohort model (i.e., following an entering cohort of students for a three-year period). Minnesota is one of only a few states that does their federal Perkins postsecondary accountability reporting based on a cohort model. This model aligns with other types of reporting done on student success within the system. The FY 2015-2017 cohort includes students entering in FY 2015 and tracked for three years: 2015, 2016 and 2017. Slide 5

Postsecondary Definitions Completer A student who received a CTE award (AAS, AS, Certificate or Diploma) in a Perkins eligible program. Concentrator A student with a Perkins eligible major who declared a CTE award (AAS, AS, Cert, Dip) and earned 12+ college level credits OR who received an award in a short term Perkins eligible program (less than 12 credits). Katie: These “working” definitions of Perkins status and the Perkins core accountability indicators are important for understanding your reporting data and what makes up each core indicator. Participant A student taking a CTE course OR who has a Perkins eligible major. Slide 6

Reporting Status —Completer —Concentrator —Participant Status used for reporting is typically the highest recorded Perkins status during the cohort period. If a Concentrator changes to non-Perkins major, that student will be counted as a Participant, not Concentrator. Katie Slide 7

1P1 Technical Skill Attainment Number of CTE concentrators who passed technical skill assessments Numerator Number of CTE concentrators who took technical skill assessments Denominator Katie: Next, we’ll move on from the Perkins reporting status definitions and review the definitions of the six postsecondary Perkins core indicators. These are the six indicators for which the state and each consortium are held responsible for setting and meeting performance targets. 1P1-includes some licensure exam data, usually at the aggregate level (nursing, law enforcement, EMT/paramedic and radiography) and data from a few third-party testing organizations (NOCTI, ASE – automotive, and NBCOT – occupational therapy). Additional assessment results are incorporated into the technical skill assessment indicator (1P1) each year as data become available and data sharing agreements are established. The reporting timeframe for 1P1 is different from the other core indicators. 1P1 includes a mixture of licensure exam data from the most recent calendar year available (at the time we assemble data for reporting) and data from NOCTI (or other third party assessments) from testers during the most recently completed fiscal (reporting) year. Slide 8

2P1 Credential, Certificate or Degree Number of CTE concentrators in a given entry cohort who, anytime in the cohort time frame, received a CTE certificate, diploma, AAS or AS Numerator Katie Number of CTE concentrators in the cohort Denominator December 2017 CAR will be based on the FY2015-2017 cohort. Slide 9

3P1 Student Retention or Transfer Number of CTE concentrators in a given entry cohort who, at the end of the last year of the cohort time frame, were still intending to complete their program or had transferred to a 2-year or 4-year institution Numerator Katie Number of CTE concentrators in the cohort Denominator December 2017 CAR will be based on the FY2015-2017 cohort. Retention and transfer data are frozen as of 8/31. Slide 10

4P1 Student Placement Number of CTE completers in a given entry cohort who were placed or retained in employment in the 2nd quarter following the program year in which they left postsecondary education Numerator Katie Denominator Number of CTE completers December 2017 CAR will be based on the FY2014-2016 cohort. There is a one year lag from other indicators. Slide 11

5P1 Nontraditional Participation In a given entry cohort, the number of CTE participants during the cohort tracking period who were enrolled in a CTE program classified as nontraditional for their gender Numerator In a given entry cohort, the number of CTE participants during the cohort tracking period who were enrolled in a CTE program classified as nontraditional Katie Denominator December 2017 CAR will be based on the FY2015-2017 cohort. Slide 12

5P2 Nontraditional Completion In a given entry cohort, the number of CTE completers in a CTE program classified as nontraditional for their gender Numerator In a given entry cohort, the number of CTE completers in CTE programs classified as nontraditional Denominator Katie Example: # of Male Nursing Graduates + # of Female Welding Graduates # Total Nursing Graduates + # Total Welding Graduates December 2017 CAR will be based on the FY2015-2017 cohort. Slide 13

Accountability Indicators: Data Sources Postsecondary Accountability Indicators: Data Sources Indicator Data Source 1P1 – Technical Skill Attainment Licensure data for Radiography, Law Enforcement, Nursing; added EMT/Paramedic licensure data beginning in 2016 (aggregate data only) Other TSA data also included from NOCTI; ASE (Automotive) and NBCOT (occupational therapy) beginning in 2016 2P1 – Completion 3P1 – Retention and Transfer 5P1 – Nontraditional Participation 5P2 – Nontraditional Completion ISRS (Integrated Statewide Record System) enrollment and completion data on CTE students National Student Clearinghouse data on transfers to other colleges or universities 4P1 – Placement Minnesota Dept. of Employment & Economic Development (DEED) and Graduate Follow-up data on employment following program completion Katie: The data used to complete the postsecondary core indicator reporting comes from a number of sources. Knowing the data sources is important to understanding how the core indicators are measured and understanding the types of changes and issues that may impact consortium performance on those measures. You should have a basic understanding of the data sources for the core accountability indicators so you are aware of key impact points on the data/on the indicators. Slide 14

Program Inventory Tool http://www.asa.mnscu.edu/academicprograms/inventory/index.html Use the Program Inventory tool to produce a PDF of your institution’s current academic programs. Slide 15

Program Inventory Slide 16

State and Local Negotiations Federal/State Negotiations State/Local Consortia Negotiations Evaluation of State Performance Improvement Plans Katie: The process of negotiating performance levels for core indicators and evaluating annual performance against those targets is similar between the state and local consortium level. This process of establishing agreed upon levels of performance for all of the core indicators (secondary and postsecondary) is required at both the state and local level by the Perkins IV act. The process followed at both the state and local levels is quite similar. Both state and local negotiated levels of performance need to show continued progress toward improving the performance of CTE students. In addition, local performance targets are negotiated with the consortium (not with individual members of the consortium), so if you are a consortium with multiple colleges, you negotiate a single “consortium” target for each indicator (not a target for each college). That is a bigger issue for the secondary side where several schools/school districts are members of a single consortium. Evaluation of Local Performance Improvement Plans Slide 17

State Negotiated Postsecondary Targets Katie: This slide shows what the state level negotiated targets on the six postsecondary core indicators were/are for a five-year timeframe along with actual state level performance on these indicators for 2014-2016. We will begin preparing data for reporting on 2017 performance in next couple months. Slide 18

State/Local Negotiation Process for FY 2018 Targets Target Time Frame Accountability Webinars: Postsecondary September 7, 2017 Development of State Proposed Targets November/December 2017 Distribute Proposed Targets to Consortia December 2017 Consortia Responses Due December 2017/January 2018 Initial State Response January 2018 Local Negotiations Completed January/February 2018 Evaluation of 2018 Performance January 2019 Katie: Here are some of the dates that are important to keep in mind for the negotiation process. Slide 19

Negotiating Your Targets Review the proposed targets Examine your Consortia & District level data Look at trends/preliminary data if available Discuss the targets with your partners/constituents Have there been any changes in policy or practice that may have affected performance outcomes in the coming year? Understand the context and rationale for the state proposed targets Expectation of continuous improvement at the state and local level Need for the state to be able to meet state targets negotiated with OCTAE So, how does the target negotiation process work? What do you need to do? Slide 20

Negotiating Your Targets If you DO accept the state proposals: Return the FY 2018 Negotiated Performance form Include BOTH secondary AND postsecondary signatures Determine if you will accept the proposed targets, or, propose an alternative target If you DO NOT accept the state proposals: Notify the state Submit alternative target(s) along with the rationale for the proposed alternative(s) Katie Slide 21

Negotiating Your Targets If submitting a counter proposal, the proposal must include: The state proposal and your alternative proposed target Trends in past performance Rationale supporting the counter proposal, including any additional relevant data The rationale must document the context of what is driving observed trends and factors that may impact future performance. New or closed programs TSA’s added, removed, or changed Shifts in enrollment and the impact on programs Changes in policies or practices that may impact program performance or enrollment Katie Keep in mind that in reviewing counter proposals the state must consider the ability of the state to meet their negotiated target (i.e., the ability of the other consortia to make up the difference). Slide 22

Evaluation of Annual Local Performance No action At or above target Negotiated Level Report Required Between 90% and 99% Katie 90% of Negotiated Level Improvement Plan Below 90% of target Slide 23

Accountability Indicators: How/Where Do I Get My Data? Postsecondary Accountability Indicators: How/Where Do I Get My Data? The System Office distributes four reports to Consortia Coordinators early in the calendar year (for the most recent reporting year): Annual Core Indicator Report Three-Year Trend Performance Report 1P1 (Technical Skill Attainment) Report 4P1 (Student Placement) Report More detailed data for indicators 2P1, 3P1, 5P1 & 5P2 are available in EPM 11, from aggregate data all the way down to individual person level. Katie: Data that are not available beyond an aggregate level and/or that are reported on a different timeframe from the majority of the postsecondary indicators (i.e., 1P1 and 4P1) are distributed to consortia as a PDF report once/year. Data for the other indicators are available in EPM 11, which allows consortium staff to “dig in” to that data and analyze it down to the student level (although it is also available rolled up at the aggregate level within EPM 11 as well). Official annual and trend core indicator performance reports are also distributed to consortia coordinators for ALL core indicators (secondary and postsecondary) as a PDF report once/year. Slide 24

Annual Core Indicator Report Katie Your Annual Core Indicator Report, which specifies your consortium’s actual performance level as compared to your consortium’s negotiated target for each core indicator, is typically distributed in January/February. Slide 25

Three-Year Trend Performance Report Katie A Trend Report showing performance status for the most recent three years along with an indication of whether technical assistance is needed for an indicator, is also distributed in January/February with the Performance Report. Slide 26

Technical Skill Assessment Data (1P1) Katie The 1P1: Technical Skill Attainment report is prepared and distributed in January for the most recent reporting year. The report displays the consortium overall 1P1 performance rate as well as performance by specific assessments included in the measure Slide 27

Student Placement Data (4P1) Katie The 4P1: Student Placement report is prepared and distributed in January for the most recent reporting year. The report displays the consortium overall 4P1 performance rate and performance by each of the special population and demographic groups. Slide 28

EPM 11 Login Data for Core Accountability Indicators 2P1, 3P1, 5P1 and 5P2 are available in EPM 11. Katie www.its.mnscu.edu/reportanddataservices/datawarehouse/index.html http://www.its.mnscu.edu/reportanddataservices/datawarehouse/index.html Slide 29

Student Carl Perkins/ Perkins IV Cohort Data Folder Katie Slide 30

Process Data for Your Institution Katie Slide 31

Select Your Institution RC_ID Katie Slide 32

CAR Report Info (Summary Page) Katie Slide 33

Results Katie Slide 34

Indicator Reports (Summary) Katie Slide 35

Pivot Tables by Indicator Katie Slide 36

EPM 11 – Accessing the Data “Accessing Perkins IV Postsecondary Data Using EPM 11” document (scroll down to “Fall Webinar Series” section – Postsecondary Accountability): http://www.minnstate.edu/system/cte/professionaldevelopment/monthly-webinars.html Document also located in the main “Student Carl Perkins” folder in EPM 11. Katie: “Accessing Perkins IV Postsecondary Data Using EPM 11” outlines the basic steps of accessing your Perkins data. A copy is located with this presentation on the CTE Professional Development website. There is also a copy located in the main “Student Carl Perkins” folder in EPM 11, so if you can get logged in and get to that folder, this document provides an overview for accessing the Perkins data available in EPM 11. http://www.minnstate.edu/system/cte/professionaldevelopment/monthly-webinars.html Slide 37

EPM 11/ITS/Other Links EPM 11 Login Page http://www.its.mnscu.edu/reportanddataservices/datawarehouse/index.html Operational Data Security Request Form (add access for the Carl Perkins Data module) http://www.mnscu.edu/system/its/security/securityforms/mnscuinfosystems/pdf/OperationalDataSecurityForm.pdf EPM 11 Training (General) http://www.its.mnscu.edu/reportanddataservices/training/index.html ITS Service Desk https://servicedesk.mnscu.edu/CherwellPortal/MNSO#0 Reporting and Data Services SharePoint Site (Must log in with Star ID) https://mnscu.sharepoint.com/sites/its/SitePages/Page.aspx?topicID=29&topicName=Reporting%20and%20Data%20Services Program Inventory (official) (Click “Program Inventory” link, which will open EPM 11) http://asa.mnscu.edu/academicprograms/inventory/index.html Katie: Here are also some other sites that might be useful to you. Slide 38

CTE Professional Development Resources www.minnstate.edu/system/cte/ professionaldevelopment/monthly-webinars.html Past Webinar Recordings and PowerPoints Calendar of Monthly Webinars (Thursdays, 9am & 10:30am) Sept 14, 10:30am: Preparing for your Annual Performance Review (APR) Sept 21: Postsecondary Fiscal Management Denise Slide 39

Perkins Accountability ~ Postsecondary Katie Vaccari, (651) 201-1718 Katie.Vaccari@minnstate.edu Jeralyn Jargo, (651) 201-1650 Jeralyn.Jargo@minnstate.edu Debra Hsu, (651) 201-1686 Debra.Hsu@minnstate.edu Denise Enter your questions in the “Chat” field OR request to unmute your phone line and ask your questions. Slide 40