1. TRAINING PHASE 1

2. Part one: The Survey
Purpose: To discover what is well-known and what specific training needs there are across the state on
WIOA performance indicators and cohorts
Staff knowledge of WF1 data entry requirements
The Statistical Adjustment Model
Local area negotiation process

3. Survey Results
Information added to the webinar after reviewing survey responses
Possible reasons numerators and denominators are different than you expected
Demonstration on the cohort tool created by MN Performs
Eligible Training Provider List requirements
Measurable Skill Gains data entry

4. Three repeated training webinars
Part two: The Webinar
Three repeated training webinars
January 26, 30, and 31
1:00 – 4:00 PM

5. Each Split into Two Sections
First Section’s Objective
Refresher on WIOA Performance and the current
Statistical Model Mechanics
Second Section’s Objective
Review of important data entry elements impacting
performance negotiations and other important information

6. Specific Topics in the First Section Estimated Length of Time
Performance Standards, Indicators, Cohorts, and
The Statistical Model’s key definitions, formulas, old data cohorts, and Tool Demonstration
Estimated Length of Time
2 Hours
Targeted Audience
Managers/Directors/Performance Staff
(Numbers People)

7. Specific Topics in the Second Section Estimated Length of Time
Workforce One Data Entry: Where participant demographic data impacting the statistical model is entered in the system and what data is pulled from other databases. Review ETPL requirements, the ETPL website, and Measurable Skill Gains
Estimated Length of Time
1 Hour
Targeted Audience
Case Managers/Data Entry Clerks/Performance Staff

8. Performance and Data Coordinator
First Section WIOA Adult and Dislocated Worker Standards, Cohorts, and the Statistical Adjustment Model ,
Performance and Data Coordinator

9. Program Year 2016 and 2017 State Performance Standards

10. WIOA Performance Indicators
Performance Measure
Numerator
Denominator is
Activity in WF1 that triggers performance
2nd Quarter Employment
Has at least $1 in Wage Detail in the 2nd quarter after Exiting
All Exiters during reporting period except those exited with exclusion
Any WF1 Activities
4th Quarter Employment
Has at least $1 in Wage Detail in the 4th quarter after Exiting

11. WIOA Performance Indicators
Performance Measure
Calculation
Included
Activity in WF1 that triggers performance
Median Earnings
See Below
Total quarter wages for each participant showing wages in the 2nd quarter after exiting except for those exited with exclusion
Any WF1 Activities

12. WIOA Performance Indicators
Performance Measure
Numerator
Denominator
Activity in WF1 that triggers performance
Credential Attainment (Post-Secondary)
Credential was captured in WF1, Attainment date was during enrollment or within 1-year of exit, AND the type of credential attained is recognized
Date of exit is within reporting period AND was enrolled in "Post-Secondary" except those exited with exclusion
Apprenticeship, Classroom Training, Credential Attained without Training, Entrepreneurial Training, Incumbent Worker Training, and GED training if the GED student exits to employment

13. Baseline Indicator for Program Years 2016 and 2017
Performance Measure
Numerator
Denominator
Activity in WF1 that triggers performance
Measurable Skills Gain
Documented increase of functioning level, transcript, report card, milestone, or skills progression during reporting period
All participants attending training during the fiscal year leading to a recognized postsecondary credential or employment except those exited with exclusion. It encompasses all WIOA training, OJT, Apprenticeship, and customized training program enrollments.
Apprenticeship, Classroom Training, Credential Attained without Training, Customized Training, Entrepreneurial Training, GED Training, Incumbent Worker Training, Incumbent Worker Credential Waiver, Occupational Skills Training, OJT, OJT - Public or Private, Prerequisite Training, Remedial Training, TAA Apprentice

14. Why does everyone keep saying MULLIGAN YEAR

15. Program Year 2016 WIOA Cohorts
Measure
PY17 Based On
Full Reportable Year?
2nd Quarter Employment
No data available
NO – No employment data available
4th Quarter Employment
2nd Quarter Median Earnings
Credential Attainment
Measurable Skills Gain
All enrolled and in Training 07/01/2016 – 06/30/2017
YES

16. Program Year 2017 WIOA Cohorts
Measure
PY17 Based On
Full Reportable Year?
2nd Quarter Employment
Exiters 07/01/2016 – June 30, 2017
YES
4th Quarter Employment
Exiters 07/01/2016 – December 2016
NO – Only 2 quarters of data available
2nd Quarter Median Earnings
Credential Attainment
Measurable Skills Gain
All enrolled and in Training 07/01/2017 – 06/30/2018

17. Program Year 2018 WIOA Cohorts
Measure
PY17 Based On
Full Reportable Year?
2nd Quarter Employment
Exiters 07/01/2017 – June 30, 2018
YES
4th Quarter Employment
Exiters 01/01/2017 – December 2017
2nd Quarter Median Earnings
Credential Attainment
Measurable Skills Gain
All enrolled and in Training 07/01/2018 – 06/30/2019

18. WIOA Cohorts

19. MN Performs Cohort Tool Demonstration

20. What does Exited with Exclusion Mean?
If the participant is exited with any of these exit reasons that participant will not be counted in performance outcomes
DOL Definition of Authorized Exclusion Based on TEGL Attachment 2
WF1 Exit Reason
Exits while receiving services due to incarceration in correctional facility or becomes resident of institution providing 24 hour support such as hospital or treatment center
Institutionalized
Exits after being called up to active duty for at least 90 days as member of National Guard or other reserve military unit.
Reservist Called to Active Duty
Participant is deceased
Death
Exits due to medical treatment expected to last longer than 90 days which would preclude employment or continued participation.
Medical Treatment

21. What WF1 Exits Reasons will be counted in
Performance Measures
DEED will check wage detail/Supplemental Wage Information and other performance indicator data for participants exiting with these final Exit Reasons
Programs Using
Called Back
Dislocated Worker
Cannot Locate
Dislocated Worker and Adult
Entered Armed Forces
Entered Registered Apprentice Training
Entered Unsubsidized Employment
Family Care Problems
Found Ineligible
Moved from Area
Program/Type Transfer
Refused to Continue
Remained Employed
Retirement
Started Business/Self-Employed

22. Why participants with “positive” WF1 exit reasons might not be positive in official performance
* A pseudo Social Security Number was used on the case
* Wages were not found in wage detail reports
* Supplemental wage information was not entered into
WF1’s Follow-Up tab
* DOL considers the original exit reason and/or the
secondary exit reason exclusionary

23. Why numerators and denominators might be different than expected
Participants who access multiple providers
If a participant accessed services from a provider then after exiting they reenrolled with a new provider
within 90 days
Performance counts
go back to the original provider

24. Why numerators and denominators might be different than expected
Participant who only access State funds
If all activities on the participant’s case have State Dislocated Worker funding selected within them
That participant will not be included in
WIOA Dislocated Worker measures
At least one activity on the case needs to be supported by WIOA Dislocated Worker funding

25. TEGL 26-15 The Statistical Adjustment Model

26. Key Definitions Target Outcome – The level expected (standard)
Predicted Outcome – The level at which service area will most likely be
Simulated Outcome – The individual records for the last program year of each indicator's exiter cohort found in the Wagner-Peyser Employment Service data. (WIASRD)

27. You can get a national list of all county FIPS here
Key Definitions
FIPS – stands for Federal Information Processing Standard. It is a five-digit code which uniquely identifies counties and county equivalents in the United States. The first two digits are the FIPS state code and the last three are the county code within the state.
Minnesota’s FIPS is 27
You can get a national list of all county FIPS here

28. How Many FIPS do you Serve?
Local Area
WIB Code
Number of County (fips) Codes in Service Areas
WDA 01 - NW MN PIC
27045 7
WDA 02 - Rural CEP
27040 19
WDA 03 - NEMOJT
27035
WDA 04 - City of Duluth
27005 1
WDA 05 - CMJTC
27105 10
WDA 06 - SW MN PIC
27055 14
WDA 07 - S Central
27030 9
WDA 08 - SE WDI
27075
WDA 09 - Hennepin/Carver
27120 2
WDA 10 - METP
27010
WDA 12 - Anoka
27085
WDA 14 - Dakota/Scott
27125
WDA 15 - Ramsey
27115
WDA 16 - Washington
27100
WDA 17 - SBETC
27110
WDA 18 - Winona
27080

29. How is the Statistical Model Calculated?

30. The Statistical Model: Formula
Target Outcome
Multiply each coefficient by the population percentage within that program indicator’s cohort. Repeat this for every coefficient and its input value.
Add up all the products found in step 1.
Calculate the average state fixed effect by averaging all state fixed effects for that indicator.
Add the results from step 2 and step 3 to get the Target Outcome
In other words use this formula “=SUMPRODUCT($C$9:$C$58,$D9:$D58)+C61”

31. In other words use this formula “=SUMPRODUCT($C$9:$C$58,$D9:$D58)+C85”
Formula (continued)
Predicted Outcome
Multiply each coefficient by the population percentage within that program indicator’s cohort. Repeat this for every coefficient and its input value.
Add up all the products found in step 1.
Add Minnesota’s fixed effect for that indicator to the results from step 2 to get the Predicted Outcome
In other words use this formula “=SUMPRODUCT($C$9:$C$58,$D9:$D58)+C85”

32. Formula (continued) Simulated Outcome
Does not use an average state fixed effect
This calculation is based only on the participants who were actually served during the year the data was pulled

33. What participants are included in the model’s data

34. Each indicator uses different participant data
Adult Indicators
- Employment Rate 2nd Quarter After Exit Cohort =
Exit Date from 7/1/13 to 6/30/14 and Adult
- Median Earnings 2nd Quarter After Exit Cohort =
and Wages 2nd Quarter After Exit > 0
and Wages 2nd Quarter After Exit ~=

35. The Statistical Model: Demographic Data Used
Adult Indicators
- Employment Rate 4th Quarter After Exit Cohort =
Exit Date from 1/1/13 to 12/31/13 and Adult
- Credential Attainment Rate Cohort =
Exit Date from 10/1/13 to 9/30/14 and Adult
and Received Training

36. The Statistical Model: Demographic Data Used
Dislocated Worker Indicators
- Employment Rate 2nd Quarter After Exit Cohort =
Exit Date from 7/1/13 to 6/30/14 and WIA DW
(excluding NEGs)
- Median Earnings 2nd Quarter After Exit Cohort =
and Wages 2nd Quarter After Exit > 0
and Wages 2nd Quarter After Exit ~=

37. The Statistical Model: Demographic Data Used
Dislocated Worker Indicators
- Employment Rate 4th Quarter After Exit Cohort =
Exit Date from 1/1/13 to 12/31/13 and WIA DW
(excluding NEGs)
- Credential Attainment Rate Cohort =
Exit Date from 10/1/13 to 9/30/14 and WIA DW
and Received Training

38. DID YOU KNOW The Statistical Model has more than your agency information in it In fact, it has more than the state of Minnesota’s information in it WHY? It is a required tool for every state and WDA in the nation

39. Where do you find the Statistical Model created by the Department of Labor
TEGL provides guidance on PY16 and PY17 negotiations using the statistical model
A new TEGL is due for release in February 2018 for guidance on PY18 and PY19 negotiations -

40. Differences between the model on DOL’s site and the Model AD/DW staff email you
DOL’s Statistical Model found on their website
Does not have data narrow to your WIB or FIPS
Does not calculate or input your region’s Labor Market Information
Does not calculate or input your region’s Unemployment rate
Statistical Model ed to you from Adult/Dislocated Worker program staff
Narrows the data to your WDA’s WIB
Narrows the data to FIPS served by your WDA
Calculates your region’s Labor Market Information and Unemployment rates then inputs it into the WIA Data tab (directions for these calculations are in the SOP tab)

41. Program Staff does for you
Statistical Model LMI
Construction Employment
Educational, or Health Care Related Employment
Federal, State, or Local Government Employment
Information, Finance, or Technology Employment
Leisure, Hospitality, or Entertainment Related Employment
How it works
DEED calculates all of these for each local area using the “SOP” tab instructions provided by DOL prior to ing statistical models to WDAs
Depending on the performance indicator you are researching, there are multiple Industry lag and Unemployment Rate lag tabs to use
Lag
Program(s)
Measure(s) A
Adult, DW, Youth
Credential Attainment
Youth
Q2 employment rate B
Adult, DW, WP
Q2 employment rate, Median earnings C WP
Q4 employment rate D
Adult, DW

42. Program Staff does for you
Statistical Model
LMI and Unemployment Rate
Manufacturing Related Employment
Natural Resources Employment
Other Services Employment
Unemployment Rate Not Seasonally Adjusted
How it is works (continued)
Local area calculations are based on the FIPS codes (county codes) in which the WDA serves
Individual industry employment counts for all FIPS that WDA serves are summed then divided by the total employment count in their service area to get the regional percentage
Enter each percentage into the appropriate WIA tab Data columns and program indicator lines
(Columns BO – BZ)

43. Statistical Model Demonstration

44. PY2018-2019 Negotiation TEGL Estimated Release: Late February 2018
Known differences between the new TEGL and TEGL 26-15
New TEGL is for Negotiating PY2018 & 2019 instead of PY2016 & 2017
New TEGL has updated data applied to the Statistical model using PY2016 data instead of PY2014 data

45. REMINDER Phase 2: Training Webinar coming February _________________________________________________________________ How to update the model with new data using MN Performs

46. Questions?

47. Now we know about the model but where does the model get its data?

48. Performance and Data Coordinator
Second Section Workforce One (WF1) Data entry impacting the Statistical Adjustment Model ,
Performance and Data Coordinator

49. Most of the Data comes from these areas of WF1 The Eligibility and Enrollment Screens Support Services tab and within participant activities

50. Entered into the WF1 Eligibility Screen
Statistical Model Elements in
Age 26 to 35,
Age 36 to 45,
Age 46 to 55,
Age 56 to 65,
and Age 66+
Female

51. Entered into the WF1 Enrollment Screen
Statistical Model Elements in
Employed at participation
Homeless

52. Entered into the WF1 Enrollment Screen
Statistical Model Elements in
Hispanic ethnicity
American Indian or Native Alaskan, Asian, Black, Hawaiian/Pacific Islander, or More than one Race

53. Entered into the WF1 Enrollment Screen
Statistical Model Elements in
Highest grade completed:
Associates Degree
Bachelor Degree
Certificate or Other Post- secondary Degree
High School Equivalency
Less than High School Graduate
Some College

54. Entered into the WF1 Enrollment Screen
Statistical Model Elements in
Individual with a disability
Limited English-language proficiency
Offender

55. Entered into the WF1 Enrollment Screen
Statistical Model Elements in
Single Parent
UI exhaustee
UI Claimant, Non-exhaustee

56. Entered into the WF1 Enrollment Screen
Statistical Model Elements in
TANF Recipient
Other Public Assistance Recipient
At least One: Yes

57. Entered into the WF1 Enrollment Screen
Statistical Model Elements in
Low Income
Veteran
At least One: Yes

58. Entered into WF1 Activity Screens
Statistical Model Elements in
Pell Grant Recipient
Received pre-vocational activity services
Required in Credential activities
Optional in Non-credential activities

59. Entered into WF1 Activity Screens
Statistical Model Elements in
Received Training Services
(WIOA Certified)
Established Individual Training Account (ITA)

60. Entered into WF1 Activity Screens
Statistical Model Elements in
Received Intensive Services

61. Entered into WF1 Support Service Screen
Statistical Model Element: Received Support Services
At least one using WIOA funds during enrollment period

62. Where do other elements come from?

63. Elements NOT Captured from WF1 (Besides LMI and Unemployment Rates)
Statistical Model Elements
Had earnings in 2nd and 3rd preprogram quarters
Had earnings in 2nd preprogram quarter
Had earnings in 3rd preprogram quarter
Received services financially assisted under the Wagner-Peyser Act
Received needs-related payments
How it is captured in Reports
Wage Detail
Minnesotaworks.net account
Not provided for WIOA Adult and Dislocated Worker programs in Minnesota

64. Since we have time… How about we talk about other important stuff

65. Eligible Training Provider List (ETPL)
Managed by Rachel Vilsack

66. https://mn.gov/deed/data/data- tools/career-education-explorer/
Located here
tools/career-education-explorer/

67. The ETPL MUST be used for
Credential
and Non-credential
WIOA funded Training

68. WIOA funded Credential Training Must be listed on the ETPL as
WIOA Certified

69. Non-Credential Training Must be listed on the ETPL
WIOA funded
Non-Credential Training
Must be listed on the ETPL

70. Measurable Skill Gains (MSG)

71. What is the minimum? AT LEAST ONE NEEDS TO BE CAPTURED
EVERY FISCAL YEAR

72. Five MSG Options
Educational Functioning Level Date: New documented assessments showing an increase in abilities;
Secondary Report Card Date: Documented attainment of a secondary school diploma or its recognized equivalent;

73. MSG Options
Post-Secondary Transcript Date: showing sufficient number of credit hours in accordance with academic standards;
Training Milestone: OJT or Apprenticeship progress report from the employer or training provider;

74. MSG Options
Skills Progression Date: Successful passage of an exam that is required for a particular occupation or progress in attaining technical or occupational skills

75. Where is it entered in WF1? “Reporting Collection”
The ONLY place
to capture
MEASURABLE SKILL GAINS

76. Reporting Collection
2. Click here
1. Click here

77. Click here

78. 1. Choose the correct one
2. Enter the attainment date
3. Click OK

79. Don’t forget to click “Save”!
(The most recently MSG is listed on the bottom)
Don’t forget to click “Save”!

80. REMINDER Phase 2: Training Webinar coming February _________________________________________________________________ How to update the model with new data using MN Performs

81. WIOA Adult and Dislocated Worker programs Performance Coordinator
Questions?
Amy Carlson
WIOA Adult and Dislocated Worker programs
Performance Coordinator