1. Nick Kremer, VERATAC Sharon Boivin, NCES Michelle Massie, WDQC

2. Overview – The challenge – gauging success The context—view from the field California specific example Federal efforts to improve data collection National effort to facilitate data matching/sharing

3. Workforce Accountability Chair the Vocational Education Research and Technical Advisory Committee (VERATAC) Field based committee advisory to State Chancellors Office Membership CTE deans Project administrators Faculty Institutional researchers Charge is to provide input into community college related accountability from a workforce perspective

4. California Context Outcomes Scorecard Degrees and certificates General measure and a CTE sub-measure Transfer ready, Momentum, remedial Salary Surfer Effort to expand outcomes Skill builders Take units but don’t complete a college credential State licensure Industry certification

5. Success –let a 1000 flowers bloom Current definitions limited to institutionally granted degrees and certificates What about Licenses Industry certification

6. View from the field We are concerned about licensure and third party certification because— Accountability Student success Feedback loop

7. The challenges -- licensure Effort to connect with state licensure boards unsuccessful 82 boards under Department of Consumer Affairs Concerns Data security/privacy Control over data Suspicion What will colleges do with it? Lack of perceived connection with colleges Colleges prepare students with content

8. A successful example Apprenticeship is governed by California Bureau of Apprenticeship standards Grant Journey person status after training and work requirements are fulfilled Willingly agreed to share information Once I asked Process took over 20 months MOU Technical Data will be part of State Scorecard and Perkins Accountability

9. Industry certification Examples: Cisco, NIMS, CompTia, Adobe, MCSC Colleges prepare students with content but testing by others Colleges don’t know result unless student shares Students many times don’t complete college degree or certificate and a classified as a drop out Matching with CompTIA not been very successful Lack of common identifier

10. Industry Certification The key is match data College course taking Testing results Finding common data field to facilitate matching is challenging Social security numbers Michelle will go into this in detail

11. Credential Data Pioneers: Forging New Partnerships to Measure Certifications and Licenses October 31, 2014 www.workforcedqc.org

12. Advocate for inclusive, aligned and market-relevant education and workforce data that can help our nation’s human capital policies meet the challenges of a changing economy. Promote federal and state reforms for data systems that provide useful information for policymakers, students and workers, business leaders and educators. –State Blueprint with 13 key features of a high-quality data infrastructure –Address federal legislation, funding and technical assistance –Policy agenda developed by broad coalition of national organizations, state leaders and technical experts across education/workforce spectrum WDQC Mission

13. National Partners

14. WDQC is grateful for the support of our funders: Apollo Education Group Bill & Melinda Gates Foundation The Joyce Foundation Lumina Foundation Funders

15. WDQC supports efforts to measure the full range of credentials — not just those associated with traditional education pathways. Data systems that capture all types of credentials can be used to: –Show policymakers a fuller picture of the skilled workforce –Help educators know whether their programs are appropriately preparing students to obtain credentials needed to advance in particular industries –Assist students and workers in making education, career and credentialing choices –Attract businesses seeking to expand or locate in areas with a supply of workers Meeting the Non-Degree Credential Challenge

16. These data linkages would enable richer and more complete analysis of education and training program alignment with industry requirements, as well as provide evidence on which licenses and certifications demonstrate value in the labor market over time. The report and webinar highlight states and schools that have taken steps to broker data sharing agreements with certification bodies and licensing agencies in order to better understand the attainment and value of selected non-degree credentials. Meeting the Non-Degree Credential Challenge

17. Credential Data Pioneers: Forging New Partnerships to Measure Certifications and Licenses To download the full report, visit: www.workforcedqc.org/resources-events www.workforcedqc.org/resources-events The Report

18. Names DOB (but not completely populated) Gender (not completely populated) Last 4 digits of SSN Place of birth (but very limited) If the licensee is an organization Issue date Expiration date Status of license: active, pending, expired, suspended, inactive, and others Date of status Addresses (some are out of state) Limited education (~1/3 of the licensees have school names, date enrolled, and limited program/degree information) Employment table (sparse) with job title, start and end date Indiana Case Study

19. Records date back to the late 1800s Majority of those are for registered nurses Documentation, definitions difficult to come by Database may be consolidated, but there are dozens of boards Indiana: Surprises

20. 40% match rate for entire 1.5 million records (matched to wage records) 80% match rate for records dated 2000 forward Unmatched Metrics: 17% were organizations and not people 30% had a null DOB and many of these were often due to the very early licenses. For more current integration, where there is no DOB we use additional identifiers for probabilistic matching. ~4% had a DOB before 1935 or after 1994 Remaining unmatched, most (~60%) had a license that was expired, denied, suspended, withdrawn, or inactive. Indiana: Linking Metrics

21. Built dynamic universe of professional licenses/certificates/etc. by Year of Active Status (updated monthly) Only ACTIVE license holders in each year based combination of status (active) and date of status and within the dates Persons only (no organizations) Linked to wage/employer records: Match based on each year, based on employment in at least one quarter with wages Match to NAICS (2 digit, 3 digit) industry Unemployment New Hires Indiana: The Build

22. Active “professionals” by year by type of license Average wage by type of license NAICS industry of employment by license type Unemployment and new hire patterns Public higher education linkage confirms education on license Indiana: The Results

23. Complete “business rules” and meta data based on actual certification and license requirements and state forms Publish http://iwis.in.gov Indiana: Next Steps

24. A voluntary effort to expand and improve data exchange between industry certification organizations and state longitudinal data systems by establishing a national data exchange clearinghouse Step 1: Raise awareness and gain consensus on need Step 2: Develop use cases and conduct pilot projects Step 3: Develop standards and guidelines on data sharing Step 4. Establish national data exchange clearinghouse Certification Data Exchange Project Case Study

25. Report on characteristics and employment outcomes of students who took and passed exams Based on a recent Illinois and CompTIA project Second Stage: –Round 1 States: California, Kansas, Maryland, North Carolina, Ohio and Oklahoma –Round 2 States: DC, Iowa, Kentucky and Washington Pilot Participants

26. Limited variables available in the CompTIA dataset –First name, last name, geographic code (zip) –CompTIA provided the individual-level data CompTIA records –1/1/2005 to 12/31/2010 –3,412 records provided from candidates who “opted in” Illinois Community College Board records –All credit students from fiscal years 2006–2010 (A1) –48 colleges with approximately 750,000 records per year Illinois Data Matching Protocol

27. – Facilitated data exchange through signing an indemnification agreement with CompTIA – ICCB reports provided additional variables –Demographics –Special populations –Major program areas –Employment and earnings –Statewide and by provider Illinois Process

28. 78 percent match achieved Certified students had slightly higher employment rates Certified students had substantially higher earnings –About 1/3 higher in the exam quarter –Almost 1 ½ times higher by the 3rd post exam quarter Certified students had better outcomes to begin with and the gap widened with the passage of time Illinois Results

29. Partners formed Certification Data Exchange Project and Advisory Group Created Multi-year Roadmap to develop a National Third-Party Certification Data Exchange Clearinghouse Execute data sharing agreements with one or both of the participating certification organizations. Develop a data matching protocol and conduct matching in cooperation with industry certification partners in conformance with applicable privacy and confidentiality laws and regulations. Develop data tables on number of test takers and passage rates by student demographics including age, gender and race/ethnicity as well as 2-digit, 4-digit and 6- digit CIP. Develop additional tables by special populations, if available. Develop data tables on employment and earnings of targeted students comparing students with and without certifications. www.acteonline.org/certification_data Next Phase

30. Trade Assistance Act Community College Career Training Grant – emphasis on certifications and tracking students California Community College System – initiating a new comprehensive data collection system and dashboard NC Community College System – Large Data Initiative Nation-wide interest and common data issues Workforce Credentials Coalition Case Study

31. Contact Michelle Massie, Policy Analyst michellem@workforcedqc.org 202-223-8355

32. New Data Sources to Measure Success Sharon A. Boivin Chair, Interagency Working Group on Expanded Measures of Enrollment and Attainment (GEMEnA) NCWE Conference October 31, 2014

33. Background The need for better data on education, training, and credentials for work became apparent during the recession President’s 2009 State of the Nation address called for every American to have at least one year of education or training beyond high school Interagency Working Group on Expanded Measures of Enrollment and Attainment (GEMEnA) formed in fall 2009

34. Percent of 25 to 34 year olds with at least a high school diploma who report “some college”, by state (Source: 2011 ACS)

35. GEMEnA A collaborative effort of federal statistical agencies and policy offices 1) to develop new survey items to measure the prevalence and key characteristics of non-degree credentials: Industry-recognized certifications Occupational licenses Educational certificates and…

36. 2) to improve federal data on enrollment and participation in education and training for work including initial on-the-job training programs such as apprenticeships and internships, non- credit college classes and skills oriented work- related training.

37. GEMEnA Member Organizations National Center for Education Statistics (NCES), Chair Bureau of Labor Statistics Census Bureau Council of Economic Advisors Department of Education, Office of the Under Secretary National Center for Science and Engineering Statistics (NSF) OMB Office of Statistical and Science Policy

38. GEMEnA Values Rigorous developmental work is time consuming and expensive Documentation contributes to the field of survey methodology, may undergird future work, and informs survey sponsors about the validity and reliability of items they may wish to adopt or adapt A collaborative approach to development and deployment ensures value and use

39. 5-Year Development Process Expert groups helped define the credentials Focus groups of potential respondents provided insight into relevant words/terms Cognitive interviews refined question wording and instructions Pilot studies tested survey items on a large number of respondents

40. Synergy happens when: Theoretical definitions developed by experts are used consistently in legislation, regulation, and accountability systems Operational definitions developed through respondent testing are used consistently in surveys

41. Theoretical Definitions— Certifications and Licenses Certification: A credential awarded by a certification body based on an individual demonstrating through an examination process that he or she has acquired the designated knowledge, skills, and abilities to perform a specific job. The examination can be either written, oral, or performance-based. Certification is a time-limited credential that is renewed through a recertification process. License: A credential awarded by a government agency that constitutes legal authority to do a specific job. Licenses are based on some combination of degree or certificate attainment, certifications, assessments, or work experience; are time-limited; and must be renewed periodically.

42. Is it a Certification or a License? Respondents do not easily distinguish between certifications and licenses particularly when the certification is required for the license Based on development work, surveys ask respondent to indicate whether the credential was issued by a government or other kind of certifying body.

43. Operational Definition—Certification/License Do you have a currently active professional certification or a state or industry license? Do not include business licenses, such as a liquor license or vending license. A professional certification or license shows you are qualified to perform a specific job and includes things like Licensed Realtor, Certified Medical Assistant, Certified Teacher, or an IT certification.

44. Theoretical Definition—Educational Certificate Educational certificate: A credential awarded by an educational institution based on completion of all requirements for a program of study, including coursework and test or other performance evaluations. Certificates are typically awarded for life (like a degree). Certificates of attendance or participation in a short-term training (e.g., 1 day) are not in the definitional scope for educational certificates.

45. Operational Definition—Certificate People sometimes earn certificates for completing an education or training program. A certificate is different from a certification or license. Do NOT include certifications/licenses here; include them in the previous section only. Have you ever earned any of the following types of certificates?

46. Let them tell us what they have a. A certificate for completing a training program from an employer, employment agency, union, software or equipment manufacturer, or other training company, but NOT from a school. b. A certificate for completing a vocational program at a high school. c. A high school equivalency certificate, such as a GED®. What we’re really after: d. A certificate—rather than a degree—for completing courses from a community or technical college, or other school after high school. Do not include college degrees.

47. First Results Census Bureau released the first official statistics in January 2014 Measuring Alternative Educational Credentials: 2012 http://www.census.gov/hhes/socdemo/educati on/data/files/p70-138.pdf http://www.census.gov/hhes/socdemo/educati on/data/files/p70-138.pdf

51. Data Currently Available 2008 Survey of Income and Program Participation (SIPP), Wave 13 topical module fielded in fall 2012 2002/2012 Education Longitudinal Survey 10 year follow up of 10 th grade class of 2002 (12 th grade class of 2004)

52. Data Available in the Near Future Redesigned SIPP (January 2015) Baccalaureate and Beyond (February 2015) Current Population Survey (February 2016) National Survey of College Graduates (2016) Beginning Postsecondary Students (2016) Credentials for Work Survey (2017)

53. Website: nces.ed.gov/surveys/gemena –List of surveys with links to questionnaires and data (as they become available) –Links to documentation on development process and published reports –GEMEnA-developed working definitions of certifications, licenses, and certificates –Link to LinkedIn group: GEMEnA Community

54. For More Information Sharon A. Boivin, Ph.D. Chair, Interagency Working Group on Expanded Measures of Enrollment and Attainment National Center for Education Statistics sharon.boivin@ed.gov