Initial Research Findings from Arkansas’ Statewide Longitudinal Data System: Using Standards-Based Research to Solve Real Education Problems Jake Walker, PhD, Project Director Arkansas Research Center Greg Nadeau, Director, State Core Model, Public Consulting Group 2012 MIS Conference 1

The CEDS Data Model Process CEDS Elements Domain Entity Schema Normalized Data Schema 2

A foundation for standards convergence

CEDS Domain Entity Schema (DES) DomainEntity ELEL Child ELEL Enrollment ELEL Assessment ELEL Program ELEL Group ELEL Staff ELEL Parent DomainEntity K12K12 Student K12K12 Incident K12K12 Parent K12K12 Staff K12K12 Staff Assignment K12K12 Section K12K-12 Course K12K12 Migrant Facility K12K12 Program K12K12 School K12K12 District K12State Education Agency K12Assessment K12Learning Standard DomainEntity PSPS Student PSPS Assessment PSPS Staff PSPS Section PSPS Program PSPS Defined Program PSPS Teacher Program PSPS Campus PSPS School PSPS Institution

CEDS NDS Conceptual Model OrganizationsPeople Assessment Standards [Content] Roles Time

CEDS Logical Model (NDS): High Level 6

7 Parsing NCES Forum Questions with a Standard Syntax

Using CEDS for Analysis #Logic Statement 1The purpose of this artifact is to provide a multi-criterion decision framework to improve Secondary Data Analysis from P-20 Longitudinal Data Systems. It assumes a set of education and workforce data that is common across the 50 states. 2In this artifact we will lay out a P-20 Standard Analytic Model (P-20 SAM) built off of the new CEDS 2.0 Data Model. 3The P-20 SAM is a framework for quantitative research and performance management utilizing valid metrics based on standard data definitions and relationships. It can be used as a logical framework for dimensional models. 4Valid metrics are aggregate measures (counts, averages, ratios) that have been demonstrated to correlate with educational indicators. For example Student Growth Percentile (SGP) is a measure that correlates with K12 quality learning and Degree Production Rate (DPR) is a measure that correlates with postsecondary effectiveness. A list of P-20 Metrics is provided with a standard definition with transformation rules tied directly to standard CEDS DES element definitions and NDS tables and fields. 5A list of P-20 Metrics is provided with a standard definition with transformation rules tied directly to standard CEDS DES element definitions and NDS tables and fields. 6The P-20 SAM provides Standard Syntax for expressing research questions and performance reports. This syntax must specify the metric, entity/ies, and daterange/s to be considered. 7Metrics are often expressed as rates. Rates express the ratio between a sub-group count (little n or numerator) and a general population count (big N or denominator). 8Application of a metric to a given data set must consider the quality and quantity (n-size) of the data as well as the sensitivity of the metric to determine the overall reliability of the measure.

A Framework for Using CEDS for Analysis and Performance Management

Using the Framework Step 1: Framing the Issue The Question The Audience The Audience’s Focuses The Approach Step 2: Selecting the Right Metrics Data Sources Data Matching Variables Used Step 3: Constraining the Metrics Output Step 4: Data Use Wages by Degree Level by Major UAMS Study 10

The Question What is the value of public higher education in the state of Arkansas? 11

The Audience State Level Policy Makers Administrators of Public Institutions of Higher Education Students and Families 12

The Audience Focuses Policy and Funding Issues Program Planning and Staffing Issues College Outcomes for Life 13

The Approach Original Report What percentage of Arkansas residents and non-residents who graduate from public colleges and universities remain in Arkansas? How does level of education impact salary? What is the average salary by field of study and level of degree of Arkansas graduates? New Version What are the employment outcomes of graduates from 2002 to 2009 as of 2010 in the state? What percentage of Arkansas residents and non-residents who graduate from public colleges and universities remain in Arkansas either in the workforce or enrolled in the higher education system? How does level of education impact salary? What is the median salary by field of study and level of degree of Arkansas graduates? In which industry areas are Arkansas graduates employed after they finish college? 14

Variables Used Original Report SSN Graduation Year Degree Level Degree Area Residency Status Employment Status Annual Wages at Year 1 Annual Wages at Year 5 Residency Status New Version SSN DOB First Name Last Name Graduation Date (MM/YYYY) Degree Level Degree Area Residency Status Enrollment Status Employment Status Average Quarters Employed Annual Wages at Year 1 Annual Wages at Year 2 Annual Wages at Year 5 NAICS Code 15 ADHE ADWS

Data Sources Original Report Arkansas Department of Higher Education Arkansas Department of Workforce Services New Version Arkansas Department of Higher Education Arkansas Department of Workforce Services National Student ClearingHouse 16

Data Matching Original Report SSN from ADHE matched to SSN from ADWS New Version TrustEd Broker System SSN DOB First Name Last Name Uses multiple pieces of information about a person to determine if they match. Personally identifiable information is removed and replaced with a non- traceable id. 17

~55,000,000 records for 10 years, 2,938,718 unique SSNs, no DOBs, inconsistent naming standards. 7,865 SSNs used by two or more people, for a total of 18,278 different individuals. Those would be combined incomes and treated as the same person if SSN was the primary key. The same person has two or more SSNs (because of a typo/transposition) 13,373 times. There would be 13,373 additional (non-existent) people with separate incomes if SSN was the primary key. Identity Resolution (Wage Data)

A trusted broker maintains a cross reference table, encoding the identifiers for various agencies and for various representations of the entities. Trusted Broker Bob SmithP0236 Robert SmithK4297 Agency 1Agency 2 Internal IdentifierIdentity Information Identifier Agency1 Identifier Agency2 KB5765Bob Smith, BartonP0236K4297 KB5765Robert Smith, BartonP0236K4297 KB5765Bob Smith, WilsonP0236K4297

Dual Database: Regulatory Compliance and Privacy Identity Information: Agency ID Generated Information of Interest: Stored With Agency ID Knowledge Base: Identity Management Edge Servers: Shareable Data

De-identified Data

Output Original Report Percentage of Graduates Remaining in Arkansas Average Salary by Degree Level Average Salary by Degree Level by Degree Area/Field of Study New Version Percentage of Graduates Remaining in Arkansas Median Wages and Average Quarters Employed by Degree Level Median Wages and Average Quarters Employed by Degree Level by Degree Area/Field of Study Industry Area Employed in by Degree Level 22

Population of Interest and Entry Time Into the Workforce Old Report Graduates from an academic year would only be counted as employed if they were present in wage data during the 4 th QUARTER AFTER THE END OF THE GRADUATION YEAR. New Report Graduates were classified if they were employed at any point during the FIRST FOUR FULL QUARTERS AFTER THEIR GRADUATION DATE. 23

Population of Interest and Entry Time Into the Workforce Example Graduates from the academic year Anyone who graduated from 07/01/2005 to 06/30/2006 Old Report 4 th Quarter after the graduation year=2 nd quarter of 2007 (April-June 2007) Graduates during July-September of 2005 would not be classified as employed unless they had wages for the 2 nd quarter of 2007 which is actually 7 quarters after they graduated not 4. 24

Population of Interest and Entry Time Into the Workforce Example Graduates from the academic year Anyone who graduated from 07/01/2005 to 06/30/2006 New Version First Full Four Quarters after the graduate date 25 1st Four Full Quarters After Graduation Grad DateQ1Q2Q3Q4 07/2005 to 09/ /2005 to 12/ /2006 to 03/ /2006 to 06/

Remained vs. Employed 26 OLD Percentage Remained = # Employed/# of Graduates NEW Percentage Remained = # Employed Only + # Enrolled Only + # Enrolled and Employed/# Graduates Original Report Degree Level Percentage Employed Year 5 Percentage Employed Year Certificate63%64%61%75%76% Associate73% 72%78% 76% Bachelor62%63%64%74%72%71% Master62%60%58%67%68%66% Doctoral44%32%44%49%54%43% 1st Professional55%61%57%51%63%62%

Remained vs. Employed 27 OLD Percentage Employed = # Employed/# of Graduates NEW Percentage Employed = # Employed + #Employed and Enrolled/# Graduates Original Report Degree Level Percentage Employed Year 5 Percentage Employed Year Certificate63%64%61%75%76% Associate73% 72%78% 76% Bachelor62%63%64%74%72%71% Master62%60%58%67%68%66% Doctoral44%32%44%49%54%43% 1st Professional55%61%57%51%63%62%

Average Salary vs. Median Wages 28 OLD Average Wages = Sum(4 th Quarter Wages * 4)/# Employed NEW Median Wages Y1 = Median of Sum(Q1+Q2+Q3+Q4) Original Report Degree Level Average Salary Year Certificate $ 19,396 $ 19,876 $ 18,639 Associate $ 18,868 $ 19,261 $ 19,244 Bachelor $ 26,860 $ 27,332 $ 26,116 Master $ 40,767 $ 41,274 $ 42,305 Doctoral $ 54,966 $ 54,707 $ 54,153 1st Professional $ 44,102 $ 40,551 $ 46,888 Median Wages Year 1 Grad. Year Wage Year CP $ 12,693 $ 13,370 $ 12,657 $ 11,203 TC $ 20,656 $ 20,396 $ 20,261 $ 20,140 AA $ 16,128 $ 16,468 $ 17,350 $ 16,519 BS $ 21,232 $ 21,574 $ 20,831 $ 19,708 MS $ 37,735 $ 38,593 $ 39,780 $ 39,583 1st Pro $ 35,998 $ 28,004 $ 44,250 $ 45,356 PhD $ 50,059 $ 48,981 $ 47,728 $ 44,511 Median Wages Year 2 Grad. Year Wage Year CP $ 16,459 $ 17,207 $ 15,272 $ 14,306 TC $ 23,943 $ 23,650 $ 23,206 $ 23,240 AA $ 19,747 $ 19,380 $ 19,949 $ 19,449 BS $ 28,162 $ 28,749 $ 27,595 $ 27,217 MS $ 41,104 $ 41,842 $ 42,928 $ 42,721 1st Pro $ 42,495 $ 48,347 $ 48,424 $ 49,392 PhD $ 53,900 $ 55,079 $ 54,430 $ 53,225 Median Wages Y2 = Median of Sum(Q5+Q6+Q7+Q8)

Average Quarters Employed 29 OLD Not Used NEW Average Quarters Employed =Sum(Quarters Employed)/# Employed Average Quarters Employed Year 1Average Quarters Employed Year 2 Grad. Year Wage Year CP TC AA BS MS st Pro PhD

Average Salary vs. Median Wages 30 OLD Average Wages = Sum(20 th Quarter Wages * 4)/# Employed NEW Median Wages Y5 = Median of Sum(Q17+Q18+Q19+Q20) Original Report Degree Level Average Salary Year Certificate $ 25,921 $ 26,008 $ 24,317 Associate $ 28,902 $ 29,837 $ 29,835 Bachelor $ 35,866 $ 36,766 $ 37,227 Master $ 45,277 $ 46,836 $ 47,888 Doctoral $ 63,028 $ 61,145 $ 66,059 1st Professional $ 78,095 $ 72,426 $ 79,324 Median Wages Year 1 Grad. Year Wage Year CP $ 14,995 $ 12,864 $ 11,909 $ 11,865 TC $ 16,397 $ 17,650 $ 18,392 $ 20,285 AA $ 14,509 $ 14,483 $ 15,120 $ 15,139 BS $ 17,186 $ 17,500 $ 19,325 $ 20,175 MS $ 32,417 $ 32,987 $ 34,398 $ 35,694 1st Pro $ 29,996 $ 35,647 $ 37,477 $ 31,500 PhD $ 44,673 $ 41,398 $ 45,897 $ 47,013 Median Wages Year 2 Grad. Year Wage Year CP $ 18,382 $ 17,044 $ 15,662 $ 15,790 TC $ 20,161 $ 20,786 $ 21,836 $ 23,023 AA $ 17,874 $ 17,122 $ 17,798 $ 18,089 BS $ 23,908 $ 24,923 $ 26,998 $ 27,404 MS $ 35,785 $ 37,496 $ 39,732 $ 39,228 1st Pro $ 43,731 $ 46,101 $ 48,638 $ 47,242 PhD $ 47,059 $ 48,477 $ 54,713 $ 55,730 Median Wages Year 5 Grad. Year Wage Year CP $ 26,504 $ 26,445 $ 21,794 $ 22,176 TC $ 26,920 $ 26,312 $ 27,865 $ 29,282 AA $ 29,899 $ 30,017 $ 30,617 $ 29,851 BS $ 36,580 $ 37,644 $ 37,762 $ 37,797 MS $ 46,555 $ 47,114 $ 48,648 $ 47,574 1st Pro $ 82,954 $ 83,162 $ 81,453 $ 68,626 PhD $ 59,657 $ 67,444 $ 67,665 $ 67,169

Average Quarters Employed 31 OLD Not Used NEW Average Quarters Employed =Sum(Quarters Employed)/# Employed Average Quarters Employed Year 1Average Quarters Employed Year 2Average Quarters Employed Year 5 Grad. Year Wage Year CP TC AA BS MS st Pro PhD

Wages by Degree Level by Major 32 OLD Average Wages = Sum(4 th Quarter Wages * 4)/# Employed NEW Median Wages 2010 = Median of Sum2010(Q1+Q2+Q3+Q4) Original Report-HEALTH SCIENCES MAJORS Degree Level Average Salary Year 5Average Salary Year Bachelor $ 43,713 $ 44,882 $ 47,581 $ 37,448 $ 38,034 $ 38,317 Years Since Graduation1 Year2 Years3 Years4 Years5 Years6 Years7 Years8 Years Graduation Year HEALTH PROFESSIONS AND RELATED PROGRAMS CP %Employed81%79% 75% 77%71%72% Median Salary11,66215,00716,61222,37421,43219,44922,60828,253 TC %Employed86%88%85%82%79% 76%70% Median Salary25,67625,59328,13129,40229,27430,22031,24731,587 AA %Employed83%75%76%75%73%71%69%67% Median Salary38,14439,15940,31442,69141,98243,56844,71345,641 BS %Employed76%70%68% 66%67%63% Median Salary37,06739,15643,68144,75144,71148,97748,20046,942 MS %Employed66%63%60%53%55%61%58% Median Salary55,07052,75458,68658,53255,55055,50358,55156,490 1STPRO %Employed62%67%64%68%64%57%58%61% Median Salary56,93952,35180,197105,787111,052118,980138,125126,844 PHD %Employed69%79%75%79%65%54%38%46% Median Salary56,00057,92762,48260,47966,67168,80375,74978,659

Wages by Degree Level by Major 33 OLD Average Wages = Sum(4 th Quarter Wages * 4)/# Employed NEW Median Wages 2010 = Median of Sum2010(Q1+Q2+Q3+Q4) Original Report-EDUCATION Degree Level Average Salary Year 5Average Salary Year Bachelor$34,791$36,121$36,218$30,233$30,806$30,534

UAMS Study

Arkansas Research Center

Contact Information arc.arkansas.gov 36